Log files source: follow the logs of Traefik, nginx and other programs

Files source (source_type=file), declared in Settings > Log sources > Log
files (/data/files.json, 50 at most): name (= app), path as mounted in the
container (file or glob, rotated copies .1/.gz left out, 200 files per
glob), format, optional host, switch.

- formats: plain text (level detected, leading timestamp read and removed),
  JSON lines (msg/time/level keys, numeric pino levels, other keys as
  fields, nested ones dotted), logfmt, access log CLF (nginx, Apache,
  Traefik CLF with router/backend/duration), Traefik access log in JSON
  (summary "GET host/path 200 · 12.5 ms · router · client"), regular
  expression with named groups time/level/msg/host/app/status; 5xx = err,
  4xx = warning; fields named like LogStream's own renamed field_<name>
- indented lines (text) and non-matching lines (regex) added to the
  previous entry, which waits 2 s for them
- inode + offset per file in /data/files-state.json, rotation (end of
  .1 read first), copytruncate, file renamed within the glob; last
  FILES_BACKFILL (1h) of dated lines read when turned on, new files read
  from their start
- status per file (files followed, entries read, not found / permission)

UI: rows with switch, name, path, format, host (regex field when needed),
status line; green edge, tint and FILE badge in the list, "Files" in the
source filter, HTTP fields first in the details, FR/EN. docker-compose,
.env.example and README (EN/FR): mounts, formats, FILES_BACKFILL.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
cedricandClaude Opus 5.5 committed 2026-10-05 10:27:54 +02:00
1 parent e87186e476
commit 087d9e7b30
14 files changed
+1934 -9

No files matched your search

+4
View File
@@ -67,3 +67,7 @@ DMESG_BACKFILL=1h
AUDIT_BACKFILL=1h
# Host name of the audit events (empty: node= of the records, else /host/etc/hostname, else localhost)
AUDIT_HOST=
# Log files of other programs (Traefik, nginx, applications…), declared in Settings > Log sources >
# Log files: mount their directory read-only in docker-compose.yml (e.g. /var/log/traefik:/logs/traefik:ro).
# History read when a file is turned on (e.g. 30m, 1h, 24h; 0 = only new lines)
FILES_BACKFILL=1h
+57 -1
View File
@@ -289,10 +289,64 @@ source est **désactivée par défaut** : activez-la dans **Réglages > Sources
enregistrés dans `/data/audit-state.json` : après un redémarrage, la lecture reprend là ;
après une rotation par auditd, la fin de `audit.log.1` est lue avant le nouveau fichier.
## Fichiers de logs (Traefik, nginx, applications…)
Logstream peut suivre les fichiers de logs d'autres programmes : Traefik, nginx, Apache, ou
toute application qui écrit dans un fichier. Déclarez chaque fichier dans **Paramètres > Sources
de logs > Fichiers de logs** (enregistré dans `/data/files.json`, 50 au plus) : un **nom** (l'**app**
de ses entrées), le **chemin** tel que monté dans le conteneur, un **format**, un **hôte**
facultatif (par défaut `/host/etc/hostname`, sinon `localhost`) et un interrupteur.
- **Montage** du dossier en lecture seule, par ex. dans `docker-compose.yml` :
`- /var/log/traefik:/logs/traefik:ro`, puis `docker compose up -d`. Les fichiers de `/var/log`
sont déjà accessibles sous `/host/var/log/…`. Les fichiers doivent être lisibles par l'UID
`10001` ou le groupe de `HOST_LOGS_GID` (`adm` par défaut) ; Paramètres > Sources de logs
indique quand aucun fichier ne correspond ou que l'accès est refusé.
- **Chemin** : un fichier, ou un motif avec `*`, `?` ou `[…]` : `/logs/traefik/*.log` suit tous
les fichiers `.log` du dossier, y compris ceux créés plus tard (200 fichiers au plus par
motif). Les copies de rotation (`.1`, `.2.gz`…) sont ignorées.
- **Formats** :
- *Texte brut* : une entrée par ligne ; le niveau est détecté comme pour les logs Docker, un
horodatage en début de ligne (ISO 8601, `2006/01/02 15:04:05`, `[02/Jan/2006:15:04:05 +0200]`,
`Jan 2 15:04:05`) devient l'heure du message et est retiré du message. Les lignes indentées
(stack traces) sont rattachées à l'entrée précédente.
- *JSON lines* : `msg` / `message`, `time` / `ts` / `@timestamp` (texte ou heure Unix) et
`level` (mot ou nombre, pino/bunyan compris) sont reconnus ; les autres clés deviennent des
champs, les objets imbriqués avec des noms pointés (`req.id`).
- *logfmt* : paires `clé=valeur`, comme les logs par défaut de Traefik ou des programmes Go ;
mêmes clés que le JSON.
- *Journal d'accès (CLF)* : Common / Combined Log Format de nginx et Apache, et le journal
d'accès de Traefik dans ce format (routeur, URL du serveur et durée compris). Champs
`client_ip`, `http_method`, `http_path`, `http_status`, `http_size`, `user_agent`,
`http_referer`, `router`, `backend`, `duration_ms`.
- *Accès Traefik (JSON)* : le journal d'accès JSON de Traefik (`accessLog.format: json`),
résumé en `GET app.example.com/ 200 · 12.5 ms · app@docker · 203.0.113.9`, toutes les clés
dans les champs. Les logs de Traefik lui-même dans le même fichier sont lus comme du JSON.
- *Expression régulière* (syntaxe RE2 / Go) : les groupes nommés `time`, `level`, `msg`,
`host` et `app` remplissent ces champs, `status` donne le niveau d'un journal d'accès, les
autres groupes deviennent des champs, par ex.
`^(?P<time>\S+ \S+) \[(?P<level>\w+)\] (?P<module>\S+): (?P<msg>.*)$`. Une ligne qui ne
correspond pas est rattachée à l'entrée précédente.
Pour les journaux d'accès, les statuts 5xx sont en `err`, les 4xx en `warning`, les autres en
`info`. Une ligne qui ne suit pas le format choisi est gardée comme du texte brut. Les champs
qui portent le nom d'un champ de LogStream (`host`, `app`…) sont renommés `field_host`,
`field_app`…
- **Affichage** : source `file`, le chemin dans `log_file`. Dans la liste, ces lignes ont un
liseré gauche vert, une légère teinte et un badge **FILE** avant l'app ; le filtre **Source**
les affiche seules.
- À l'activation d'un fichier, les lignes de la dernière heure déjà présentes sont lues d'abord
(`FILES_BACKFILL`, `0` = seulement les nouvelles lignes ; les lignes sans horodatage sont alors
ignorées). Les fichiers qui apparaissent ensuite sont lus depuis leur début. L'inode et une
position sont enregistrés par fichier dans `/data/files-state.json` : après un redémarrage la
lecture reprend là ; après une rotation (`access.log` renommé `access.log.1`), la fin de
l'ancien fichier est lue avant le nouveau ; un fichier vidé sur place (`copytruncate`) est
relu depuis son début.
## Logs internes de LogStream
Logstream enregistre ses propres événements comme des logs, avec l'**hôte** `logstream`,
l'**app** du composant (`store`, `syslog`, `docker`, `host`, `dmesg`, `audit`, `main`) et la
l'**app** du composant (`store`, `syslog`, `docker`, `host`, `dmesg`, `audit`, `files`, `main`) et la
source `logstream` : VictoriaLogs injoignable puis de retour, lots perdus, lots refusés depuis
le tampon disque, erreurs du listener syslog, API Docker ou flux d'un conteneur décrochés,
sources qui ne peuvent pas lire, démarrage et arrêt. Ils sont actifs par défaut à partir du
@@ -503,6 +557,7 @@ résolutions.
| `AUDIT_LOG` | `/var/log/audit/audit.log` | chemin par défaut du journal d'audit Linux (modifiable dans Réglages > Sources) |
| `AUDIT_BACKFILL` | `1h` | historique lu à l'activation de la source « audit Linux » |
| `AUDIT_HOST` | vide | nom d'hôte des événements d'audit quand les enregistrements n'ont pas de `node=` |
| `FILES_BACKFILL` | `1h` | historique lu à l'activation d'un fichier de logs (Paramètres > Sources de logs > Fichiers de logs) |
| `TZ` | `Europe/Paris` | fuseau horaire des horodatages RFC 3164 (qui n'en portent pas) |
| `BATCH_SIZE`, `FLUSH_MS`, `QUEUE_SIZE` | `1000`, `1000`, `100000` | réglage de l'ingestion |
| `SPOOL_MAX_MB` | `1024` | taille du tampon disque des lots refusés par VictoriaLogs (`0` = pas de tampon : 15 s de tentatives, puis perte) |
@@ -565,6 +620,7 @@ Pour mettre à jour l'une d'elles :
| `hostlogs.go`, `journal.go` | logs système de l'hôte : lecteur du journal systemd (sans `journalctl`) et suivi de `/var/log` |
| `dmesg.go` | messages du noyau : lecture de `/dev/kmsg`, analyse des enregistrements, position par identifiant de démarrage et numéro de séquence |
| `audit.go` | audit Linux : suivi de `audit.log` (rotation, position enregistrée), analyse des enregistrements, regroupement par événement, résumé |
| `filesource.go`, `fileformats.go` | fichiers de logs : suivi (motifs, rotation, positions enregistrées, lignes de suite) et formats (texte, JSON, logfmt, CLF, Traefik, expression régulière) |
| `selflog.go` | logs internes de LogStream : `logEvent`, niveau minimal, limitation des répétitions, file bornée |
| `tags.go` | stockage des tags de couleur |
| `presets.go`, `presets.json` | préréglages de tags (`/api/presets`), liste intégrée |
+52 -1
View File
@@ -265,10 +265,59 @@ also be changed (saved in `/data/audit.json`; `AUDIT_LOG` sets the default path)
`/data/audit-state.json`: after a restart, reading resumes there; after a rotation by auditd,
the end of `audit.log.1` is read before the new file.
## Log files (Traefik, nginx, applications…)
Logstream can follow the log files of other programs: Traefik, nginx, Apache, or any application
that writes to a file. Declare each file in **Settings > Log sources > Log files** (saved in
`/data/files.json`, up to 50): a **name** (the **app** of its entries), the **path** as mounted in
the container, a **format**, an optional **host** (default: `/host/etc/hostname`, else
`localhost`) and a switch.
- **Mount** the directory read-only, e.g. in `docker-compose.yml`:
`- /var/log/traefik:/logs/traefik:ro`, then `docker compose up -d`. Files under `/var/log`
are already reachable as `/host/var/log/…`. The files must be readable by UID `10001` or the
group of `HOST_LOGS_GID` (`adm` by default); Settings > Log sources says when no file matches
or access is denied.
- **Path**: a file, or a pattern with `*`, `?` or `[…]`: `/logs/traefik/*.log` follows every
`.log` file of the directory, also those created later (200 files at most per pattern).
Rotated copies (`.1`, `.2.gz`…) are left out.
- **Formats**:
- *Plain text*: one entry per line; the level is detected as for Docker logs, a timestamp at
the start of the line (ISO 8601, `2006/01/02 15:04:05`, `[02/Jan/2006:15:04:05 +0200]`,
`Jan 2 15:04:05`) becomes the message time and is removed from the message. Indented
lines (stack traces) are added to the previous entry.
- *JSON lines*: `msg` / `message`, `time` / `ts` / `@timestamp` (text or Unix time) and
`level` (word or number, pino/bunyan included) are recognized; the other keys become
fields, nested objects with dotted names (`req.id`).
- *logfmt*: `key=value` pairs, as in Traefik's default logs or Go programs; same keys as JSON.
- *Access log (CLF)*: Common / Combined Log Format of nginx and Apache, and Traefik's access
log in this format (router, server URL and duration included). Fields `client_ip`,
`http_method`, `http_path`, `http_status`, `http_size`, `user_agent`, `http_referer`,
`router`, `backend`, `duration_ms`.
- *Traefik access (JSON)*: the JSON access log of Traefik (`accessLog.format: json`),
summarized as `GET app.example.com/ 200 · 12.5 ms · app@docker · 203.0.113.9`, every key in
the fields. Traefik's own logs in the same file are read as JSON lines.
- *Regular expression* (RE2 / Go syntax): the named groups `time`, `level`, `msg`, `host`
and `app` fill these fields, `status` gives the level of an access log, the other groups
become fields, e.g. `^(?P<time>\S+ \S+) \[(?P<level>\w+)\] (?P<module>\S+): (?P<msg>.*)$`.
A line that does not match is added to the previous entry.
For access logs, 5xx statuses are `err`, 4xx `warning`, the others `info`. A line that does
not fit the chosen format is kept as plain text. Fields named like LogStream's own ones
(`host`, `app`…) are renamed `field_host`, `field_app`…
- **Display**: source `file`, the path in `log_file`. In the list these lines have a green left
edge, a light tint and a **FILE** badge before the app; the **Source** filter shows them alone.
- When a file is turned on, the lines of the last hour already in it are read first
(`FILES_BACKFILL`, `0` = only new lines; lines without a timestamp are then left out). Files
that appear later are read from their start. The inode and an offset are saved per file in
`/data/files-state.json`: after a restart reading resumes there; after a rotation
(`access.log` renamed `access.log.1`), the end of the old file is read before the new one;
a file truncated in place (`copytruncate`) is read again from its start.
## LogStream internal logs
Logstream records its own events as logs, with **host** `logstream`, **app** the component
(`store`, `syslog`, `docker`, `host`, `dmesg`, `audit`, `main`) and the `logstream` source:
(`store`, `syslog`, `docker`, `host`, `dmesg`, `audit`, `files`, `main`) and the `logstream` source:
VictoriaLogs unreachable and back, lost batches, batches refused from the disk buffer, syslog
listener errors, Docker API or container streams lost, sources that cannot read, start and
stop. They are on by default from level **warning**; **Settings > Log sources > LogStream internal
@@ -460,6 +509,7 @@ are only known by your router or a local DNS (Pi-hole, AdGuard, Unbound…), set
| `AUDIT_LOG` | `/var/log/audit/audit.log` | default path of the Linux audit log (can be changed in Settings > Log sources) |
| `AUDIT_BACKFILL` | `1h` | history read when the Linux audit source is turned on |
| `AUDIT_HOST` | empty | host name of the audit events when the records carry no `node=` |
| `FILES_BACKFILL` | `1h` | history read when a log file is turned on (Settings > Log sources > Log files) |
| `TZ` | `Europe/Paris` | time zone for RFC 3164 timestamps (which carry none) |
| `BATCH_SIZE`, `FLUSH_MS`, `QUEUE_SIZE` | `1000`, `1000`, `100000` | ingestion tuning |
| `SPOOL_MAX_MB` | `1024` | disk buffer size for batches VictoriaLogs could not take (`0` = no buffer: retried for 15 s, then dropped) |
@@ -520,6 +570,7 @@ To update one of them:
| `hostlogs.go`, `journal.go` | host system logs: systemd journal reader (no `journalctl`) and `/var/log` follower |
| `dmesg.go` | kernel messages: `/dev/kmsg` reader, record parsing, position by boot ID and sequence number |
| `audit.go` | Linux audit: `audit.log` follower (rotation, saved offset), record parsing, grouping by event, summary |
| `filesource.go`, `fileformats.go` | log files: follower (patterns, rotation, saved offsets, continuation lines) and formats (text, JSON, logfmt, CLF, Traefik, regular expression) |
| `selflog.go` | LogStream internal logs: `logEvent`, minimum level, throttling, bounded queue |
| `tags.go` | color tag storage |
| `presets.go`, `presets.json` | color tag presets (`/api/presets`), built-in list |
+24
View File
@@ -25,6 +25,7 @@ type API struct {
host *HostLogs
dmesg *Dmesg
audit *Audit
files *Files
selfLog *SelfLog
dbstats dbStatsCache
}
@@ -56,6 +57,8 @@ func (a *API) Routes(mux *http.ServeMux) {
mux.HandleFunc("PUT /api/dmesg", a.dmesgConfigure)
mux.HandleFunc("GET /api/audit", a.auditStatus)
mux.HandleFunc("PUT /api/audit", a.auditConfigure)
mux.HandleFunc("GET /api/files", a.filesStatus)
mux.HandleFunc("PUT /api/files", a.filesConfigure)
mux.HandleFunc("GET /api/selflog", a.selfLogStatus)
mux.HandleFunc("PUT /api/selflog", a.selfLogConfigure)
}
@@ -80,6 +83,27 @@ func (a *API) auditConfigure(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, a.audit.Status())
}
// GET /api/files: the log files of the "Files" source (Settings > Sources).
func (a *API) filesStatus(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, a.files.Status())
}
// PUT /api/files {"sources": [{"id", "name", "path", "format", "regex", "host", "enabled"}]}
// (no id: a new file).
func (a *API) filesConfigure(w http.ResponseWriter, r *http.Request) {
var cfg filesConfig
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 64<<10)).Decode(&cfg); err != nil {
writeErr(w, http.StatusBadRequest, err)
return
}
if err := a.files.Configure(cfg); err != nil {
writeErr(w, http.StatusBadRequest, err)
return
}
log.Printf("file sources changed from %s: %d files", r.RemoteAddr, len(cfg.Sources))
writeJSON(w, http.StatusOK, a.files.Status())
}
// GET /api/selflog: state of the LogStream internal logs (Settings > Sources).
func (a *API) selfLogStatus(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, a.selfLog.Status())
+3
View File
@@ -39,6 +39,7 @@ services:
HOST_LOGS_BACKFILL: ${HOST_LOGS_BACKFILL:-1h} # historique lu a l'activation des logs systeme de l'hote
DMESG_BACKFILL: ${DMESG_BACKFILL:-1h} # historique lu a l'activation des messages du noyau (dmesg)
AUDIT_BACKFILL: ${AUDIT_BACKFILL:-1h} # historique lu a l'activation de l'audit Linux (auditd)
FILES_BACKFILL: ${FILES_BACKFILL:-1h} # historique lu a l'activation d'un fichier de logs (Reglages > Sources)
AUDIT_HOST: ${AUDIT_HOST:-} # nom d'hote des evenements d'audit (vide : node= des enregistrements, sinon /host/etc/hostname)
group_add:
- "${HOST_LOGS_GID:-4}" # groupe autorise a lire les logs de l'hote (4 = adm sur Debian/Ubuntu)
@@ -63,6 +64,8 @@ services:
# noms des utilisateurs et de l'hote pour les evenements d'audit (optionnel)
# - /etc/passwd:/host/etc/passwd:ro
# - /etc/hostname:/host/etc/hostname:ro
# fichiers de logs d'autres programmes (Traefik, nginx...), a declarer dans Reglages > Sources > Fichiers de logs
# - /var/log/traefik:/logs/traefik:ro
# logo de la page de connexion, avec LOGIN_LOGO=/config/logo.png dans .env
# - ./logo.png:/config/logo.png:ro
# prereglages de tags personnalises, avec PRESETS_FILE=/config/presets.json dans .env
+589
View File
@@ -0,0 +1,589 @@
package main
import (
"encoding/json"
"fmt"
"regexp"
"strconv"
"strings"
"time"
"unicode/utf8"
)
// Formats of the "Files" source (Settings > Sources), one per declared file.
const (
fmtText = "text" // one line per entry, level and leading timestamp detected
fmtJSON = "json" // JSON lines: msg/time/level recognized, other keys as fields
fmtLogfmt = "logfmt" // key=value pairs (Traefik "common" logs, Go programs…)
fmtRegex = "regex" // regular expression with named groups
fmtCLF = "clf" // access logs in Common/Combined Log Format (nginx, Apache, Traefik)
fmtTraefik = "traefik" // Traefik access log in JSON
)
var fileFormats = map[string]bool{fmtText: true, fmtJSON: true, fmtLogfmt: true, fmtRegex: true, fmtCLF: true, fmtTraefik: true}
// fileRec is one parsed line (or several, for continuation lines).
type fileRec struct {
t time.Time
dated bool // t comes from the line
sev int
msg string
host string
app string
fields map[string]string
}
const (
maxFileFields = 64
maxFieldValue = 4096
maxFileMsg = 64 << 10 // message with its continuation lines
maxContinuation = 500 // continuation lines added to one entry
)
// fileParser turns a line of a file into an entry, for one format.
type fileParser struct {
format string
re *regexp.Regexp // fmtRegex
}
func newFileParser(format, expr string) (*fileParser, error) {
if !fileFormats[format] {
return nil, fmt.Errorf("unknown format %q", format)
}
p := &fileParser{format: format}
if format == fmtRegex {
if strings.TrimSpace(expr) == "" {
return nil, fmt.Errorf("the regex format needs a regular expression")
}
re, err := regexp.Compile(expr)
if err != nil {
return nil, fmt.Errorf("invalid regular expression: %v", err)
}
p.re = re
}
return p, nil
}
// multiline tells whether lines that do not start an entry are added to the
// previous one (stack traces…). Structured formats are one entry per line.
func (p *fileParser) multiline() bool {
return p.format == fmtText || p.format == fmtRegex
}
// continuation reports a line that belongs to the previous entry: indented,
// or not matching the regular expression.
func (p *fileParser) continuation(line string) bool {
if line != "" && (line[0] == ' ' || line[0] == '\t') {
return true
}
return p.format == fmtRegex && !p.re.MatchString(line)
}
// parse decodes one line; it never fails (a line in another format is kept as text).
func (p *fileParser) parse(line string, now time.Time) *fileRec {
line = cleanLine(line)
switch p.format {
case fmtJSON, fmtTraefik:
if obj, ok := decodeJSONLine(line); ok {
if p.format == fmtTraefik {
if r := traefikRec(obj, line, now); r != nil {
return r
}
}
return jsonRec(obj, line, now)
}
case fmtLogfmt:
if kv := parseLogfmt(line); len(kv) > 0 {
return logfmtRec(kv, line, now)
}
case fmtRegex:
if m := p.re.FindStringSubmatch(line); m != nil {
return p.regexRec(m, line, now)
}
case fmtCLF:
if r := clfRec(line, now); r != nil {
return r
}
}
return textRec(line, now)
}
func cleanLine(s string) string {
s = strings.TrimRight(s, "\r")
s = ansiRe.ReplaceAllString(s, "")
if !utf8.ValidString(s) {
s = strings.ToValidUTF8(s, "�")
}
return s
}
// textRec: the line itself, without its leading timestamp when there is one.
func textRec(line string, now time.Time) *fileRec {
r := &fileRec{t: now, sev: 6, msg: line}
if m := leadTimeRe.FindStringSubmatchIndex(line); m != nil {
if t, ok := parseLogTime(line[m[2]:m[3]], now); ok {
r.t, r.dated = t, true
if rest := line[m[1]:]; strings.TrimSpace(rest) != "" {
r.msg = rest
}
}
}
if n, ok := detectLevel(r.msg); ok {
r.sev = n
}
return r
}
/* ---------- Timestamps ---------- */
// leadTimeRe: a timestamp at the start of a line, possibly in brackets
// (ISO 8601 / RFC 3339, 2006/01/02 15:04:05, CLF, syslog "Jan _2 15:04:05").
var leadTimeRe = regexp.MustCompile(`^\[?(\d{4}[-/]\d{2}[-/]\d{2}[T ]\d{2}:\d{2}:\d{2}(?:[.,]\d+)?(?: ?(?:Z|[+-]\d{2}:?\d{2}))?|\d{2}/[A-Z][a-z]{2}/\d{4}:\d{2}:\d{2}:\d{2} [+-]\d{4}|[A-Z][a-z]{2} +\d{1,2} \d{2}:\d{2}:\d{2})\]?[ \t]*(?:[-|:][ \t]+)?`)
var timeLayouts = []string{
time.RFC3339Nano,
"2006-01-02T15:04:05Z0700",
"2006-01-02T15:04:05",
"02/Jan/2006:15:04:05 -0700",
"Jan _2 15:04:05",
time.RFC1123Z,
time.RFC1123,
time.UnixDate,
time.ANSIC,
}
// parseLogTime reads the usual log timestamps; times without a zone are local.
func parseLogTime(s string, now time.Time) (time.Time, bool) {
s = strings.TrimSpace(s)
if s == "" {
return time.Time{}, false
}
if s[0] >= '0' && s[0] <= '9' && !strings.ContainsAny(s, ":/-") {
if whole, _, _ := strings.Cut(s, "."); len(whole) < 9 {
return time.Time{}, false // not a Unix time of this century
}
return epochTime(s)
}
// 2006/01/02 15:04:05,123 +02:00 -> 2006-01-02T15:04:05.123+02:00
if len(s) >= 19 && s[4] == '/' && s[7] == '/' {
s = s[:4] + "-" + s[5:7] + "-" + s[8:]
}
if len(s) >= 19 && s[4] == '-' && s[10] == ' ' {
s = s[:10] + "T" + s[11:]
}
if len(s) > 19 && s[4] == '-' {
s = s[:19] + strings.Replace(strings.Replace(s[19:], ",", ".", 1), " ", "", 1)
}
s = strings.Join(strings.Fields(s), " ") // "Jan 5" -> "Jan 5"
for _, layout := range timeLayouts {
if layout == "Jan _2 15:04:05" {
layout = "Jan 2 15:04:05"
}
t, err := time.ParseInLocation(layout, s, time.Local)
if err != nil {
continue
}
if t.Year() == 0 { // syslog timestamps have no year
t = t.AddDate(now.Year(), 0, 0)
if t.After(now.Add(24 * time.Hour)) {
t = t.AddDate(-1, 0, 0)
}
}
return t, true
}
return time.Time{}, false
}
// epochTime reads a Unix time in seconds, milliseconds, microseconds or
// nanoseconds (guessed from its size), with an optional fraction.
func epochTime(s string) (time.Time, bool) {
f, err := strconv.ParseFloat(s, 64)
if err != nil || f <= 0 {
return time.Time{}, false
}
switch {
case f < 1e11:
return time.Unix(0, int64(f*1e9)), true
case f < 1e14:
return time.UnixMilli(int64(f)), true
case f < 1e17:
return time.UnixMicro(int64(f)), true
}
return time.Unix(0, int64(f)), true
}
/* ---------- Fields ---------- */
// reservedFields are stored by LogStream itself: fields of the logs with the
// same name are renamed field_<name>.
var reservedFields = map[string]bool{
"_time": true, "_msg": true, "_stream": true, "_stream_id": true, "received": true, "msg_time": true,
"host": true, "host_name": true, "host_ip": true, "app": true, "facility": true, "severity": true,
"sevnum": true, "source": true, "proto": true, "procid": true, "sd": true, "source_type": true, "log_file": true,
}
func fieldName(k string) string {
k = strings.Map(func(r rune) rune {
if r == '_' || r == '.' || r == '-' || r >= '0' && r <= '9' || r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' {
return r
}
return '_'
}, k)
if k == "" || reservedFields[k] || k[0] == '_' {
k = "field_" + strings.TrimLeft(k, "_")
}
return k
}
func (r *fileRec) set(k, v string) {
if v == "" {
return
}
if r.fields == nil {
r.fields = map[string]string{}
}
k = fieldName(k)
if _, ok := r.fields[k]; !ok && len(r.fields) >= maxFileFields {
return
}
if len(v) > maxFieldValue {
v = strings.ToValidUTF8(v[:maxFieldValue], "") + "…"
}
r.fields[k] = v
}
/* ---------- JSON lines ---------- */
func decodeJSONLine(line string) (map[string]any, bool) {
s := strings.TrimSpace(line)
if len(s) < 2 || s[0] != '{' {
return nil, false
}
dec := json.NewDecoder(strings.NewReader(s))
dec.UseNumber()
var obj map[string]any
if dec.Decode(&obj) != nil {
return nil, false
}
return obj, true
}
var (
jsonMsgKeys = []string{"msg", "message", "@message", "log", "text"}
jsonTimeKeys = []string{"time", "ts", "timestamp", "@timestamp", "@t", "t", "datetime", "date"}
jsonLevelKeys = []string{"level", "lvl", "severity", "log.level", "@level", "@l", "loglevel", "levelname"}
)
// takeString removes the first of keys found in obj and returns it as text.
func takeString(obj map[string]any, keys []string) (string, bool) {
for _, k := range keys {
if v, ok := obj[k]; ok {
if s := scalarString(v); s != "" {
delete(obj, k)
return s, true
}
}
}
return "", false
}
func scalarString(v any) string {
switch v := v.(type) {
case string:
return v
case json.Number:
return v.String()
case bool:
return strconv.FormatBool(v)
}
return ""
}
func jsonRec(obj map[string]any, line string, now time.Time) *fileRec {
r := &fileRec{t: now, sev: 6, msg: line}
for _, k := range jsonTimeKeys {
if t, ok := parseLogTime(scalarString(obj[k]), now); ok {
r.t, r.dated = t, true
delete(obj, k)
break
}
}
lvl, hasLevel := takeString(obj, jsonLevelKeys)
if hasLevel {
if n, ok := levelNum(lvl); ok {
r.sev = n
} else if n, err := strconv.Atoi(lvl); err == nil {
r.sev = numericLevel(n)
} else {
r.set("level", lvl)
}
}
if s, ok := takeString(obj, jsonMsgKeys); ok {
r.msg = strings.TrimRight(s, "\n")
}
flattenJSON("", obj, r)
if !hasLevel {
if n, ok := detectLevel(r.msg); ok {
r.sev = n
}
}
return r
}
// numericLevel maps numeric levels: syslog (0-7) or pino/bunyan (10-60).
func numericLevel(n int) int {
switch {
case n >= 0 && n <= 7:
return n
case n >= 60:
return 2
case n >= 50:
return 3
case n >= 40:
return 4
case n >= 30:
return 6
}
return 7
}
// flattenJSON stores the remaining keys as fields; nested objects give
// dotted names, arrays stay in JSON.
func flattenJSON(prefix string, obj map[string]any, r *fileRec) {
for k, v := range obj {
name := prefix + k
switch v := v.(type) {
case map[string]any:
flattenJSON(name+".", v, r)
case []any:
b, _ := json.Marshal(v)
r.set(name, string(b))
case nil:
default:
r.set(name, scalarString(v))
}
}
}
/* ---------- logfmt ---------- */
// parseLogfmt splits `k=v k2="v 2"` (with \" escapes); words without a value
// are skipped. Nil when the line has no pair.
func parseLogfmt(s string) [][2]string {
var out [][2]string
for i := 0; i < len(s); {
for i < len(s) && (s[i] == ' ' || s[i] == '\t') {
i++
}
start := i
for i < len(s) && s[i] != '=' && s[i] != ' ' && s[i] != '\t' {
i++
}
if i >= len(s) || s[i] != '=' || i == start {
for i < len(s) && s[i] != ' ' && s[i] != '\t' {
i++
}
continue
}
k := s[start:i]
i++
var v string
if i < len(s) && s[i] == '"' {
var b strings.Builder
i++
for i < len(s) && s[i] != '"' {
if s[i] == '\\' && i+1 < len(s) {
i++
switch s[i] {
case 'n':
b.WriteByte('\n')
case 't':
b.WriteByte('\t')
default:
b.WriteByte(s[i])
}
} else {
b.WriteByte(s[i])
}
i++
}
i++ // closing quote
v = b.String()
} else {
end := i
for end < len(s) && s[end] != ' ' && s[end] != '\t' {
end++
}
v, i = s[i:end], end
}
out = append(out, [2]string{k, v})
}
return out
}
func logfmtRec(kv [][2]string, line string, now time.Time) *fileRec {
obj := make(map[string]any, len(kv))
for _, p := range kv {
obj[p[0]] = p[1]
}
return jsonRec(obj, line, now)
}
/* ---------- Regular expression ---------- */
func (p *fileParser) regexRec(m []string, line string, now time.Time) *fileRec {
r := &fileRec{t: now, sev: 6, msg: line}
hasLevel := false
for i, name := range p.re.SubexpNames() {
if i == 0 || name == "" || m[i] == "" {
continue
}
v := m[i]
switch strings.ToLower(name) {
case "time", "timestamp", "ts":
if t, ok := parseLogTime(v, now); ok {
r.t, r.dated = t, true
}
case "level", "severity", "lvl":
if n, ok := levelNum(v); ok {
r.sev, hasLevel = n, true
}
case "msg", "message":
r.msg = v
case "host":
r.host = v
case "app":
r.app = v
case "status":
r.set(name, v)
if n, ok := statusLevel(v); ok && !hasLevel {
r.sev, hasLevel = n, true
}
default:
r.set(name, v)
}
}
if !hasLevel {
if n, ok := detectLevel(r.msg); ok {
r.sev = n
}
}
return r
}
/* ---------- Access logs ---------- */
// statusLevel: 5xx are errors, 4xx warnings, the rest info.
func statusLevel(s string) (int, bool) {
n, err := strconv.Atoi(s)
if err != nil || n < 100 || n > 999 {
return 0, false
}
switch {
case n >= 500:
return 3, true
case n >= 400:
return 4, true
}
return 6, true
}
// clfRe: Common Log Format, the combined extension (referer, user agent) and
// the Traefik one (request count, router, server URL, duration).
var clfRe = regexp.MustCompile(`^(\S+) (\S+) (\S+) \[([^\]]+)\] "((?:[^"\\]|\\.)*)" (\d{3}) (\d+|-)(?: "((?:[^"\\]|\\.)*)" "((?:[^"\\]|\\.)*)")?(?: (\d+) "([^"]*)" "([^"]*)" (\d+)ms)?`)
func clfRec(line string, now time.Time) *fileRec {
m := clfRe.FindStringSubmatch(line)
if m == nil {
return nil
}
r := &fileRec{t: now, sev: 6, msg: line}
if t, ok := parseLogTime(m[4], now); ok {
r.t, r.dated = t, true
}
dash := func(s string) string {
if s == "-" {
return ""
}
return s
}
r.set("client_ip", m[1])
r.set("http_user", dash(m[3]))
if req := strings.Fields(m[5]); len(req) == 3 {
r.set("http_method", req[0])
r.set("http_path", req[1])
r.set("http_proto", req[2])
} else {
r.set("http_request", m[5])
}
r.set("http_status", m[6])
r.set("http_size", dash(m[7]))
r.set("http_referer", dash(m[8]))
r.set("user_agent", dash(m[9]))
r.set("router", dash(m[11]))
r.set("backend", dash(m[12]))
r.set("duration_ms", m[13])
r.sev, _ = statusLevel(m[6])
return r
}
// traefikRec reads a Traefik access log line in JSON; nil for the other
// lines (Traefik's own logs), read as generic JSON.
func traefikRec(obj map[string]any, line string, now time.Time) *fileRec {
status := scalarString(obj["DownstreamStatus"])
if status == "" {
status = scalarString(obj["OriginStatus"])
}
method := scalarString(obj["RequestMethod"])
if status == "" && method == "" {
return nil
}
r := &fileRec{t: now, sev: 6}
if t, ok := parseLogTime(firstNonEmpty(scalarString(obj["StartUTC"]), scalarString(obj["time"])), now); ok {
r.t, r.dated = t, true
}
if n, ok := statusLevel(status); ok {
r.sev = n
}
rename := map[string]string{
"ClientHost": "client_ip", "ClientUsername": "http_user", "RequestMethod": "http_method",
"RequestHost": "http_host", "RequestPath": "http_path", "RequestProtocol": "http_proto",
"DownstreamStatus": "http_status", "DownstreamContentSize": "http_size", "OriginStatus": "origin_status",
"RouterName": "router", "ServiceName": "service", "ServiceURL": "backend", "entryPointName": "entrypoint",
"request_User-Agent": "user_agent", "request_Referer": "http_referer",
}
dur := ""
if d, err := strconv.ParseFloat(scalarString(obj["Duration"]), 64); err == nil {
dur = strconv.FormatFloat(d/1e6, 'f', -1, 64)
r.set("duration_ms", dur)
}
for _, k := range []string{"Duration", "StartUTC", "StartLocal", "time", "level", "msg"} {
delete(obj, k)
}
for k, v := range obj {
if to, ok := rename[k]; ok {
r.set(to, scalarString(v))
delete(obj, k)
}
}
flattenJSON("", obj, r)
// "GET example.com/path 404 · 12.5 ms · router@docker"
var b strings.Builder
b.WriteString(firstNonEmpty(method, "-"))
b.WriteByte(' ')
b.WriteString(r.fields["http_host"])
b.WriteString(r.fields["http_path"])
b.WriteByte(' ')
b.WriteString(firstNonEmpty(status, "-"))
if dur != "" {
b.WriteString(" · " + dur + " ms")
}
if rt := r.fields["router"]; rt != "" {
b.WriteString(" · " + rt)
}
if ip := r.fields["client_ip"]; ip != "" {
b.WriteString(" · " + ip)
}
r.msg = b.String()
return r
}
+660
View File
@@ -0,0 +1,660 @@
package main
import (
"bytes"
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"regexp"
"sort"
"strings"
"sync"
"time"
)
// fileSourceDef is one log file (or glob) declared in Settings > Sources.
type fileSourceDef struct {
ID string `json:"id"`
Name string `json:"name"` // app of the entries
Path string `json:"path"` // file, or glob (/logs/traefik/*.log)
Format string `json:"format"` // text, json, logfmt, regex, clf, traefik
Regex string `json:"regex,omitempty"` // format regex: named groups time, level, msg, host, app…
Host string `json:"host,omitempty"` // host of the entries (empty: the host name of the machine)
Enabled bool `json:"enabled"`
}
// filesConfig is saved in /data/files.json.
type filesConfig struct {
Sources []fileSourceDef `json:"sources"`
}
// filePos is where reading resumes in a file: its inode and an offset before
// which every entry was sent. Until: end of the history found when the
// source was turned on; entries before it are kept only when dated within
// FILES_BACKFILL.
type filePos struct {
Ino uint64 `json:"ino"`
Off int64 `json:"off"`
Until int64 `json:"until,omitempty"`
}
type filesState struct {
Pos map[string]filePos `json:"pos"` // key: source id + "|" + path
Started map[string]time.Time `json:"started"` // when each source was turned on
}
type fileSourceStatus struct {
Files int
Read uint64
Code string
Error string
}
// fileSource is a definition with its parser.
type fileSource struct {
fileSourceDef
parser *fileParser
}
// pendingEntry is an entry waiting for its continuation lines.
type pendingEntry struct {
rec *fileRec
off int64 // offset of its first line
seen time.Time // when its last line was read
history bool // read from the history found when the source was turned on
lines int
}
// Files follows log files of other programs (Traefik, nginx, applications…)
// mounted in the container, one entry per line in the format chosen for each
// file. Source type "file".
type Files struct {
sink func(*Entry)
hostName string
backfill time.Duration
cfgPath string
statePath string
wake chan struct{}
// Owned by the Run goroutine.
st filesState
readOff map[string]int64
pending map[string]*pendingEntry
dirty bool
mu sync.Mutex
cfg []*fileSource
resets map[string]bool // sources to read again from FILES_BACKFILL ago
status map[string]*fileSourceStatus
}
func NewFiles(hostRoot, dataDir string, backfill time.Duration, sink func(*Entry)) *Files {
f := &Files{
sink: sink,
hostName: readTrim(filepath.Join(hostRoot, "etc/hostname")),
backfill: backfill,
cfgPath: filepath.Join(dataDir, "files.json"),
statePath: filepath.Join(dataDir, "files-state.json"),
wake: make(chan struct{}, 1),
readOff: map[string]int64{},
pending: map[string]*pendingEntry{},
resets: map[string]bool{},
status: map[string]*fileSourceStatus{},
}
if f.hostName == "" {
f.hostName = "localhost"
}
var cfg filesConfig
if b, err := os.ReadFile(f.cfgPath); err == nil {
_ = json.Unmarshal(b, &cfg)
}
for _, d := range cfg.Sources {
src, err := compileFileSource(d)
if err != nil {
logEvent("files", 4, "file %q ignored: %v", d.Name, err)
continue
}
f.cfg = append(f.cfg, src)
}
if b, err := os.ReadFile(f.statePath); err == nil {
_ = json.Unmarshal(b, &f.st)
}
if f.st.Pos == nil {
f.st.Pos = map[string]filePos{}
}
if f.st.Started == nil {
f.st.Started = map[string]time.Time{}
}
return f
}
const maxFileSources = 50
var fileIDRe = regexp.MustCompile(`^[0-9a-f]{1,32}$`)
// compileFileSource checks a definition and builds its parser.
func compileFileSource(d fileSourceDef) (*fileSource, error) {
d.Name = strings.TrimSpace(d.Name)
d.Path = strings.TrimSpace(d.Path)
d.Host = strings.TrimSpace(d.Host)
if d.Format == "" {
d.Format = fmtText
}
if d.Format != fmtRegex {
d.Regex = ""
}
switch {
case !fileIDRe.MatchString(d.ID):
return nil, errors.New("invalid id")
case d.Name == "" || len(d.Name) > 64:
return nil, errors.New("the name is required (64 characters at most)")
case !filepath.IsAbs(d.Path):
return nil, fmt.Errorf("%s: the path must be absolute (as mounted in the container)", d.Name)
case len(d.Host) > 253:
return nil, fmt.Errorf("%s: host name too long", d.Name)
}
d.Path = filepath.Clean(d.Path)
if _, err := filepath.Match(d.Path, ""); err != nil {
return nil, fmt.Errorf("%s: invalid path pattern: %v", d.Name, err)
}
p, err := newFileParser(d.Format, d.Regex)
if err != nil {
return nil, fmt.Errorf("%s: %v", d.Name, err)
}
return &fileSource{fileSourceDef: d, parser: p}, nil
}
func newFileID() string {
b := make([]byte, 6)
_, _ = rand.Read(b)
return hex.EncodeToString(b)
}
// Configure checks, saves and applies the list of files. Sources turned off,
// removed or given another path are read again from FILES_BACKFILL ago the
// next time they are on.
func (f *Files) Configure(cfg filesConfig) error {
if len(cfg.Sources) > maxFileSources {
return fmt.Errorf("%d files at most", maxFileSources)
}
var srcs []*fileSource
ids := map[string]bool{}
for _, d := range cfg.Sources {
if d.ID == "" {
d.ID = newFileID()
}
if ids[d.ID] {
return fmt.Errorf("duplicate id %s", d.ID)
}
ids[d.ID] = true
src, err := compileFileSource(d)
if err != nil {
return err
}
srcs = append(srcs, src)
}
saved := filesConfig{Sources: []fileSourceDef{}}
for _, s := range srcs {
saved.Sources = append(saved.Sources, s.fileSourceDef)
}
if err := writeJSONFile(f.cfgPath, saved); err != nil {
return err
}
f.mu.Lock()
old := map[string]*fileSource{}
for _, s := range f.cfg {
old[s.ID] = s
}
for _, s := range srcs {
if o := old[s.ID]; o != nil && ((o.Enabled && !s.Enabled) || o.Path != s.Path) {
f.resets[s.ID] = true
delete(f.status, s.ID)
}
delete(old, s.ID)
}
for id := range old { // removed
f.resets[id] = true
delete(f.status, id)
}
f.cfg = srcs
f.mu.Unlock()
select {
case f.wake <- struct{}{}:
default:
}
return nil
}
// Status is the state shown in Settings > Sources.
func (f *Files) Status() map[string]any {
f.mu.Lock()
defer f.mu.Unlock()
list := make([]map[string]any, 0, len(f.cfg))
for _, s := range f.cfg {
m := map[string]any{
"id": s.ID, "name": s.Name, "path": s.Path, "format": s.Format, "regex": s.Regex,
"host": s.Host, "enabled": s.Enabled, "files": 0, "read": 0, "code": "", "error": "",
}
if st := f.status[s.ID]; st != nil && s.Enabled {
m["files"], m["read"], m["code"], m["error"] = st.Files, st.Read, st.Code, st.Error
}
list = append(list, m)
}
return map[string]any{"sources": list, "defaultHost": f.hostName}
}
// Run polls the files every second.
func (f *Files) Run(ctx context.Context) {
tick := time.NewTicker(time.Second)
defer tick.Stop()
n := 0
for {
select {
case <-ctx.Done():
now := time.Now()
for key := range f.pending {
f.flushPending(key, now)
}
f.checkpointAll()
f.saveState()
return
case <-tick.C:
case <-f.wake:
}
f.poll(time.Now())
if n++; n%5 == 0 {
f.saveState()
}
}
}
func (f *Files) saveState() {
if !f.dirty {
return
}
f.dirty = false
if err := writeJSONFile(f.statePath, f.st); err != nil {
logEvent("files", 4, "saving positions: %v", err)
}
}
func posKey(id, path string) string { return id + "|" + path }
// forget drops the positions of a source.
func (f *Files) forget(id string) {
prefix := id + "|"
for key := range f.st.Pos {
if strings.HasPrefix(key, prefix) {
delete(f.st.Pos, key)
delete(f.readOff, key)
delete(f.pending, key)
}
}
delete(f.st.Started, id)
f.dirty = true
}
// poll reads what was added to the files of every source that is on.
func (f *Files) poll(now time.Time) {
f.mu.Lock()
srcs := f.cfg
resets := f.resets
f.resets = map[string]bool{}
f.mu.Unlock()
for id := range resets {
f.forget(id)
}
active := map[string]bool{}
for _, s := range srcs {
if !s.Enabled {
continue
}
active[s.ID] = true
f.pollSource(s, now)
}
// Sources removed or off: nothing kept but their positions (removed ones
// were forgotten above).
for key := range f.pending {
id, _, _ := strings.Cut(key, "|")
if !active[id] {
delete(f.pending, key)
}
}
f.checkpointAll()
}
const (
maxFileRead = 4 << 20 // per file and per poll
maxFilesPerGlob = 200
fileContinueDelay = 2 * time.Second // an entry waits this long for continuation lines
)
// rotatedName: copies left by logrotate (access.log.1, app.log.2.gz,
// app-20261005.log.gz…), not read when a glob matches them.
var rotatedName = regexp.MustCompile(`(\.\d+|\.gz|\.bz2|\.xz|\.zst|\.zip|\.lz4)$`)
// matches lists the files of a source.
func (s *fileSource) matches() ([]string, error) {
if !strings.ContainsAny(s.Path, "*?[") {
fi, err := os.Stat(s.Path)
if err != nil {
return nil, err
}
if !fi.Mode().IsRegular() {
return nil, fmt.Errorf("%s is not a regular file", s.Path)
}
return []string{s.Path}, nil
}
paths, err := filepath.Glob(s.Path)
if err != nil {
return nil, err
}
var out []string
for _, p := range paths {
if rotatedName.MatchString(p) {
continue
}
if fi, err := os.Stat(p); err == nil && fi.Mode().IsRegular() {
out = append(out, p)
}
}
sort.Strings(out)
if len(out) > maxFilesPerGlob {
out = out[:maxFilesPerGlob]
}
if len(out) == 0 {
// Tell a missing directory (not mounted) from an empty one.
if _, err := os.Stat(filepath.Dir(s.Path)); err != nil && !strings.ContainsAny(filepath.Dir(s.Path), "*?[") {
return nil, err
}
return nil, fs.ErrNotExist
}
return out, nil
}
func (f *Files) setStatus(id string, files int, err error) {
code, detail := "", ""
if err != nil {
code, detail = errCode(err), err.Error()
if errors.Is(err, fs.ErrNotExist) {
code = "not_found"
}
}
f.mu.Lock()
st := f.status[id]
if st == nil {
st = &fileSourceStatus{}
f.status[id] = st
}
changed := code != st.Code
st.Files, st.Code, st.Error = files, code, detail
name := id
for _, s := range f.cfg {
if s.ID == id {
name = s.Name
}
}
f.mu.Unlock()
if changed && code != "" && code != "not_found" {
logEvent("files", 4, "cannot read the file %s: %v", name, err)
}
}
func (f *Files) count(id string) {
f.mu.Lock()
if st := f.status[id]; st != nil {
st.Read++
}
f.mu.Unlock()
}
func (f *Files) pollSource(s *fileSource, now time.Time) {
first := false
if _, ok := f.st.Started[s.ID]; !ok {
f.st.Started[s.ID], f.dirty, first = now, true, true
}
paths, err := s.matches()
if err != nil {
f.setStatus(s.ID, 0, err)
return
}
prefix := s.ID + "|"
matched := map[string]bool{}
for _, p := range paths {
matched[posKey(s.ID, p)] = true
}
// Files that left the glob, by inode: a file renamed within it goes on
// from where it was.
gone := map[uint64]string{}
for key, pos := range f.st.Pos {
if strings.HasPrefix(key, prefix) && !matched[key] {
gone[pos.Ino] = key
}
}
var firstErr error
read := 0
for _, p := range paths {
if err := f.pollFile(s, p, first, gone, now); err != nil {
firstErr = keepFirst(firstErr, err)
continue
}
read++
}
for key := range f.st.Pos {
if strings.HasPrefix(key, prefix) && !matched[key] {
f.flushPending(key, now)
delete(f.st.Pos, key)
delete(f.readOff, key)
f.dirty = true
}
}
if read > 0 {
firstErr = nil
}
f.setStatus(s.ID, read, firstErr)
}
func (f *Files) pollFile(s *fileSource, path string, first bool, gone map[uint64]string, now time.Time) error {
key := posKey(s.ID, path)
fi, err := os.Stat(path)
if err != nil {
return err
}
ino := inodeOf(fi)
pos, known := f.st.Pos[key]
if !known {
if old, ok := gone[ino]; ok && ino != 0 {
// Renamed within the glob.
pos, known = f.st.Pos[old], true
if off, ok := f.readOff[old]; ok {
f.readOff[key] = off
}
if pe := f.pending[old]; pe != nil {
f.pending[key] = pe
}
delete(f.st.Pos, old)
delete(f.readOff, old)
delete(f.pending, old)
delete(gone, ino)
} else if first {
// Turned on: the last FILES_BACKFILL of the files already there
// (0: only new lines).
pos = filePos{Ino: ino}
if f.backfill > 0 {
pos.Until = fi.Size()
} else {
pos.Off = fi.Size()
}
} else {
pos = filePos{Ino: ino} // new file: read from its start
}
f.st.Pos[key], f.dirty = pos, true
}
off, ok := f.readOff[key]
if !ok {
off = pos.Off
}
if pos.Ino != ino {
// Rotated (access.log -> access.log.1): finish the old file, then
// read the new one from its start.
if old, err := os.Stat(path + ".1"); err == nil && inodeOf(old) == pos.Ino {
for {
next, more, err := f.readFrom(s, key, path+".1", off, pos.Until, now)
if err != nil || !more {
break
}
off = next
}
}
f.flushPending(key, now)
pos, off = filePos{Ino: ino}, 0
f.st.Pos[key], f.dirty = pos, true
} else if fi.Size() < off {
f.flushPending(key, now) // truncated (copytruncate)
pos.Until, off = 0, 0
f.st.Pos[key], f.dirty = pos, true
}
if fi.Size() > off {
next, _, err := f.readFrom(s, key, path, off, pos.Until, now)
if err != nil {
f.readOff[key] = off
return err
}
off = next
}
f.readOff[key] = off
if pe := f.pending[key]; pe != nil && now.Sub(pe.seen) >= fileContinueDelay {
f.flushPending(key, now)
}
return nil
}
// readFrom parses the complete lines written after off and returns the new
// offset; more tells whether the read window was full.
func (f *Files) readFrom(s *fileSource, key, path string, off, until int64, now time.Time) (int64, bool, error) {
fh, err := os.Open(path)
if err != nil {
return off, false, err
}
defer fh.Close()
buf, err := io.ReadAll(io.NewSectionReader(fh, off, maxFileRead))
if err != nil {
return off, false, err
}
full := len(buf) == maxFileRead
end := bytes.LastIndexByte(buf, '\n')
if end < 0 {
if !full {
return off, false, nil // partial line: wait for its end
}
end = len(buf) - 1 // line longer than the read window: cut it
}
start := off
for _, b := range bytes.Split(buf[:end], []byte{'\n'}) {
f.line(s, key, path, string(b), start, start < until, now)
start += int64(len(b)) + 1
}
return off + int64(end) + 1, full, nil
}
// line handles one line: a continuation of the pending entry, or a new one.
func (f *Files) line(s *fileSource, key, path, line string, off int64, history bool, now time.Time) {
if strings.TrimSpace(line) == "" {
return
}
pe := f.pending[key]
if s.parser.multiline() && pe != nil && s.parser.continuation(line) {
if pe.lines < maxContinuation && len(pe.rec.msg) < maxFileMsg {
pe.rec.msg += "\n" + strings.TrimRight(cleanLine(line), " \t")
pe.lines++
}
pe.seen = now
return
}
f.flushPending(key, now)
rec := s.parser.parse(line, now)
pe = &pendingEntry{rec: rec, off: off, seen: now, history: history}
if s.parser.multiline() {
f.pending[key] = pe
return
}
f.emit(s, path, pe, now)
}
func (f *Files) flushPending(key string, now time.Time) {
pe := f.pending[key]
if pe == nil {
return
}
delete(f.pending, key)
id, path, _ := strings.Cut(key, "|")
f.mu.Lock()
var src *fileSource
for _, s := range f.cfg {
if s.ID == id {
src = s
}
}
f.mu.Unlock()
if src != nil {
f.emit(src, path, pe, now)
}
}
// emit sends an entry; history entries are kept only when dated within
// FILES_BACKFILL of when the source was turned on.
func (f *Files) emit(s *fileSource, path string, pe *pendingEntry, now time.Time) {
r := pe.rec
if pe.history {
notBefore := f.st.Started[s.ID].Add(-f.backfill)
if !r.dated || r.t.Before(notBefore) {
return
}
}
if len(r.msg) > maxFileMsg {
r.msg = strings.ToValidUTF8(r.msg[:maxFileMsg], "") + "…"
}
extra := map[string]string{"log_file": path}
for k, v := range r.fields {
extra[k] = v
}
f.sink(&Entry{
Time: r.t,
Received: now,
Host: firstNonEmpty(r.host, s.Host, f.hostName),
App: firstNonEmpty(r.app, s.Name),
SevNum: r.sev,
Severity: severityNames[r.sev],
Message: r.msg,
Proto: "file",
SourceType: "file",
Extra: extra,
Wait: true, // the file keeps the lines: wait for room in the queue
})
f.count(s.ID)
}
// checkpointAll saves, for each file, an offset before which every entry is
// sent: the start of its pending entry, or the read offset.
func (f *Files) checkpointAll() {
for key, off := range f.readOff {
pos, ok := f.st.Pos[key]
if !ok {
continue
}
if pe := f.pending[key]; pe != nil && pe.off < off {
off = pe.off
}
if off != pos.Off {
pos.Off, f.dirty = off, true
f.st.Pos[key] = pos
}
}
}
+337
View File
@@ -0,0 +1,337 @@
package main
import (
"os"
"path/filepath"
"sort"
"strings"
"sync"
"testing"
"time"
)
func mustParser(t *testing.T, format, expr string) *fileParser {
t.Helper()
p, err := newFileParser(format, expr)
if err != nil {
t.Fatal(err)
}
return p
}
func TestParseLogTime(t *testing.T) {
now := time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC)
utc := func(s string) time.Time { t, _ := time.Parse(time.RFC3339Nano, s); return t }
tests := []struct {
in string
want time.Time
}{
{"2026-10-05T10:00:00.123Z", utc("2026-10-05T10:00:00.123Z")},
{"2026-10-05T10:00:00+02:00", utc("2026-10-05T08:00:00Z")},
{"2026-10-05 10:00:00,250 +0200", utc("2026-10-05T08:00:00.25Z")},
{"2026/10/05 10:00:00Z", utc("2026-10-05T10:00:00Z")},
{"05/Oct/2026:10:00:00 +0200", utc("2026-10-05T08:00:00Z")},
{"1759572000", time.Unix(1759572000, 0)},
{"1759572000123", time.UnixMilli(1759572000123)},
}
for _, tt := range tests {
got, ok := parseLogTime(tt.in, now)
if !ok || !got.Equal(tt.want) {
t.Errorf("%q: got %v %v, want %v", tt.in, got, ok, tt.want)
}
}
// Syslog timestamps: no year, the last one before tomorrow.
if got, ok := parseLogTime("Dec 31 23:00:00", now); !ok || got.Year() != 2025 {
t.Errorf("Dec 31: %v %v", got, ok)
}
if got, ok := parseLogTime("Oct 5 11:00:00", now); !ok || got.Year() != 2026 || got.Day() != 5 {
t.Errorf("Oct 5: %v %v", got, ok)
}
for _, bad := range []string{"", "hello", "200", "12345"} {
if _, ok := parseLogTime(bad, now); ok {
t.Errorf("%q parsed", bad)
}
}
}
func TestParseText(t *testing.T) {
now := time.Now()
p := mustParser(t, fmtText, "")
r := p.parse("2026-10-05 10:00:00.5 ERROR db: connection refused", now)
if !r.dated || r.msg != "ERROR db: connection refused" || r.sev != 3 {
t.Errorf("got %+v", r)
}
r = p.parse("[2026-10-05T10:00:00Z] - started", now)
if !r.dated || r.msg != "started" || r.sev != 6 {
t.Errorf("got %+v", r)
}
r = p.parse("\x1b[33mWARN\x1b[0m no timestamp here", now)
if r.dated || r.msg != "WARN no timestamp here" || r.sev != 4 || !r.t.Equal(now) {
t.Errorf("got %+v", r)
}
}
func TestParseJSON(t *testing.T) {
now := time.Now()
p := mustParser(t, fmtJSON, "")
r := p.parse(`{"time":"2026-10-05T10:00:00Z","level":"warn","msg":"slow query","host":"db.example","req":{"id":7,"tags":["a"]},"ok":true}`, now)
if !r.dated || r.sev != 4 || r.msg != "slow query" {
t.Errorf("got %+v", r)
}
for k, want := range map[string]string{"field_host": "db.example", "req.id": "7", "req.tags": `["a"]`, "ok": "true"} {
if r.fields[k] != want {
t.Errorf("%s = %q, want %q", k, r.fields[k], want)
}
}
// pino: numeric level, epoch in ms.
r = p.parse(`{"level":50,"time":1759572000123,"msg":"boom"}`, now)
if r.sev != 3 || !r.t.Equal(time.UnixMilli(1759572000123)) {
t.Errorf("got %+v", r)
}
// No message key: the whole line; not JSON: text.
line := `{"event":"login","user":"bob"}`
if r = p.parse(line, now); r.msg != line || r.fields["user"] != "bob" {
t.Errorf("got %+v", r)
}
if r = p.parse("plain ERROR line", now); r.msg != "plain ERROR line" || r.sev != 3 {
t.Errorf("got %+v", r)
}
}
func TestParseLogfmt(t *testing.T) {
p := mustParser(t, fmtLogfmt, "")
r := p.parse(`time="2026-10-05T10:00:00Z" level=error msg="Cannot connect: \"docker\" down" providerName=docker flag`, time.Now())
if !r.dated || r.sev != 3 || r.msg != `Cannot connect: "docker" down` || r.fields["providerName"] != "docker" {
t.Errorf("got %+v", r)
}
}
func TestParseRegex(t *testing.T) {
if _, err := newFileParser(fmtRegex, "("); err == nil {
t.Error("invalid regex accepted")
}
if _, err := newFileParser(fmtRegex, ""); err == nil {
t.Error("empty regex accepted")
}
p := mustParser(t, fmtRegex, `^(?P<time>\S+ \S+) \[(?P<level>\w+)\] (?P<module>[\w.]+): (?P<msg>.*)$`)
r := p.parse("2026-10-05 10:00:00 [warning] app.db: pool exhausted", time.Now())
if !r.dated || r.sev != 4 || r.msg != "pool exhausted" || r.fields["module"] != "app.db" {
t.Errorf("got %+v", r)
}
if !p.continuation("Traceback (most recent call last):") || p.continuation("2026-10-05 10:00:01 [info] a: b") {
t.Error("continuation")
}
}
func TestParseCLF(t *testing.T) {
p := mustParser(t, fmtCLF, "")
r := p.parse(`203.0.113.9 - - [05/Oct/2026:10:00:00 +0200] "GET /api/x?y=1 HTTP/1.1" 502 157 "-" "curl/8.0" 4211 "api@docker" "http://172.18.0.5:8080" 31ms`, time.Now())
if !r.dated || r.sev != 3 || !strings.HasPrefix(r.msg, "203.0.113.9 - - [") {
t.Errorf("got %+v", r)
}
for k, want := range map[string]string{"client_ip": "203.0.113.9", "http_method": "GET", "http_path": "/api/x?y=1", "http_status": "502",
"user_agent": "curl/8.0", "router": "api@docker", "backend": "http://172.18.0.5:8080", "duration_ms": "31"} {
if r.fields[k] != want {
t.Errorf("%s = %q, want %q", k, r.fields[k], want)
}
}
if _, ok := r.fields["http_referer"]; ok {
t.Error("empty referer kept")
}
// nginx combined, 404.
r = p.parse(`10.0.0.1 - alice [05/Oct/2026:10:00:00 +0000] "POST /login HTTP/2.0" 404 0 "https://example.com/" "Mozilla/5.0"`, time.Now())
if r.sev != 4 || r.fields["http_user"] != "alice" || r.fields["http_referer"] != "https://example.com/" {
t.Errorf("got %+v", r)
}
}
func TestParseTraefikJSON(t *testing.T) {
p := mustParser(t, fmtTraefik, "")
r := p.parse(`{"ClientHost":"203.0.113.9","DownstreamStatus":200,"Duration":12500000,"RequestHost":"app.example.com","RequestMethod":"GET","RequestPath":"/","RequestProtocol":"HTTP/2.0","RouterName":"app@docker","StartUTC":"2026-10-05T10:00:00.1Z","entryPointName":"websecure","level":"info","msg":"","time":"2026-10-05T10:00:00Z","TLSVersion":"1.3"}`, time.Now())
if !r.dated || r.sev != 6 || r.msg != "GET app.example.com/ 200 · 12.5 ms · app@docker · 203.0.113.9" {
t.Errorf("got %+v", r)
}
for k, want := range map[string]string{"entrypoint": "websecure", "http_status": "200", "TLSVersion": "1.3", "duration_ms": "12.5"} {
if r.fields[k] != want {
t.Errorf("%s = %q, want %q", k, r.fields[k], want)
}
}
// Traefik's own logs in the same file: generic JSON.
r = p.parse(`{"level":"error","msg":"Provider error","time":"2026-10-05T10:00:00Z"}`, time.Now())
if r.sev != 3 || r.msg != "Provider error" {
t.Errorf("got %+v", r)
}
}
func TestFileSourceValidation(t *testing.T) {
f := NewFiles(t.TempDir(), t.TempDir(), time.Hour, func(*Entry) {})
bad := []fileSourceDef{
{Name: "", Path: "/logs/a.log"},
{Name: "a", Path: "relative.log"},
{Name: "a", Path: "/logs/[.log"},
{Name: "a", Path: "/logs/a.log", Format: "xml"},
{Name: "a", Path: "/logs/a.log", Format: fmtRegex, Regex: "(?P<x"},
}
for _, d := range bad {
if err := f.Configure(filesConfig{Sources: []fileSourceDef{d}}); err == nil {
t.Errorf("%+v accepted", d)
}
}
if err := f.Configure(filesConfig{Sources: []fileSourceDef{{Name: " traefik ", Path: "/logs/traefik/*.log", Format: fmtCLF}}}); err != nil {
t.Fatal(err)
}
src := f.Status()["sources"].([]map[string]any)
if len(src) != 1 || src[0]["name"] != "traefik" || src[0]["id"] == "" {
t.Errorf("got %+v", src)
}
}
type fileSink struct {
mu sync.Mutex
e []*Entry
}
func (s *fileSink) add(e *Entry) { s.mu.Lock(); s.e = append(s.e, e); s.mu.Unlock() }
func (s *fileSink) take() []*Entry {
s.mu.Lock()
defer s.mu.Unlock()
e := s.e
s.e = nil
return e
}
func appendFile(t *testing.T, path, s string) {
t.Helper()
fh, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644)
if err != nil {
t.Fatal(err)
}
if _, err := fh.WriteString(s); err != nil {
t.Fatal(err)
}
fh.Close()
}
func msgs(es []*Entry) []string {
var out []string
for _, e := range es {
out = append(out, e.Message)
}
return out
}
func TestFilesFollow(t *testing.T) {
dir, data := t.TempDir(), t.TempDir()
logs := filepath.Join(dir, "logs")
os.Mkdir(logs, 0o755)
now := time.Now()
old := now.Add(-3 * time.Hour).UTC().Format(time.RFC3339)
recent := now.Add(-10 * time.Minute).UTC().Format(time.RFC3339)
app := filepath.Join(logs, "app.log")
appendFile(t, app, old+" INFO too old\n"+recent+" ERROR recent\n at line 1\nundated history line\n")
appendFile(t, filepath.Join(logs, "app.log.1"), recent+" INFO rotated copy\n")
sink := &fileSink{}
f := NewFiles(dir, data, time.Hour, sink.add)
if err := f.Configure(filesConfig{Sources: []fileSourceDef{{Name: "myapp", Path: filepath.Join(logs, "*.log"), Format: fmtText, Host: "web1", Enabled: true}}}); err != nil {
t.Fatal(err)
}
f.poll(now)
f.poll(now.Add(3 * time.Second)) // the pending entry waited for continuation lines
got := sink.take()
if len(got) != 1 || got[0].Message != "ERROR recent\n at line 1" || got[0].SevNum != 3 || got[0].App != "myapp" ||
got[0].Host != "web1" || got[0].SourceType != "file" || got[0].Extra["log_file"] != app {
t.Fatalf("history: got %q %+v", msgs(got), got)
}
// New lines, a new file in the glob, then a rotation.
appendFile(t, app, "plain new line\n\tcontinued\n")
appendFile(t, filepath.Join(logs, "other.log"), "first of other\n")
f.poll(now.Add(4 * time.Second))
appendFile(t, app, "after\n")
f.poll(now.Add(5 * time.Second))
f.poll(now.Add(8 * time.Second))
got = sink.take()
m := msgs(got)
sort.Strings(m)
if strings.Join(m, "|") != "after|first of other|plain new line\n\tcontinued" {
t.Fatalf("new lines: got %q", msgs(got))
}
appendFile(t, app, "before rotation\n")
if err := os.Rename(app, app+".1"); err != nil {
t.Fatal(err)
}
appendFile(t, app, "after rotation\n")
f.poll(now.Add(9 * time.Second))
f.poll(now.Add(12 * time.Second))
if got = sink.take(); strings.Join(msgs(got), "|") != "before rotation|after rotation" {
t.Fatalf("rotation: got %q", msgs(got))
}
// Restart: resumes where it stopped, nothing sent twice.
f.checkpointAll()
f.dirty = true
f.saveState()
appendFile(t, app, "while stopped\n")
f2 := NewFiles(dir, data, time.Hour, sink.add)
f2.poll(now.Add(13 * time.Second))
f2.poll(now.Add(16 * time.Second))
if got = sink.take(); strings.Join(msgs(got), "|") != "while stopped" {
t.Fatalf("restart: got %q", msgs(got))
}
// Truncated in place (copytruncate).
os.Truncate(app, 0)
appendFile(t, app, "x\n")
f2.poll(now.Add(17 * time.Second))
f2.poll(now.Add(20 * time.Second))
if got = sink.take(); strings.Join(msgs(got), "|") != "x" {
t.Fatalf("truncate: got %q", msgs(got))
}
st := f2.Status()["sources"].([]map[string]any)[0]
if st["files"] != 2 || st["code"] != "" {
t.Errorf("status %+v", st)
}
}
func TestFilesJSONAndErrors(t *testing.T) {
dir, data := t.TempDir(), t.TempDir()
path := filepath.Join(dir, "access.json")
sink := &fileSink{}
f := NewFiles(dir, data, 0, sink.add) // FILES_BACKFILL=0: only new lines
appendFile(t, path, `{"msg":"existing"}`+"\n")
cfg := filesConfig{Sources: []fileSourceDef{
{ID: "a1", Name: "api", Path: path, Format: fmtJSON, Enabled: true},
{ID: "b2", Name: "missing", Path: filepath.Join(dir, "nope.log"), Enabled: true},
}}
if err := f.Configure(cfg); err != nil {
t.Fatal(err)
}
now := time.Now()
f.poll(now)
appendFile(t, path, `{"msg":"one","level":"error"}`+"\n"+`{"msg":"two"}`+"\n"+`{"msg":"partial`)
f.poll(now.Add(time.Second))
got := sink.take()
if strings.Join(msgs(got), "|") != "one|two" || got[0].SevNum != 3 {
t.Fatalf("got %q", msgs(got))
}
src := f.Status()["sources"].([]map[string]any)
if src[0]["files"] != 1 || src[1]["code"] != "not_found" {
t.Errorf("status %+v", src)
}
// Turned off then on: starts over (only new lines here).
cfg.Sources[0].Enabled = false
f.Configure(cfg)
f.poll(now.Add(2 * time.Second))
appendFile(t, path, `"}`+"\n")
cfg.Sources[0].Enabled = true
f.Configure(cfg)
f.poll(now.Add(3 * time.Second))
appendFile(t, path, `{"msg":"three"}`+"\n")
f.poll(now.Add(4 * time.Second))
if got = sink.take(); strings.Join(msgs(got), "|") != "three" {
t.Fatalf("re-enabled: got %q", msgs(got))
}
}
+6
View File
@@ -43,6 +43,7 @@ type config struct {
auditLog string
auditHost string
auditFill time.Duration
filesFill time.Duration
batchSize int
queueSize int
flushEvery time.Duration
@@ -125,6 +126,7 @@ func main() {
auditLog: getenv("AUDIT_LOG", "/var/log/audit/audit.log"),
auditHost: os.Getenv("AUDIT_HOST"),
auditFill: getenvDuration("AUDIT_BACKFILL", time.Hour),
filesFill: getenvDuration("FILES_BACKFILL", time.Hour),
batchSize: getenvInt("BATCH_SIZE", 1000),
queueSize: getenvInt("QUEUE_SIZE", 100000),
flushEvery: time.Duration(getenvInt("FLUSH_MS", 1000)) * time.Millisecond,
@@ -218,6 +220,10 @@ func main() {
// Linux audit log of the host (auditd), off until enabled in Settings > Sources.
api.audit = NewAudit(cfg.auditLog, cfg.hostRoot, cfg.auditHost, cfg.dataDir, cfg.auditFill, sink)
go api.audit.Run(ctx)
// Log files of other programs (Traefik, nginx…) mounted in the container,
// declared in Settings > Sources.
api.files = NewFiles(cfg.hostRoot, cfg.dataDir, cfg.filesFill, sink)
go api.files.Run(ctx)
api.selfLog = selfLog
api.Routes(mux)
mux.Handle("GET /", http.FileServer(http.FS(static)))
+2 -2
View File
@@ -119,10 +119,10 @@ func (f Filter) filterExpr() string {
parts = append(parts, "app:="+strconv.Quote(f.App))
}
switch f.Source {
case "docker", "host", "kernel", "audit", "logstream":
case "docker", "host", "kernel", "audit", "logstream", "file":
parts = append(parts, `source_type:=`+strconv.Quote(f.Source))
case "syslog": // also matches logs stored before source_type existed
parts = append(parts, `!(source_type:in("docker","host","kernel","audit","logstream"))`)
parts = append(parts, `!(source_type:in("docker","host","kernel","audit","logstream","file"))`)
}
if f.Severity >= 0 && f.Severity < 7 {
names := make([]string, 0, 8)
+1 -1
View File
@@ -36,7 +36,7 @@ type Entry struct {
Proto string // udp or tcp
HostIP string // original host value when it was an IP resolved through DNS
SourceType string // "syslog", "docker", "host", "kernel", "audit" or "logstream"
SourceType string // "syslog", "docker", "host", "kernel", "audit", "file" or "logstream"
Extra map[string]string // additional fields (docker: container, image, …)
// Not stored. Wait: the producer can be slowed down when the queue is full
+158 -2
View File
@@ -78,6 +78,28 @@ const I18N = {
sourceAria: 'Source', srcAll: 'All sources', tabSources: 'Log sources',
srcHost: 'Host system',
srcKernel: 'Kernel (dmesg)',
srcFile: 'Files',
filesTitle: 'Log files',
addFile: '+ Add a file',
filesNone: 'No file yet: add one to follow the logs of Traefik, nginx or any other program.',
fileName: 'Name (app)',
filePath: '/logs/traefik/*.log',
fileHost: 'Host',
fileRegex: '(?P<time>\\S+ \\S+) \\[(?P<level>\\w+)\\] (?P<msg>.*)',
fileEnabledAria: 'Follow this file',
fmt_text: 'Plain text',
fmt_json: 'JSON lines',
fmt_logfmt: 'logfmt (key=value)',
fmt_clf: 'Access log (CLF)',
fmt_traefik: 'Traefik access (JSON)',
fmt_regex: 'Regular expression',
fileOff: 'Off.',
fileDraft: 'Fill in the name and the path to save this file.',
fileOk: ({ f, r }) => `${f} file${f > 1 ? 's' : ''} followed: ${r} entries since startup.`,
file_not_found: 'No file at this path: mount its directory in the container (see docker-compose.yml) and check the path. ',
file_permission: 'Permission denied: the file must be readable by UID 10001 or the adm group (see the README). ',
file_read: 'Cannot read the file: ',
filesHelp: 'Follows the log files of other programs, mounted read-only in the container (e.g. /var/log/traefik:/logs/traefik:ro). The path can use wildcards (/logs/traefik/*.log follows every .log file of the directory, also those created later); rotated copies (.1, .gz) are left out. Each line gives an entry (badge FILE, app = the name): plain text (level and leading timestamp detected, indented lines added to the previous entry), JSON lines, logfmt, access logs of nginx, Apache or Traefik (5xx are errors, 4xx warnings) or a regular expression whose named groups time, level, msg, host and app fill these fields, the other groups giving extra fields. When turned on, the last hour of dated lines is read first (FILES_BACKFILL); rotations are followed and reading resumes where it stopped.',
auditTitle: 'Linux audit (auditd)',
auditEnabled: 'Collect the audit events of this machine',
auditPath: 'File',
@@ -278,6 +300,28 @@ const I18N = {
sourceAria: 'Source', srcAll: 'Toutes les sources', tabSources: 'Sources de logs',
srcHost: 'Système hôte',
srcKernel: 'Noyau (dmesg)',
srcFile: 'Fichiers',
filesTitle: 'Fichiers de logs',
addFile: '+ Ajouter un fichier',
filesNone: 'Aucun fichier pour l\'instant : ajoutez-en un pour suivre les logs de Traefik, nginx ou de tout autre programme.',
fileName: 'Nom (app)',
filePath: '/logs/traefik/*.log',
fileHost: 'Hôte',
fileRegex: '(?P<time>\\S+ \\S+) \\[(?P<level>\\w+)\\] (?P<msg>.*)',
fileEnabledAria: 'Suivre ce fichier',
fmt_text: 'Texte brut',
fmt_json: 'JSON lines',
fmt_logfmt: 'logfmt (clé=valeur)',
fmt_clf: 'Journal d\'accès (CLF)',
fmt_traefik: 'Accès Traefik (JSON)',
fmt_regex: 'Expression régulière',
fileOff: 'Désactivé.',
fileDraft: 'Renseignez le nom et le chemin pour enregistrer ce fichier.',
fileOk: ({ f, r }) => `${f} fichier${f > 1 ? 's' : ''} suivi${f > 1 ? 's' : ''} : ${r} entrées depuis le démarrage.`,
file_not_found: 'Aucun fichier à ce chemin : montez son dossier dans le conteneur (voir docker-compose.yml) et vérifiez le chemin. ',
file_permission: 'Accès refusé : le fichier doit être lisible par l\'UID 10001 ou le groupe adm (voir le README). ',
file_read: 'Lecture du fichier impossible : ',
filesHelp: 'Suit les fichiers de logs d\'autres programmes, montés en lecture seule dans le conteneur (par ex. /var/log/traefik:/logs/traefik:ro). Le chemin accepte les jokers (/logs/traefik/*.log suit tous les fichiers .log du dossier, y compris ceux créés plus tard) ; les copies de rotation (.1, .gz) sont ignorées. Chaque ligne donne une entrée (badge FILE, app = le nom) : texte brut (niveau et horodatage de début de ligne détectés, lignes indentées rattachées à l\'entrée précédente), JSON lines, logfmt, journaux d\'accès de nginx, Apache ou Traefik (5xx en erreur, 4xx en warning) ou expression régulière dont les groupes nommés time, level, msg, host et app remplissent ces champs, les autres groupes donnant des champs supplémentaires. À l\'activation, la dernière heure de lignes datées est lue d\'abord (FILES_BACKFILL) ; les rotations sont suivies et la lecture reprend où elle s\'était arrêtée.',
auditTitle: 'Audit Linux (auditd)',
auditEnabled: 'Collecter les événements d\'audit de cette machine',
auditPath: 'Fichier',
@@ -598,6 +642,7 @@ function setLang(next) {
renderDmesg();
renderAudit();
renderSelfLog();
renderFiles(true);
renderDocker();
}
@@ -907,7 +952,7 @@ function fmtTime(s) {
// Sources with their own look in the list: a colored edge and a badge
// before the app name (the severity badge stays the same).
const SRC_BADGE = { audit: 'AUDIT', logstream: 'LS' };
const SRC_BADGE = { audit: 'AUDIT', logstream: 'LS', file: 'FILE' };
function rowHTML(r, isNew) {
const sev = r.severity || '';
@@ -1025,7 +1070,7 @@ function updateCount(tookMs) {
}
function detailsHTML(r) {
const order = ['received', 'msg_time', '_time', 'host', 'host_name', 'host_ip', 'source_type', 'audit_type', 'audit_serial', 'auid', 'user', 'exe', 'res', 'key', 'container', 'image', 'compose_project', 'compose_service', 'stream', 'unit', 'log_file', 'subsystem', 'device', 'app', 'procid', 'severity', 'facility', 'source', 'proto', 'sd', '_msg'];
const order = ['received', 'msg_time', '_time', 'host', 'host_name', 'host_ip', 'source_type', 'audit_type', 'audit_serial', 'auid', 'user', 'exe', 'res', 'key', 'container', 'image', 'compose_project', 'compose_service', 'stream', 'unit', 'log_file', 'client_ip', 'http_method', 'http_host', 'http_path', 'http_status', 'duration_ms', 'router', 'subsystem', 'device', 'app', 'procid', 'severity', 'facility', 'source', 'proto', 'sd', '_msg'];
const hidden = new Set(['_stream_id', '_stream', 'sevnum', 'audit_raw']); // audit_raw: shown under the message
if (r.msg_time) hidden.add('_time'); // same as the reception time
const keys = order.filter((k) => r[k] != null && r[k] !== '' && !hidden.has(k))
@@ -2182,6 +2227,115 @@ async function configureSelfLog(change) {
$('#selfEnabled').addEventListener('change', (ev) => configureSelfLog({ enabled: ev.target.checked }));
$('#selfLevel').addEventListener('change', (ev) => configureSelfLog({ min_level: ev.target.value }));
/* ================= Log files ================= */
const FILE_FORMATS = ['text', 'json', 'logfmt', 'clf', 'traefik', 'regex'];
let filesState = null; // {sources, defaultHost} from the server
let fileDrafts = []; // rows added here, saved once they have a name and a path
let fileSeq = 0;
async function loadFiles() {
try { filesState = await api('/api/files'); } catch { filesState = null; }
renderFiles();
}
function fileStatus(s) {
if (s.draft) return [t('fileDraft'), 'muted'];
if (!s.enabled) return [t('fileOff'), 'muted'];
if (s.code) return [t('file_' + s.code) + (s.code === 'not_found' ? '' : s.error || ''), 'bad'];
if (s.files > 0) return [t('fileOk', { f: s.files, r: fmtNum(s.read) }), 'good'];
return [t('hostWaiting'), 'muted'];
}
function fileRowHTML(s) {
const key = s.id || s.draft;
const [text, cls] = fileStatus(s);
return `<div class="file-row${s.enabled ? '' : ' off'}" data-key="${esc(key)}">
<input type="checkbox" class="switch" data-f="enabled"${s.enabled ? ' checked' : ''} aria-label="${esc(t('fileEnabledAria'))}">
<input type="text" data-f="name" value="${esc(s.name)}" placeholder="${esc(t('fileName'))}" aria-label="${esc(t('fileName'))}" spellcheck="false" maxlength="64">
<input type="text" data-f="path" class="mono" value="${esc(s.path)}" placeholder="${esc(t('filePath'))}" aria-label="${esc(t('filePath'))}" spellcheck="false">
<select class="field" data-f="format">${FILE_FORMATS.map((f) => `<option value="${f}"${s.format === f ? ' selected' : ''}>${esc(t('fmt_' + f))}</option>`).join('')}</select>
<input type="text" data-f="host" value="${esc(s.host)}" placeholder="${esc(filesState?.defaultHost || t('fileHost'))}" aria-label="${esc(t('fileHost'))}" title="${esc(t('fileHost'))}" spellcheck="false">
<button class="icon-btn danger" type="button" data-del title="${esc(t('del'))}" aria-label="${esc(t('del'))}">
<svg viewBox="0 0 24 24"><path d="M4 7h16M10 11v6M14 11v6M6 7l1 13h10l1-13M9 7V4h6v3"/></svg>
</button>
<input type="text" data-f="regex" class="mono fr-regex" value="${esc(s.regex)}" placeholder="${esc(t('fileRegex'))}" aria-label="${esc(t('fmt_regex'))}" spellcheck="false"${s.format === 'regex' ? '' : ' hidden'}>
<p class="fr-status docker-status ${cls}">${esc(text)}</p>
</div>`;
}
// full: rebuild the rows even while one is being edited (language change).
function renderFiles(full) {
const list = $('#fileList');
if (!filesState) { list.innerHTML = ''; return; }
const rows = filesState.sources.concat(fileDrafts);
const keys = rows.map((s) => s.id || s.draft).join(',');
if (!full && list.dataset.keys === keys && list.contains(document.activeElement)) {
// Being edited: only refresh the status lines.
for (const s of rows) {
const p = list.querySelector(`.file-row[data-key="${CSS.escape(s.id || s.draft)}"] .fr-status`);
if (!p) continue;
const [text, cls] = fileStatus(s);
p.textContent = text;
p.className = 'fr-status docker-status ' + cls;
}
return;
}
list.dataset.keys = keys;
list.innerHTML = rows.map(fileRowHTML).join('') || `<p class="muted small">${esc(t('filesNone'))}</p>`;
}
function fileRowOf(key) {
return filesState?.sources.find((s) => s.id === key) || fileDrafts.find((s) => s.draft === key);
}
async function saveFiles() {
const ready = fileDrafts.filter((d) => d.name.trim() && d.path.trim());
const sources = filesState.sources.concat(ready).map(({ id, name, path, format, regex, host, enabled }) => ({ id: id || '', name, path, format, regex, host, enabled }));
try {
filesState = await api('/api/files', { method: 'PUT', body: { sources } });
fileDrafts = fileDrafts.filter((d) => !ready.includes(d));
} catch (e) {
toast(e.message);
await loadFiles();
return;
}
renderFiles();
setTimeout(loadFiles, 1500); // the files are read in the background
}
$('#fileList').addEventListener('change', (ev) => {
const row = ev.target.closest('.file-row');
const s = row && fileRowOf(row.dataset.key);
const f = ev.target.dataset.f;
if (!s || !f) return;
s[f] = f === 'enabled' ? ev.target.checked : ev.target.value.trim();
if (f === 'format') row.querySelector('.fr-regex').hidden = s.format !== 'regex';
if (f === 'format' && s.format === 'regex' && !s.regex) { row.querySelector('.fr-regex').focus(); return; }
if (s.draft && !(s.name && s.path)) return;
saveFiles();
});
$('#fileList').addEventListener('click', (ev) => {
const btn = ev.target.closest('[data-del]');
if (!btn) return;
const key = btn.closest('.file-row').dataset.key;
if (fileDrafts.some((d) => d.draft === key)) {
fileDrafts = fileDrafts.filter((d) => d.draft !== key);
renderFiles(true);
return;
}
filesState.sources = filesState.sources.filter((s) => s.id !== key);
saveFiles();
});
$('#addFile').addEventListener('click', () => {
if (!filesState) return;
fileDrafts.push({ draft: 'new' + (++fileSeq), name: '', path: '', format: 'text', regex: '', host: '', enabled: true });
renderFiles(true);
$('#fileList').lastElementChild.querySelector('[data-f="name"]').focus();
});
/* ================= Docker source ================= */
let dockerState = null;
@@ -2301,6 +2455,7 @@ setInterval(() => {
loadDmesg();
loadAudit();
loadSelfLog();
loadFiles();
}
}, 4000);
@@ -2597,6 +2752,7 @@ $('#settingsBtn').addEventListener('click', () => {
loadDmesg();
loadAudit();
loadSelfLog();
loadFiles();
loadDocker();
renderDbStats();
loadDbStats();
+10
View File
@@ -105,6 +105,7 @@
<option value="host" data-i18n="srcHost">Host system</option>
<option value="kernel" data-i18n="srcKernel">Kernel (dmesg)</option>
<option value="audit">Audit</option>
<option value="file" data-i18n="srcFile">Files</option>
<option value="logstream">LogStream</option>
</select>
<span class="spacer"></span>
@@ -312,6 +313,15 @@
<p class="muted small src-help" data-i18n="selfHelp"></p>
</section>
<section class="set-section span">
<h3 data-i18n="filesTitle">Log files</h3>
<div id="fileList" class="file-list"></div>
<div class="file-toolbar">
<button id="addFile" class="btn primary" type="button" data-i18n="addFile">+ Add a file</button>
</div>
<p class="muted small src-help" data-i18n="filesHelp"></p>
</section>
<section class="set-section span">
<h3 data-i18n="dockerTitle">Docker containers</h3>
<p id="dockerStatus" class="docker-status"></p>
+31 -2
View File
@@ -24,6 +24,7 @@
--chip-l: 38%; /* lightness of project-colored text (labels, Docker apps) */
--src-audit: #7c4dbd; /* audit events (auditd) */
--src-ls: #1f9aa3; /* LogStream internal logs */
--src-file: #5f8a1f; /* log files (Settings > Sources > Log files) */
--src-tint: 6%; /* background tint of their rows */
--sev-crit: #be123c;
@@ -75,6 +76,7 @@
--chip-l: 70%;
--src-audit: #a983e6;
--src-ls: #3cc3cc;
--src-file: #9ccc54;
--src-tint: 9%;
--sev-crit: #fb7185;
--sev-err: #f87171;
@@ -110,6 +112,8 @@
--chip-l: 70%;
--src-audit: #a983e6;
--src-ls: #3cc3cc;
--src-file: #9ccc54;
--src-file: #9ccc54;
--src-tint: 9%;
--sev-crit: #fb7185;
--sev-err: #f87171;
@@ -424,8 +428,9 @@ body.col-resizing, body.col-resizing * { cursor: col-resize !important; user-sel
one), light tint, badge before the app name. */
.row.src-audit { --src: var(--src-audit); }
.row.src-logstream { --src: var(--src-ls); }
.row.src-audit, .row.src-logstream { border-left-color: var(--src); background: color-mix(in srgb, var(--src) var(--src-tint), transparent); }
.row.src-audit:hover, .row.src-logstream:hover { background: color-mix(in srgb, var(--src) 13%, var(--row-hover)); }
.row.src-file { --src: var(--src-file); }
.row.src-audit, .row.src-logstream, .row.src-file { border-left-color: var(--src); background: color-mix(in srgb, var(--src) var(--src-tint), transparent); }
.row.src-audit:hover, .row.src-logstream:hover, .row.src-file:hover { background: color-mix(in srgb, var(--src) 13%, var(--row-hover)); }
.src-badge {
display: inline-block; margin-right: 5px; padding: 0 4px; border-radius: 4px; vertical-align: 1px;
font: 700 9.5px/1.45 var(--sans); letter-spacing: .04em;
@@ -751,11 +756,35 @@ input.switch:focus-visible { outline: 2px solid var(--accent); outline-offset: 2
.tag-row .icon-btn svg { width: 15px; height: 15px; }
.opt input { display: none; }
.opt:has(input:checked) { color: var(--accent); border-color: var(--accent); background: var(--accent-soft); }
/* Log files (Settings > Sources) */
.file-list { display: flex; flex-direction: column; gap: 4px; }
.file-row {
display: grid; align-items: center; gap: 6px;
grid-template-columns: auto minmax(7rem, 1fr) minmax(10rem, 2fr) minmax(9rem, 1.4fr) minmax(6rem, .8fr) 26px;
padding: 6px 6px 6px 10px; border: 1px solid var(--border); border-radius: 8px; background: var(--panel);
}
.file-row.off > input[type="text"], .file-row.off > select { opacity: .6; }
.file-row input[type="text"] {
height: 28px; padding: 0 8px; min-width: 0;
border: 1px solid var(--border); border-radius: 6px; background: var(--panel); font-size: 12.5px; outline: 0;
}
.file-row input.mono { font-family: var(--mono); }
.file-row input[type="text"]:focus { border-color: var(--accent); box-shadow: 0 0 0 3px var(--accent-soft); }
.file-row select.field { height: 28px; }
.file-row .icon-btn { width: 26px; height: 26px; border-radius: 6px; }
.file-row .icon-btn svg { width: 15px; height: 15px; }
.file-row .fr-regex, .file-row .fr-status { grid-column: 2 / -1; }
.file-row .fr-regex[hidden] { display: none; }
.file-row .fr-status { margin: 0 !important; font-size: 12px; }
.file-toolbar { margin-top: 10px; }
.tag-row .err { grid-column: 1 / -1; color: var(--sev-err); font-size: 12px; }
.tag-row .err:empty { display: none; }
/* ---------- Mobile ---------- */
@media (max-width: 760px) {
.file-row { grid-template-columns: auto minmax(0, 1fr) 26px; }
.file-row [data-del] { grid-column: 3; grid-row: 1; }
.file-row > [data-f="path"], .file-row > [data-f="format"], .file-row > [data-f="host"] { grid-column: 2 / -1; }
body { font-size: 13.5px; }
.topbar { flex-wrap: wrap; padding: 8px 16px; gap: 8px 10px; }
.search { order: 3; flex-basis: 100%; max-width: none; }