Source badge at the start of every log line

Each row now starts with a fixed-width badge naming its source, one color
per source: SYSLOG, DOCKER, KERNEL (dmesg), HOST (/var/log), AUDIT, FILE
and LS. The AUDIT/FILE/LS badges move there from before the app name; those
sources keep their colored edge and tint.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.env.example

@@ -1,6 +1,6 @@
 # Copy this file to .env and adjust it.
-SYSLOG_PORT=514
-HTTP_PORT=8080
+SYSLOG_PORT=5514
+HTTP_PORT=8989
 TZ=Europe/Paris
 # How long logs are kept (e.g. 7d, 30d, 12w, 1y)
 RETENTION=30d
@@ -67,3 +67,7 @@ DMESG_BACKFILL=1h
 AUDIT_BACKFILL=1h
 # Host name of the audit events (empty: node= of the records, else /host/etc/hostname, else localhost)
 AUDIT_HOST=
+# Log files of other programs (Traefik, nginx, applications…), declared in Settings > Log sources >
+# Log files: mount their directory read-only in docker-compose.yml (e.g. /var/log/traefik:/logs/traefik:ro).
+# History read when a file is turned on (e.g. 30m, 1h, 24h; 0 = only new lines)
+FILES_BACKFILL=1h

README.fr.md

@@ -92,6 +92,10 @@ une base propre.
 Les badges de sévérité `err`/`crit` et `warning` reprennent les couleurs des tags `error` et
 `warning`.
 
+Chaque ligne commence par un badge de largeur fixe qui indique sa source, avec une couleur par
+source : **SYSLOG**, **DOCKER**, **KERNEL** (dmesg), **HOST** (fichiers de `/var/log`), **AUDIT**,
+**FILE** et **LS** (logs de LogStream lui-même).
+
 Raccourcis : `/` place le curseur dans la recherche, `Esc` la vide. Un clic sur une ligne
 affiche tous ses champs.
 
@@ -278,7 +282,7 @@ source est **désactivée par défaut** : activez-la dans **Réglages > Sources
   ne peut pas être résolu), `exe`, `res` (`success` / `failed`, d'après `res=` ou `success=`) et
   `key` (clé de la règle). La sévérité est `warning` en cas d'échec, `info` sinon ; facility
   `authpriv` ; source `audit`. Dans la liste, les lignes d'audit ont un liseré gauche violet,
-  un fond légèrement teinté et un badge **AUDIT** devant l'app ; le filtre **Source** permet de
+  un fond légèrement teinté et le badge de source **AUDIT** ; le filtre **Source** permet de
   les afficher seules.
 - **Noms** : `user` vient du format enrichi (`log_format = ENRICHED`, le défaut des versions
   récentes d'auditd), sinon du `/etc/passwd` de l'hôte s'il est monté en `/host/etc/passwd`.
@@ -289,10 +293,71 @@ source est **désactivée par défaut** : activez-la dans **Réglages > Sources
   enregistrés dans `/data/audit-state.json` : après un redémarrage, la lecture reprend là ;
   après une rotation par auditd, la fin de `audit.log.1` est lue avant le nouveau fichier.
 
+## Fichiers de logs (Traefik, nginx, applications…)
+
+Logstream peut suivre les fichiers de logs d'autres programmes : Traefik, nginx, Apache, ou
+toute application qui écrit dans un fichier. Déclarez chaque fichier dans **Paramètres > Sources
+de logs > Fichiers de logs** (enregistré dans `/data/files.json`, 50 au plus) : un **nom** (l'**app**
+de ses entrées, par défaut le nom du fichier), le **chemin** du fichier tel que monté dans le
+conteneur, un **format** (*Automatique* par défaut), un **hôte** facultatif (par défaut
+`/host/etc/hostname`, sinon `localhost`) et un interrupteur.
+
+- **Montage** du dossier en lecture seule, par ex. dans `docker-compose.yml` :
+  `- /var/log/traefik:/logs/traefik:ro`, puis `docker compose up -d`. Les fichiers de `/var/log`
+  sont déjà accessibles sous `/host/var/log/…`. Les fichiers doivent être lisibles par l'UID
+  `10001` ou le groupe de `HOST_LOGS_GID` (`adm` par défaut) ; Paramètres > Sources de logs
+  indique quand aucun fichier ne correspond ou que l'accès est refusé.
+- **Chemin** : un fichier, par ex. `/logs/traefik/access.log` (pas de jokers). Dès la saisie, le
+  fichier est vérifié : trouvé ou non, lisible ou non, son format détecté et sa dernière ligne.
+- **Détection du format** (*Automatique*) : à partir des dernières lignes du fichier (50 au
+  plus), le format structuré que suivent au moins 80 % d'entre elles — accès Traefik JSON, JSON
+  lines, journal d'accès CLF, logfmt — sinon texte brut. Elle est refaite quand le format est
+  changé ; le format trouvé s'affiche dans l'état du fichier. Choisissez un autre format pour
+  le forcer (une expression régulière n'est jamais détectée).
+- **Formats** :
+  - *Texte brut* : une entrée par ligne ; le niveau est détecté comme pour les logs Docker, un
+    horodatage en début de ligne (ISO 8601, `2006/01/02 15:04:05`, `[02/Jan/2006:15:04:05 +0200]`,
+    `Jan  2 15:04:05`) devient l'heure du message et est retiré du message. Les lignes indentées
+    (stack traces) sont rattachées à l'entrée précédente.
+  - *JSON lines* : `msg` / `message`, `time` / `ts` / `@timestamp` (texte ou heure Unix) et
+    `level` (mot ou nombre, pino/bunyan compris) sont reconnus ; les autres clés deviennent des
+    champs, les objets imbriqués avec des noms pointés (`req.id`).
+  - *logfmt* : paires `clé=valeur`, comme les logs par défaut de Traefik ou des programmes Go ;
+    mêmes clés que le JSON.
+  - *Journal d'accès (CLF)* : Common / Combined Log Format de nginx et Apache, et le journal
+    d'accès de Traefik dans ce format (routeur, URL du serveur et durée compris). Champs
+    `client_ip`, `http_method`, `http_path`, `http_status`, `http_size`, `user_agent`,
+    `http_referer`, `router`, `backend`, `duration_ms`.
+  - *Accès Traefik (JSON)* : le journal d'accès JSON de Traefik (`accessLog.format: json`),
+    résumé en `GET app.example.com/ 200 · 12.5 ms · app@docker · 203.0.113.9`, toutes les clés
+    dans les champs. Les logs de Traefik lui-même dans le même fichier sont lus comme du JSON.
+  - *Expression régulière* (syntaxe RE2 / Go) : les groupes nommés `time`, `level`, `msg`,
+    `host` et `app` remplissent ces champs, `status` donne le niveau d'un journal d'accès, les
+    autres groupes deviennent des champs, par ex.
+    `^(?P<time>\S+ \S+) \[(?P<level>\w+)\] (?P<module>\S+): (?P<msg>.*)$`. Une ligne qui ne
+    correspond pas est rattachée à l'entrée précédente.
+
+  Pour les journaux d'accès, les statuts 5xx sont en `err`, les 4xx en `warning`, les autres en
+  `info`. Une ligne qui ne suit pas le format choisi est gardée comme du texte brut. Les champs
+  qui portent le nom d'un champ de LogStream (`host`, `app`…) sont renommés `field_host`,
+  `field_app`…
+- **Affichage** : source `file`, le chemin dans `log_file`. Dans la liste, ces lignes ont un
+  liseré gauche vert, une légère teinte et le badge de source **FILE** ; le filtre **Source**
+  les affiche seules.
+- À l'activation d'un fichier, les lignes de la dernière heure déjà présentes sont lues d'abord
+  (`FILES_BACKFILL`, `0` = seulement les nouvelles lignes ; les lignes sans horodatage sont alors
+  ignorées). Le début de cette heure est trouvé par dichotomie sur les dates des lignes : un gros
+  fichier qui contient des mois de logs n'est pas relu depuis son début ; un fichier créé ensuite
+  est lu depuis son début, jusqu'à 64 Mo par seconde, l'état indiquant ce qu'il reste à lire. L'inode et une
+  position sont enregistrés par fichier dans `/data/files-state.json` : après un redémarrage la
+  lecture reprend là ; après une rotation (`access.log` renommé `access.log.1`), la fin de
+  l'ancien fichier est lue avant le nouveau ; un fichier vidé sur place (`copytruncate`) est
+  relu depuis son début.
+
 ## Logs internes de LogStream
 
 Logstream enregistre ses propres événements comme des logs, avec l'**hôte** `logstream`,
-l'**app** du composant (`store`, `syslog`, `docker`, `host`, `dmesg`, `audit`, `main`) et la
+l'**app** du composant (`store`, `syslog`, `docker`, `host`, `dmesg`, `audit`, `files`, `main`) et la
 source `logstream` : VictoriaLogs injoignable puis de retour, lots perdus, lots refusés depuis
 le tampon disque, erreurs du listener syslog, API Docker ou flux d'un conteneur décrochés,
 sources qui ne peuvent pas lire, démarrage et arrêt. Ils sont actifs par défaut à partir du
@@ -327,8 +392,13 @@ En ligne de commande : `curl -o logs.csv 'localhost:8080/api/export.csv?q=error&
 
 ## Paramètres
 
-L'icône en forme d'engrenage ouvre les paramètres, organisés en onglets. Tout, sauf les tags de
-couleur, est mémorisé par navigateur.
+L'icône en forme d'engrenage ouvre les paramètres, organisés en onglets. Les modifications
+s'affichent tout de suite mais ne sont conservées qu'avec le bouton **Valider**, sous les libellés
+des onglets : il prend la couleur d'accent tant que des modifications sont en attente. **Annuler**
+revient aux paramètres enregistrés, et fermer la fenêtre avec des modifications non validées
+demande confirmation avant de les abandonner. Localisation et Affichage sont mémorisés par
+navigateur ; les tags de couleur et les sources de logs sont enregistrés sur le serveur. La purge
+et la réinitialisation des colonnes agissent immédiatement.
 
 - **Localisation et langue**
   - *Langue* : anglais ou français.
@@ -503,6 +573,7 @@ résolutions.
 | `AUDIT_LOG` | `/var/log/audit/audit.log` | chemin par défaut du journal d'audit Linux (modifiable dans Réglages > Sources) |
 | `AUDIT_BACKFILL` | `1h` | historique lu à l'activation de la source « audit Linux » |
 | `AUDIT_HOST` | vide | nom d'hôte des événements d'audit quand les enregistrements n'ont pas de `node=` |
+| `FILES_BACKFILL` | `1h` | historique lu à l'activation d'un fichier de logs (Paramètres > Sources de logs > Fichiers de logs) |
 | `TZ` | `Europe/Paris` | fuseau horaire des horodatages RFC 3164 (qui n'en portent pas) |
 | `BATCH_SIZE`, `FLUSH_MS`, `QUEUE_SIZE` | `1000`, `1000`, `100000` | réglage de l'ingestion |
 | `SPOOL_MAX_MB` | `1024` | taille du tampon disque des lots refusés par VictoriaLogs (`0` = pas de tampon : 15 s de tentatives, puis perte) |
@@ -565,6 +636,7 @@ Pour mettre à jour l'une d'elles :
 | `hostlogs.go`, `journal.go` | logs système de l'hôte : lecteur du journal systemd (sans `journalctl`) et suivi de `/var/log` |
 | `dmesg.go` | messages du noyau : lecture de `/dev/kmsg`, analyse des enregistrements, position par identifiant de démarrage et numéro de séquence |
 | `audit.go` | audit Linux : suivi de `audit.log` (rotation, position enregistrée), analyse des enregistrements, regroupement par événement, résumé |
+| `filesource.go`, `fileformats.go` | fichiers de logs : suivi (rotation, positions enregistrées, lignes de suite) et formats (détection, texte, JSON, logfmt, CLF, Traefik, expression régulière) |
 | `selflog.go` | logs internes de LogStream : `logEvent`, niveau minimal, limitation des répétitions, file bornée |
 | `tags.go` | stockage des tags de couleur |
 | `presets.go`, `presets.json` | préréglages de tags (`/api/presets`), liste intégrée |

README.md

@@ -83,6 +83,9 @@ versions before this change are indexed by their message timestamp; purge them f
 Settings to start clean.
 
 Severity badges `err`/`crit` and `warning` use the colors of the `error` and `warning` tags.
+Each row starts with a fixed-width badge naming its source, one color per source: **SYSLOG**,
+**DOCKER**, **KERNEL** (dmesg), **HOST** (files of `/var/log`), **AUDIT**, **FILE** and **LS**
+(LogStream's own logs).
 
 Shortcuts: `/` focuses the search box, `Esc` clears it. Clicking a row shows all its fields.
 
@@ -254,7 +257,7 @@ also be changed (saved in `/data/audit.json`; `AUDIT_LOG` sets the default path)
   `auid` (login user ID), `user` (its name, or the uid when it cannot be resolved), `exe`, `res`
   (`success` / `failed`, from `res=` or `success=`) and `key` (rule key). Severity is `warning`
   when the result is a failure, `info` otherwise; facility `authpriv`; source `audit`. In the
-  list, audit lines have a violet left edge, a light tint and an **AUDIT** badge before the app;
+  list, audit lines have a violet left edge, a light tint and the **AUDIT** source badge;
   the **Source** filter shows them alone.
 - **Names**: `user` comes from the enriched format (`log_format = ENRICHED`, the default of
   recent auditd versions), else from the host's `/etc/passwd` when it is mounted as
@@ -265,10 +268,65 @@ also be changed (saved in `/data/audit.json`; `AUDIT_LOG` sets the default path)
   `/data/audit-state.json`: after a restart, reading resumes there; after a rotation by auditd,
   the end of `audit.log.1` is read before the new file.
 
+## Log files (Traefik, nginx, applications…)
+
+Logstream can follow the log files of other programs: Traefik, nginx, Apache, or any application
+that writes to a file. Declare each file in **Settings > Log sources > Log files** (saved in
+`/data/files.json`, up to 50): a **name** (the **app** of its entries, by default the file name),
+the **path** of the file as mounted in the container, a **format** (*Automatic* by default), an
+optional **host** (default: `/host/etc/hostname`, else `localhost`) and a switch.
+
+- **Mount** the directory read-only, e.g. in `docker-compose.yml`:
+  `- /var/log/traefik:/logs/traefik:ro`, then `docker compose up -d`. Files under `/var/log`
+  are already reachable as `/host/var/log/…`. The files must be readable by UID `10001` or the
+  group of `HOST_LOGS_GID` (`adm` by default); Settings > Log sources says when no file matches
+  or access is denied.
+- **Path**: one file, e.g. `/logs/traefik/access.log` (no wildcards). As soon as it is typed, the
+  file is checked: found or not, readable or not, its detected format and its last line.
+- **Format detection** (*Automatic*): from the last lines of the file (50 at most), the
+  structured format that at least 80 % of them follow — Traefik access JSON, JSON lines, access
+  log CLF, logfmt — else plain text. It is done again when the format is changed; the format
+  found is shown in the status of the file. Choose another format to force it (a regular
+  expression is never detected).
+- **Formats**:
+  - *Plain text*: one entry per line; the level is detected as for Docker logs, a timestamp at
+    the start of the line (ISO 8601, `2006/01/02 15:04:05`, `[02/Jan/2006:15:04:05 +0200]`,
+    `Jan  2 15:04:05`) becomes the message time and is removed from the message. Indented
+    lines (stack traces) are added to the previous entry.
+  - *JSON lines*: `msg` / `message`, `time` / `ts` / `@timestamp` (text or Unix time) and
+    `level` (word or number, pino/bunyan included) are recognized; the other keys become
+    fields, nested objects with dotted names (`req.id`).
+  - *logfmt*: `key=value` pairs, as in Traefik's default logs or Go programs; same keys as JSON.
+  - *Access log (CLF)*: Common / Combined Log Format of nginx and Apache, and Traefik's access
+    log in this format (router, server URL and duration included). Fields `client_ip`,
+    `http_method`, `http_path`, `http_status`, `http_size`, `user_agent`, `http_referer`,
+    `router`, `backend`, `duration_ms`.
+  - *Traefik access (JSON)*: the JSON access log of Traefik (`accessLog.format: json`),
+    summarized as `GET app.example.com/ 200 · 12.5 ms · app@docker · 203.0.113.9`, every key in
+    the fields. Traefik's own logs in the same file are read as JSON lines.
+  - *Regular expression* (RE2 / Go syntax): the named groups `time`, `level`, `msg`, `host`
+    and `app` fill these fields, `status` gives the level of an access log, the other groups
+    become fields, e.g. `^(?P<time>\S+ \S+) \[(?P<level>\w+)\] (?P<module>\S+): (?P<msg>.*)$`.
+    A line that does not match is added to the previous entry.
+
+  For access logs, 5xx statuses are `err`, 4xx `warning`, the others `info`. A line that does
+  not fit the chosen format is kept as plain text. Fields named like LogStream's own ones
+  (`host`, `app`…) are renamed `field_host`, `field_app`…
+- **Display**: source `file`, the path in `log_file`. In the list these lines have a green left
+  edge, a light tint and the **FILE** source badge; the **Source** filter shows them alone.
+- When a file is turned on, the lines of the last hour already in it are read first
+  (`FILES_BACKFILL`, `0` = only new lines; lines without a timestamp are then left out). The
+  start of that hour is found by bisection on the dates of the lines, so a large file that holds
+  months of logs is not read from its start; a file created later is read from its start, up to
+  64 MB per second, the status showing what is left to read. The inode and an offset are saved per file in
+  `/data/files-state.json`: after a restart reading resumes there; after a rotation
+  (`access.log` renamed `access.log.1`), the end of the old file is read before the new one;
+  a file truncated in place (`copytruncate`) is read again from its start.
+
 ## LogStream internal logs
 
 Logstream records its own events as logs, with **host** `logstream`, **app** the component
-(`store`, `syslog`, `docker`, `host`, `dmesg`, `audit`, `main`) and the `logstream` source:
+(`store`, `syslog`, `docker`, `host`, `dmesg`, `audit`, `files`, `main`) and the `logstream` source:
 VictoriaLogs unreachable and back, lost batches, batches refused from the disk buffer, syslog
 listener errors, Docker API or container streams lost, sources that cannot read, start and
 stop. They are on by default from level **warning**; **Settings > Log sources > LogStream internal
@@ -298,8 +356,11 @@ From the command line: `curl -o logs.csv 'localhost:8080/api/export.csv?q=error&
 
 ## Settings
 
-The gear icon opens the settings, organized in tabs. Everything except color tags is
-remembered per browser.
+The gear icon opens the settings, organized in tabs. Changes show at once but are only kept
+with the **Save** button, under the tab labels: it takes the accent color while changes are
+waiting. **Cancel** goes back to the saved settings, and closing the window with unsaved changes
+asks before dropping them. Localization and Appearance are remembered per browser; color tags
+and log sources are saved on the server. Purging and resetting column widths act at once.
 
 - **Localization**
   - *Language*: English or French.
@@ -460,6 +521,7 @@ are only known by your router or a local DNS (Pi-hole, AdGuard, Unbound…), set
 | `AUDIT_LOG` | `/var/log/audit/audit.log` | default path of the Linux audit log (can be changed in Settings > Log sources) |
 | `AUDIT_BACKFILL` | `1h` | history read when the Linux audit source is turned on |
 | `AUDIT_HOST` | empty | host name of the audit events when the records carry no `node=` |
+| `FILES_BACKFILL` | `1h` | history read when a log file is turned on (Settings > Log sources > Log files) |
 | `TZ` | `Europe/Paris` | time zone for RFC 3164 timestamps (which carry none) |
 | `BATCH_SIZE`, `FLUSH_MS`, `QUEUE_SIZE` | `1000`, `1000`, `100000` | ingestion tuning |
 | `SPOOL_MAX_MB` | `1024` | disk buffer size for batches VictoriaLogs could not take (`0` = no buffer: retried for 15 s, then dropped) |
@@ -520,6 +582,7 @@ To update one of them:
 | `hostlogs.go`, `journal.go` | host system logs: systemd journal reader (no `journalctl`) and `/var/log` follower |
 | `dmesg.go` | kernel messages: `/dev/kmsg` reader, record parsing, position by boot ID and sequence number |
 | `audit.go` | Linux audit: `audit.log` follower (rotation, saved offset), record parsing, grouping by event, summary |
+| `filesource.go`, `fileformats.go` | log files: follower (rotation, saved offsets, continuation lines) and formats (detection, text, JSON, logfmt, CLF, Traefik, regular expression) |
 | `selflog.go` | LogStream internal logs: `logEvent`, minimum level, throttling, bounded queue |
 | `tags.go` | color tag storage |
 | `presets.go`, `presets.json` | color tag presets (`/api/presets`), built-in list |

api.go

@@ -25,6 +25,7 @@ type API struct {
 	host       *HostLogs
 	dmesg      *Dmesg
 	audit      *Audit
+	files      *Files
 	selfLog    *SelfLog
 	dbstats    dbStatsCache
 }
@@ -39,6 +40,8 @@ func (a *API) Routes(mux *http.ServeMux) {
 	mux.HandleFunc("GET /api/dbstats", a.dbStats)
 	mux.HandleFunc("GET /api/tags", a.listTags)
 	mux.HandleFunc("POST /api/tags", a.createTag)
+	mux.HandleFunc("PUT /api/tags", a.replaceTags)
+	mux.HandleFunc("GET /api/tags/defaults", a.defaultTags)
 	mux.HandleFunc("POST /api/tags/reset", a.resetTags)
 	mux.HandleFunc("GET /api/presets", a.listPresets)
 	mux.HandleFunc("PUT /api/tags/{id}", a.updateTag)
@@ -56,6 +59,9 @@ func (a *API) Routes(mux *http.ServeMux) {
 	mux.HandleFunc("PUT /api/dmesg", a.dmesgConfigure)
 	mux.HandleFunc("GET /api/audit", a.auditStatus)
 	mux.HandleFunc("PUT /api/audit", a.auditConfigure)
+	mux.HandleFunc("GET /api/files", a.filesStatus)
+	mux.HandleFunc("PUT /api/files", a.filesConfigure)
+	mux.HandleFunc("POST /api/files/detect", a.filesDetect)
 	mux.HandleFunc("GET /api/selflog", a.selfLogStatus)
 	mux.HandleFunc("PUT /api/selflog", a.selfLogConfigure)
 }
@@ -80,6 +86,41 @@ func (a *API) auditConfigure(w http.ResponseWriter, r *http.Request) {
 	writeJSON(w, http.StatusOK, a.audit.Status())
 }
 
+// GET /api/files: the log files of the "Files" source (Settings > Sources).
+func (a *API) filesStatus(w http.ResponseWriter, r *http.Request) {
+	writeJSON(w, http.StatusOK, a.files.Status())
+}
+
+// PUT /api/files {"sources": [{"id", "name", "path", "format" (auto or a format), "regex", "host", "enabled"}]}
+// (no id: a new file).
+func (a *API) filesConfigure(w http.ResponseWriter, r *http.Request) {
+	var cfg filesConfig
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 64<<10)).Decode(&cfg); err != nil {
+		writeErr(w, http.StatusBadRequest, err)
+		return
+	}
+	if err := a.files.Configure(cfg); err != nil {
+		writeErr(w, http.StatusBadRequest, err)
+		return
+	}
+	log.Printf("file sources changed from %s: %d files", r.RemoteAddr, len(cfg.Sources))
+	writeJSON(w, http.StatusOK, a.files.Status())
+}
+
+// POST /api/files/detect {"path": "/logs/traefik/access.log"}: whether the file
+// can be read, its detected format and its last line (Settings > Sources). A
+// POST, so that read-only accounts cannot read files through it.
+func (a *API) filesDetect(w http.ResponseWriter, r *http.Request) {
+	var req struct {
+		Path string `json:"path"`
+	}
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&req); err != nil {
+		writeErr(w, http.StatusBadRequest, err)
+		return
+	}
+	writeJSON(w, http.StatusOK, DetectFile(req.Path))
+}
+
 // GET /api/selflog: state of the LogStream internal logs (Settings > Sources).
 func (a *API) selfLogStatus(w http.ResponseWriter, r *http.Request) {
 	writeJSON(w, http.StatusOK, a.selfLog.Status())
@@ -492,6 +533,41 @@ func (a *API) purge(w http.ResponseWriter, r *http.Request) {
 	writeJSON(w, http.StatusOK, map[string]string{"task_id": id})
 }
 
+// PUT /api/tags [tags]: replaces the whole list. On an invalid tag, "index"
+// gives its position and nothing is saved.
+func (a *API) replaceTags(w http.ResponseWriter, r *http.Request) {
+	var in []Tag
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&in); err != nil {
+		writeErr(w, http.StatusBadRequest, err)
+		return
+	}
+	tags, err := a.tags.Replace(in)
+	var le *tagListError
+	switch {
+	case errors.As(err, &le):
+		body := map[string]any{"error": err.Error(), "index": le.index}
+		var ce *codedError
+		if errors.As(err, &ce) {
+			body["code"] = ce.code
+			if ce.detail != "" {
+				body["detail"] = ce.detail
+			}
+		}
+		writeJSON(w, http.StatusBadRequest, body)
+	case errors.Is(err, errTooManyTags):
+		writeErr(w, http.StatusBadRequest, err)
+	case err != nil:
+		writeErr(w, http.StatusInternalServerError, err)
+	default:
+		writeJSON(w, http.StatusOK, tags)
+	}
+}
+
+// GET /api/tags/defaults: the tags "Restore default tags" shows before saving.
+func (a *API) defaultTags(w http.ResponseWriter, r *http.Request) {
+	writeJSON(w, http.StatusOK, defaultTags())
+}
+
 func (a *API) resetTags(w http.ResponseWriter, r *http.Request) {
 	tags, err := a.tags.Reset()
 	if err != nil {

docker-compose.yml

@@ -7,9 +7,9 @@ services:
       - victorialogs
       - docker-proxy
     ports:
-      - "${SYSLOG_PORT:-514}:5514/udp"
-      - "${SYSLOG_PORT:-514}:5514/tcp"
-      - "${HTTP_PORT:-8080}:8080"
+      - "${SYSLOG_PORT:-5514}:5514/udp"
+      - "${SYSLOG_PORT:-5514}:5514/tcp"
+      - "${HTTP_PORT:-8989}:8080"
     environment:
       VLOGS_URL: http://victorialogs:9428
       SYSLOG_PUBLIC_PORT: ${SYSLOG_PORT:-514}     # le port d'ecoute syslog par defaut (attention aux ports <1024)
@@ -39,6 +39,7 @@ services:
       HOST_LOGS_BACKFILL: ${HOST_LOGS_BACKFILL:-1h}   # historique lu a l'activation des logs systeme de l'hote
       DMESG_BACKFILL: ${DMESG_BACKFILL:-1h}   # historique lu a l'activation des messages du noyau (dmesg)
       AUDIT_BACKFILL: ${AUDIT_BACKFILL:-1h}   # historique lu a l'activation de l'audit Linux (auditd)
+      FILES_BACKFILL: ${FILES_BACKFILL:-1h}   # historique lu a l'activation d'un fichier de logs (Reglages > Sources)
       AUDIT_HOST: ${AUDIT_HOST:-}             # nom d'hote des evenements d'audit (vide : node= des enregistrements, sinon /host/etc/hostname)
     group_add:
       - "${HOST_LOGS_GID:-4}"        # groupe autorise a lire les logs de l'hote (4 = adm sur Debian/Ubuntu)
@@ -63,6 +64,8 @@ services:
       # noms des utilisateurs et de l'hote pour les evenements d'audit (optionnel)
       # - /etc/passwd:/host/etc/passwd:ro
       # - /etc/hostname:/host/etc/hostname:ro
+      # fichiers de logs d'autres programmes (Traefik, nginx...), a declarer dans Reglages > Sources > Fichiers de logs
+      # - /var/log/traefik:/logs/traefik:ro
       # logo de la page de connexion, avec LOGIN_LOGO=/config/logo.png dans .env
       # - ./logo.png:/config/logo.png:ro
       # prereglages de tags personnalises, avec PRESETS_FILE=/config/presets.json dans .env

fileformats.go

@@ -0,0 +1,689 @@
+package main
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io"
+	"os"
+	"regexp"
+	"strconv"
+	"strings"
+	"time"
+	"unicode/utf8"
+)
+
+// Formats of the "Files" source (Settings > Sources), one per declared file.
+const (
+	fmtAuto    = "auto"    // detected from the last lines of the file
+	fmtText    = "text"    // one line per entry, level and leading timestamp detected
+	fmtJSON    = "json"    // JSON lines: msg/time/level recognized, other keys as fields
+	fmtLogfmt  = "logfmt"  // key=value pairs (Traefik "common" logs, Go programs…)
+	fmtRegex   = "regex"   // regular expression with named groups
+	fmtCLF     = "clf"     // access logs in Common/Combined Log Format (nginx, Apache, Traefik)
+	fmtTraefik = "traefik" // Traefik access log in JSON
+)
+
+var fileFormats = map[string]bool{fmtText: true, fmtJSON: true, fmtLogfmt: true, fmtRegex: true, fmtCLF: true, fmtTraefik: true}
+
+// fileRec is one parsed line (or several, for continuation lines).
+type fileRec struct {
+	t      time.Time
+	dated  bool // t comes from the line
+	sev    int
+	msg    string
+	host   string
+	app    string
+	fields map[string]string
+}
+
+const (
+	maxFileFields   = 64
+	maxFieldValue   = 4096
+	maxFileMsg      = 64 << 10 // message with its continuation lines
+	maxContinuation = 500      // continuation lines added to one entry
+)
+
+// fileParser turns a line of a file into an entry, for one format.
+type fileParser struct {
+	format string
+	re     *regexp.Regexp // fmtRegex
+}
+
+func newFileParser(format, expr string) (*fileParser, error) {
+	if !fileFormats[format] {
+		return nil, fmt.Errorf("unknown format %q", format)
+	}
+	p := &fileParser{format: format}
+	if format == fmtRegex {
+		if strings.TrimSpace(expr) == "" {
+			return nil, fmt.Errorf("the regex format needs a regular expression")
+		}
+		re, err := regexp.Compile(expr)
+		if err != nil {
+			return nil, fmt.Errorf("invalid regular expression: %v", err)
+		}
+		p.re = re
+	}
+	return p, nil
+}
+
+// multiline tells whether lines that do not start an entry are added to the
+// previous one (stack traces…). Structured formats are one entry per line.
+func (p *fileParser) multiline() bool {
+	return p.format == fmtText || p.format == fmtRegex
+}
+
+// continuation reports a line that belongs to the previous entry: indented,
+// or not matching the regular expression.
+func (p *fileParser) continuation(line string) bool {
+	if line != "" && (line[0] == ' ' || line[0] == '\t') {
+		return true
+	}
+	return p.format == fmtRegex && !p.re.MatchString(line)
+}
+
+// parse decodes one line; it never fails (a line in another format is kept as text).
+func (p *fileParser) parse(line string, now time.Time) *fileRec {
+	line = cleanLine(line)
+	switch p.format {
+	case fmtJSON, fmtTraefik:
+		if obj, ok := decodeJSONLine(line); ok {
+			if p.format == fmtTraefik {
+				if r := traefikRec(obj, line, now); r != nil {
+					return r
+				}
+			}
+			return jsonRec(obj, line, now)
+		}
+	case fmtLogfmt:
+		if kv := parseLogfmt(line); len(kv) > 0 {
+			return logfmtRec(kv, line, now)
+		}
+	case fmtRegex:
+		if m := p.re.FindStringSubmatch(line); m != nil {
+			return p.regexRec(m, line, now)
+		}
+	case fmtCLF:
+		if r := clfRec(line, now); r != nil {
+			return r
+		}
+	}
+	return textRec(line, now)
+}
+
+func cleanLine(s string) string {
+	s = strings.TrimRight(s, "\r")
+	s = ansiRe.ReplaceAllString(s, "")
+	if !utf8.ValidString(s) {
+		s = strings.ToValidUTF8(s, "�")
+	}
+	return s
+}
+
+// textRec: the line itself, without its leading timestamp when there is one.
+func textRec(line string, now time.Time) *fileRec {
+	r := &fileRec{t: now, sev: 6, msg: line}
+	if m := leadTimeRe.FindStringSubmatchIndex(line); m != nil {
+		if t, ok := parseLogTime(line[m[2]:m[3]], now); ok {
+			r.t, r.dated = t, true
+			if rest := line[m[1]:]; strings.TrimSpace(rest) != "" {
+				r.msg = rest
+			}
+		}
+	}
+	if n, ok := detectLevel(r.msg); ok {
+		r.sev = n
+	}
+	return r
+}
+
+/* ---------- Timestamps ---------- */
+
+// leadTimeRe: a timestamp at the start of a line, possibly in brackets
+// (ISO 8601 / RFC 3339, 2006/01/02 15:04:05, CLF, syslog "Jan _2 15:04:05").
+var leadTimeRe = regexp.MustCompile(`^\[?(\d{4}[-/]\d{2}[-/]\d{2}[T ]\d{2}:\d{2}:\d{2}(?:[.,]\d+)?(?: ?(?:Z|[+-]\d{2}:?\d{2}))?|\d{2}/[A-Z][a-z]{2}/\d{4}:\d{2}:\d{2}:\d{2} [+-]\d{4}|[A-Z][a-z]{2} +\d{1,2} \d{2}:\d{2}:\d{2})\]?[ \t]*(?:[-|:][ \t]+)?`)
+
+var timeLayouts = []string{
+	time.RFC3339Nano,
+	"2006-01-02T15:04:05Z0700",
+	"2006-01-02T15:04:05",
+	"02/Jan/2006:15:04:05 -0700",
+	"Jan _2 15:04:05",
+	time.RFC1123Z,
+	time.RFC1123,
+	time.UnixDate,
+	time.ANSIC,
+}
+
+// parseLogTime reads the usual log timestamps; times without a zone are local.
+func parseLogTime(s string, now time.Time) (time.Time, bool) {
+	s = strings.TrimSpace(s)
+	if s == "" {
+		return time.Time{}, false
+	}
+	if s[0] >= '0' && s[0] <= '9' && !strings.ContainsAny(s, ":/-") {
+		if whole, _, _ := strings.Cut(s, "."); len(whole) < 9 {
+			return time.Time{}, false // not a Unix time of this century
+		}
+		return epochTime(s)
+	}
+	// 2006/01/02 15:04:05,123 +02:00 -> 2006-01-02T15:04:05.123+02:00
+	if len(s) >= 19 && s[4] == '/' && s[7] == '/' {
+		s = s[:4] + "-" + s[5:7] + "-" + s[8:]
+	}
+	if len(s) >= 19 && s[4] == '-' && s[10] == ' ' {
+		s = s[:10] + "T" + s[11:]
+	}
+	if len(s) > 19 && s[4] == '-' {
+		s = s[:19] + strings.Replace(strings.Replace(s[19:], ",", ".", 1), " ", "", 1)
+	}
+	s = strings.Join(strings.Fields(s), " ") // "Jan  5" -> "Jan 5"
+	for _, layout := range timeLayouts {
+		if layout == "Jan _2 15:04:05" {
+			layout = "Jan 2 15:04:05"
+		}
+		t, err := time.ParseInLocation(layout, s, time.Local)
+		if err != nil {
+			continue
+		}
+		if t.Year() == 0 { // syslog timestamps have no year
+			t = t.AddDate(now.Year(), 0, 0)
+			if t.After(now.Add(24 * time.Hour)) {
+				t = t.AddDate(-1, 0, 0)
+			}
+		}
+		return t, true
+	}
+	return time.Time{}, false
+}
+
+// epochTime reads a Unix time in seconds, milliseconds, microseconds or
+// nanoseconds (guessed from its size), with an optional fraction.
+func epochTime(s string) (time.Time, bool) {
+	f, err := strconv.ParseFloat(s, 64)
+	if err != nil || f <= 0 {
+		return time.Time{}, false
+	}
+	switch {
+	case f < 1e11:
+		return time.Unix(0, int64(f*1e9)), true
+	case f < 1e14:
+		return time.UnixMilli(int64(f)), true
+	case f < 1e17:
+		return time.UnixMicro(int64(f)), true
+	}
+	return time.Unix(0, int64(f)), true
+}
+
+/* ---------- Fields ---------- */
+
+// reservedFields are stored by LogStream itself: fields of the logs with the
+// same name are renamed field_<name>.
+var reservedFields = map[string]bool{
+	"_time": true, "_msg": true, "_stream": true, "_stream_id": true, "received": true, "msg_time": true,
+	"host": true, "host_name": true, "host_ip": true, "app": true, "facility": true, "severity": true,
+	"sevnum": true, "source": true, "proto": true, "procid": true, "sd": true, "source_type": true, "log_file": true,
+}
+
+func fieldName(k string) string {
+	k = strings.Map(func(r rune) rune {
+		if r == '_' || r == '.' || r == '-' || r >= '0' && r <= '9' || r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' {
+			return r
+		}
+		return '_'
+	}, k)
+	if k == "" || reservedFields[k] || k[0] == '_' {
+		k = "field_" + strings.TrimLeft(k, "_")
+	}
+	return k
+}
+
+func (r *fileRec) set(k, v string) {
+	if v == "" {
+		return
+	}
+	if r.fields == nil {
+		r.fields = map[string]string{}
+	}
+	k = fieldName(k)
+	if _, ok := r.fields[k]; !ok && len(r.fields) >= maxFileFields {
+		return
+	}
+	if len(v) > maxFieldValue {
+		v = strings.ToValidUTF8(v[:maxFieldValue], "") + "…"
+	}
+	r.fields[k] = v
+}
+
+/* ---------- JSON lines ---------- */
+
+func decodeJSONLine(line string) (map[string]any, bool) {
+	s := strings.TrimSpace(line)
+	if len(s) < 2 || s[0] != '{' {
+		return nil, false
+	}
+	dec := json.NewDecoder(strings.NewReader(s))
+	dec.UseNumber()
+	var obj map[string]any
+	if dec.Decode(&obj) != nil {
+		return nil, false
+	}
+	return obj, true
+}
+
+var (
+	jsonMsgKeys   = []string{"msg", "message", "@message", "log", "text"}
+	jsonTimeKeys  = []string{"time", "ts", "timestamp", "@timestamp", "@t", "t", "datetime", "date"}
+	jsonLevelKeys = []string{"level", "lvl", "severity", "log.level", "@level", "@l", "loglevel", "levelname"}
+)
+
+// takeString removes the first of keys found in obj and returns it as text.
+func takeString(obj map[string]any, keys []string) (string, bool) {
+	for _, k := range keys {
+		if v, ok := obj[k]; ok {
+			if s := scalarString(v); s != "" {
+				delete(obj, k)
+				return s, true
+			}
+		}
+	}
+	return "", false
+}
+
+func scalarString(v any) string {
+	switch v := v.(type) {
+	case string:
+		return v
+	case json.Number:
+		return v.String()
+	case bool:
+		return strconv.FormatBool(v)
+	}
+	return ""
+}
+
+func jsonRec(obj map[string]any, line string, now time.Time) *fileRec {
+	r := &fileRec{t: now, sev: 6, msg: line}
+	for _, k := range jsonTimeKeys {
+		if t, ok := parseLogTime(scalarString(obj[k]), now); ok {
+			r.t, r.dated = t, true
+			delete(obj, k)
+			break
+		}
+	}
+	lvl, hasLevel := takeString(obj, jsonLevelKeys)
+	if hasLevel {
+		if n, ok := levelNum(lvl); ok {
+			r.sev = n
+		} else if n, err := strconv.Atoi(lvl); err == nil {
+			r.sev = numericLevel(n)
+		} else {
+			r.set("level", lvl)
+		}
+	}
+	if s, ok := takeString(obj, jsonMsgKeys); ok {
+		r.msg = strings.TrimRight(s, "\n")
+	}
+	flattenJSON("", obj, r)
+	if !hasLevel {
+		if n, ok := detectLevel(r.msg); ok {
+			r.sev = n
+		}
+	}
+	return r
+}
+
+// numericLevel maps numeric levels: syslog (0-7) or pino/bunyan (10-60).
+func numericLevel(n int) int {
+	switch {
+	case n >= 0 && n <= 7:
+		return n
+	case n >= 60:
+		return 2
+	case n >= 50:
+		return 3
+	case n >= 40:
+		return 4
+	case n >= 30:
+		return 6
+	}
+	return 7
+}
+
+// flattenJSON stores the remaining keys as fields; nested objects give
+// dotted names, arrays stay in JSON.
+func flattenJSON(prefix string, obj map[string]any, r *fileRec) {
+	for k, v := range obj {
+		name := prefix + k
+		switch v := v.(type) {
+		case map[string]any:
+			flattenJSON(name+".", v, r)
+		case []any:
+			b, _ := json.Marshal(v)
+			r.set(name, string(b))
+		case nil:
+		default:
+			r.set(name, scalarString(v))
+		}
+	}
+}
+
+/* ---------- logfmt ---------- */
+
+// parseLogfmt splits `k=v k2="v 2"` (with \" escapes); words without a value
+// are skipped. Nil when the line has no pair.
+func parseLogfmt(s string) [][2]string {
+	var out [][2]string
+	for i := 0; i < len(s); {
+		for i < len(s) && (s[i] == ' ' || s[i] == '\t') {
+			i++
+		}
+		start := i
+		for i < len(s) && s[i] != '=' && s[i] != ' ' && s[i] != '\t' {
+			i++
+		}
+		if i >= len(s) || s[i] != '=' || i == start {
+			for i < len(s) && s[i] != ' ' && s[i] != '\t' {
+				i++
+			}
+			continue
+		}
+		k := s[start:i]
+		i++
+		var v string
+		if i < len(s) && s[i] == '"' {
+			var b strings.Builder
+			i++
+			for i < len(s) && s[i] != '"' {
+				if s[i] == '\\' && i+1 < len(s) {
+					i++
+					switch s[i] {
+					case 'n':
+						b.WriteByte('\n')
+					case 't':
+						b.WriteByte('\t')
+					default:
+						b.WriteByte(s[i])
+					}
+				} else {
+					b.WriteByte(s[i])
+				}
+				i++
+			}
+			i++ // closing quote
+			v = b.String()
+		} else {
+			end := i
+			for end < len(s) && s[end] != ' ' && s[end] != '\t' {
+				end++
+			}
+			v, i = s[i:end], end
+		}
+		out = append(out, [2]string{k, v})
+	}
+	return out
+}
+
+func logfmtRec(kv [][2]string, line string, now time.Time) *fileRec {
+	obj := make(map[string]any, len(kv))
+	for _, p := range kv {
+		obj[p[0]] = p[1]
+	}
+	return jsonRec(obj, line, now)
+}
+
+/* ---------- Regular expression ---------- */
+
+func (p *fileParser) regexRec(m []string, line string, now time.Time) *fileRec {
+	r := &fileRec{t: now, sev: 6, msg: line}
+	hasLevel := false
+	for i, name := range p.re.SubexpNames() {
+		if i == 0 || name == "" || m[i] == "" {
+			continue
+		}
+		v := m[i]
+		switch strings.ToLower(name) {
+		case "time", "timestamp", "ts":
+			if t, ok := parseLogTime(v, now); ok {
+				r.t, r.dated = t, true
+			}
+		case "level", "severity", "lvl":
+			if n, ok := levelNum(v); ok {
+				r.sev, hasLevel = n, true
+			}
+		case "msg", "message":
+			r.msg = v
+		case "host":
+			r.host = v
+		case "app":
+			r.app = v
+		case "status":
+			r.set(name, v)
+			if n, ok := statusLevel(v); ok && !hasLevel {
+				r.sev, hasLevel = n, true
+			}
+		default:
+			r.set(name, v)
+		}
+	}
+	if !hasLevel {
+		if n, ok := detectLevel(r.msg); ok {
+			r.sev = n
+		}
+	}
+	return r
+}
+
+/* ---------- Access logs ---------- */
+
+// statusLevel: 5xx are errors, 4xx warnings, the rest info.
+func statusLevel(s string) (int, bool) {
+	n, err := strconv.Atoi(s)
+	if err != nil || n < 100 || n > 999 {
+		return 0, false
+	}
+	switch {
+	case n >= 500:
+		return 3, true
+	case n >= 400:
+		return 4, true
+	}
+	return 6, true
+}
+
+// clfRe: Common Log Format, the combined extension (referer, user agent) and
+// the Traefik one (request count, router, server URL, duration).
+var clfRe = regexp.MustCompile(`^(\S+) (\S+) (\S+) \[([^\]]+)\] "((?:[^"\\]|\\.)*)" (\d{3}) (\d+|-)(?: "((?:[^"\\]|\\.)*)" "((?:[^"\\]|\\.)*)")?(?: (\d+) "([^"]*)" "([^"]*)" (\d+)ms)?`)
+
+func clfRec(line string, now time.Time) *fileRec {
+	m := clfRe.FindStringSubmatch(line)
+	if m == nil {
+		return nil
+	}
+	r := &fileRec{t: now, sev: 6, msg: line}
+	if t, ok := parseLogTime(m[4], now); ok {
+		r.t, r.dated = t, true
+	}
+	dash := func(s string) string {
+		if s == "-" {
+			return ""
+		}
+		return s
+	}
+	r.set("client_ip", m[1])
+	r.set("http_user", dash(m[3]))
+	if req := strings.Fields(m[5]); len(req) == 3 {
+		r.set("http_method", req[0])
+		r.set("http_path", req[1])
+		r.set("http_proto", req[2])
+	} else {
+		r.set("http_request", m[5])
+	}
+	r.set("http_status", m[6])
+	r.set("http_size", dash(m[7]))
+	r.set("http_referer", dash(m[8]))
+	r.set("user_agent", dash(m[9]))
+	r.set("router", dash(m[11]))
+	r.set("backend", dash(m[12]))
+	r.set("duration_ms", m[13])
+	r.sev, _ = statusLevel(m[6])
+	return r
+}
+
+// traefikRec reads a Traefik access log line in JSON; nil for the other
+// lines (Traefik's own logs), read as generic JSON.
+func traefikRec(obj map[string]any, line string, now time.Time) *fileRec {
+	status := scalarString(obj["DownstreamStatus"])
+	if status == "" {
+		status = scalarString(obj["OriginStatus"])
+	}
+	method := scalarString(obj["RequestMethod"])
+	if status == "" && method == "" {
+		return nil
+	}
+	r := &fileRec{t: now, sev: 6}
+	if t, ok := parseLogTime(firstNonEmpty(scalarString(obj["StartUTC"]), scalarString(obj["time"])), now); ok {
+		r.t, r.dated = t, true
+	}
+	if n, ok := statusLevel(status); ok {
+		r.sev = n
+	}
+	rename := map[string]string{
+		"ClientHost": "client_ip", "ClientUsername": "http_user", "RequestMethod": "http_method",
+		"RequestHost": "http_host", "RequestPath": "http_path", "RequestProtocol": "http_proto",
+		"DownstreamStatus": "http_status", "DownstreamContentSize": "http_size", "OriginStatus": "origin_status",
+		"RouterName": "router", "ServiceName": "service", "ServiceURL": "backend", "entryPointName": "entrypoint",
+		"request_User-Agent": "user_agent", "request_Referer": "http_referer",
+	}
+	dur := ""
+	if d, err := strconv.ParseFloat(scalarString(obj["Duration"]), 64); err == nil {
+		dur = strconv.FormatFloat(d/1e6, 'f', -1, 64)
+		r.set("duration_ms", dur)
+	}
+	for _, k := range []string{"Duration", "StartUTC", "StartLocal", "time", "level", "msg"} {
+		delete(obj, k)
+	}
+	for k, v := range obj {
+		if to, ok := rename[k]; ok {
+			r.set(to, scalarString(v))
+			delete(obj, k)
+		}
+	}
+	flattenJSON("", obj, r)
+	// "GET example.com/path 404 · 12.5 ms · router@docker"
+	var b strings.Builder
+	b.WriteString(firstNonEmpty(method, "-"))
+	b.WriteByte(' ')
+	b.WriteString(r.fields["http_host"])
+	b.WriteString(r.fields["http_path"])
+	b.WriteByte(' ')
+	b.WriteString(firstNonEmpty(status, "-"))
+	if dur != "" {
+		b.WriteString(" · " + dur + " ms")
+	}
+	if rt := r.fields["router"]; rt != "" {
+		b.WriteString(" · " + rt)
+	}
+	if ip := r.fields["client_ip"]; ip != "" {
+		b.WriteString(" · " + ip)
+	}
+	r.msg = b.String()
+	return r
+}
+
+/* ---------- Detection ---------- */
+
+const detectWindow = 64 << 10
+
+// sampleLines returns the last complete lines of a file (50 at most).
+func sampleLines(path string) ([]string, error) {
+	fh, err := os.Open(path)
+	if err != nil {
+		return nil, err
+	}
+	defer fh.Close()
+	fi, err := fh.Stat()
+	if err != nil {
+		return nil, err
+	}
+	off := fi.Size() - detectWindow
+	if off < 0 {
+		off = 0
+	}
+	buf, err := io.ReadAll(io.NewSectionReader(fh, off, detectWindow))
+	if err != nil {
+		return nil, err
+	}
+	end := bytes.LastIndexByte(buf, '\n')
+	if end < 0 {
+		return nil, nil
+	}
+	buf = buf[:end]
+	if off > 0 { // first line cut by the window
+		if nl := bytes.IndexByte(buf, '\n'); nl >= 0 {
+			buf = buf[nl+1:]
+		} else {
+			return nil, nil
+		}
+	}
+	var lines []string
+	for _, b := range bytes.Split(buf, []byte{'\n'}) {
+		if len(bytes.TrimSpace(b)) > 0 {
+			lines = append(lines, string(bytes.TrimRight(b, "\r")))
+		}
+	}
+	if len(lines) > 50 {
+		lines = lines[len(lines)-50:]
+	}
+	return lines, nil
+}
+
+// logfmtKeys: the pairs have one of the usual logfmt keys.
+func logfmtKeys(kv [][2]string) bool {
+	for _, p := range kv {
+		switch strings.ToLower(p[0]) {
+		case "level", "lvl", "msg", "message", "time", "ts", "t":
+			return true
+		}
+	}
+	return false
+}
+
+// detectFormat guesses the format of a file from some of its lines: the
+// structured format most lines follow, else plain text (a regular
+// expression is never guessed).
+func detectFormat(lines []string) string {
+	var js, traefik, clf, logfmt int
+	for _, l := range lines {
+		l = strings.TrimSpace(l)
+		if obj, ok := decodeJSONLine(l); ok {
+			js++
+			if _, ok := obj["RequestMethod"]; ok {
+				traefik++
+			} else if _, ok := obj["DownstreamStatus"]; ok {
+				traefik++
+			}
+			continue
+		}
+		if clfRe.MatchString(l) {
+			clf++
+			continue
+		}
+		if kv := parseLogfmt(l); len(kv) >= 3 && logfmtKeys(kv) {
+			logfmt++
+		}
+	}
+	most := func(n int) bool { return n > 0 && n*5 >= len(lines)*4 } // 80 %
+	switch {
+	case most(js) && traefik*2 >= js:
+		return fmtTraefik
+	case most(js):
+		return fmtJSON
+	case most(clf):
+		return fmtCLF
+	case most(logfmt):
+		return fmtLogfmt
+	}
+	return fmtText
+}

filesource.go

@@ -0,0 +1,711 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"sync"
+	"time"
+)
+
+// fileSourceDef is one log file declared in Settings > Sources.
+type fileSourceDef struct {
+	ID      string `json:"id"`
+	Name    string `json:"name"`            // app of the entries
+	Path    string `json:"path"`            // the file, as mounted in the container
+	Format  string `json:"format"`          // auto (detected), or text, json, logfmt, regex, clf, traefik
+	Regex   string `json:"regex,omitempty"` // format regex: named groups time, level, msg, host, app…
+	Host    string `json:"host,omitempty"`  // host of the entries (empty: the host name of the machine)
+	Enabled bool   `json:"enabled"`
+}
+
+// filesConfig is saved in /data/files.json.
+type filesConfig struct {
+	Sources []fileSourceDef `json:"sources"`
+}
+
+// filePos is where reading resumes in a file: its inode and an offset before
+// which every entry was sent. Until: end of the history found when the
+// source was turned on; entries before it are kept only when dated within
+// FILES_BACKFILL.
+type filePos struct {
+	Ino   uint64 `json:"ino"`
+	Off   int64  `json:"off"`
+	Until int64  `json:"until,omitempty"`
+}
+
+type filesState struct {
+	Pos     map[string]filePos   `json:"pos"`     // by source id
+	Started map[string]time.Time `json:"started"` // when each source was turned on
+}
+
+type fileSourceStatus struct {
+	Open     bool
+	Read     uint64
+	Detected string // format found when the source is in auto
+	Behind   int64  // bytes left to read (history being read)
+	Code     string
+	Error    string
+}
+
+// fileSource is a definition with its parser (nil in auto: detected from
+// the file).
+type fileSource struct {
+	fileSourceDef
+	parser *fileParser
+}
+
+// pendingEntry is an entry waiting for its continuation lines.
+type pendingEntry struct {
+	rec     *fileRec
+	off     int64     // offset of its first line
+	seen    time.Time // when its last line was read
+	history bool      // read from the history found when the source was turned on
+	lines   int
+}
+
+// fileTail is the reading state of one file, owned by the Run goroutine.
+type fileTail struct {
+	readOff int64 // where the next read starts (after the lines of the pending entry)
+	known   bool  // readOff is set
+	pending *pendingEntry
+	auto    *fileParser // format detected from the file, in auto
+	behind  int64       // bytes left to read after the last poll
+}
+
+// Files follows log files of other programs (Traefik, nginx, applications…)
+// mounted in the container, one entry per line in the format detected or
+// chosen for each file. Source type "file".
+type Files struct {
+	sink      func(*Entry)
+	hostName  string
+	backfill  time.Duration
+	cfgPath   string
+	statePath string
+	wake      chan struct{}
+
+	// Owned by the Run goroutine.
+	st    filesState
+	tails map[string]*fileTail
+	dirty bool
+
+	mu       sync.Mutex
+	cfg      []*fileSource
+	resets   map[string]bool // sources to read again from FILES_BACKFILL ago
+	redetect map[string]bool // sources whose format changed
+	status   map[string]*fileSourceStatus
+}
+
+func NewFiles(hostRoot, dataDir string, backfill time.Duration, sink func(*Entry)) *Files {
+	f := &Files{
+		sink:      sink,
+		hostName:  readTrim(filepath.Join(hostRoot, "etc/hostname")),
+		backfill:  backfill,
+		cfgPath:   filepath.Join(dataDir, "files.json"),
+		statePath: filepath.Join(dataDir, "files-state.json"),
+		wake:      make(chan struct{}, 1),
+		tails:     map[string]*fileTail{},
+		resets:    map[string]bool{},
+		redetect:  map[string]bool{},
+		status:    map[string]*fileSourceStatus{},
+	}
+	if f.hostName == "" {
+		f.hostName = "localhost"
+	}
+	var cfg filesConfig
+	if b, err := os.ReadFile(f.cfgPath); err == nil {
+		_ = json.Unmarshal(b, &cfg)
+	}
+	for _, d := range cfg.Sources {
+		src, err := compileFileSource(d)
+		if err != nil {
+			logEvent("files", 4, "file %q ignored: %v", d.Name, err)
+			continue
+		}
+		f.cfg = append(f.cfg, src)
+	}
+	if b, err := os.ReadFile(f.statePath); err == nil {
+		_ = json.Unmarshal(b, &f.st)
+	}
+	if f.st.Pos == nil {
+		f.st.Pos = map[string]filePos{}
+	}
+	if f.st.Started == nil {
+		f.st.Started = map[string]time.Time{}
+	}
+	return f
+}
+
+const maxFileSources = 50
+
+var fileIDRe = regexp.MustCompile(`^[0-9a-f]{1,32}$`)
+
+// checkFilePath: one file, absolute, without wildcards.
+func checkFilePath(p string) (string, error) {
+	p = strings.TrimSpace(p)
+	switch {
+	case !filepath.IsAbs(p):
+		return "", errors.New("the path must be absolute (as mounted in the container)")
+	case strings.ContainsAny(p, "*?["):
+		return "", errors.New("give the path of one file (no wildcards)")
+	}
+	return filepath.Clean(p), nil
+}
+
+// compileFileSource checks a definition and builds its parser.
+func compileFileSource(d fileSourceDef) (*fileSource, error) {
+	d.Name = strings.TrimSpace(d.Name)
+	d.Host = strings.TrimSpace(d.Host)
+	if d.Format == "" {
+		d.Format = fmtAuto
+	}
+	if d.Format != fmtRegex {
+		d.Regex = ""
+	}
+	switch {
+	case !fileIDRe.MatchString(d.ID):
+		return nil, errors.New("invalid id")
+	case d.Name == "" || len(d.Name) > 64:
+		return nil, errors.New("the name is required (64 characters at most)")
+	case len(d.Host) > 253:
+		return nil, fmt.Errorf("%s: host name too long", d.Name)
+	}
+	p, err := checkFilePath(d.Path)
+	if err != nil {
+		return nil, fmt.Errorf("%s: %v", d.Name, err)
+	}
+	d.Path = p
+	src := &fileSource{fileSourceDef: d}
+	if d.Format != fmtAuto {
+		if src.parser, err = newFileParser(d.Format, d.Regex); err != nil {
+			return nil, fmt.Errorf("%s: %v", d.Name, err)
+		}
+	}
+	return src, nil
+}
+
+func newFileID() string {
+	b := make([]byte, 6)
+	_, _ = rand.Read(b)
+	return hex.EncodeToString(b)
+}
+
+// Configure checks, saves and applies the list of files. Files turned off,
+// removed or given another path are read again from FILES_BACKFILL ago the
+// next time they are on.
+func (f *Files) Configure(cfg filesConfig) error {
+	if len(cfg.Sources) > maxFileSources {
+		return fmt.Errorf("%d files at most", maxFileSources)
+	}
+	var srcs []*fileSource
+	ids := map[string]bool{}
+	for _, d := range cfg.Sources {
+		if d.ID == "" {
+			d.ID = newFileID()
+		}
+		if ids[d.ID] {
+			return fmt.Errorf("duplicate id %s", d.ID)
+		}
+		ids[d.ID] = true
+		src, err := compileFileSource(d)
+		if err != nil {
+			return err
+		}
+		srcs = append(srcs, src)
+	}
+	saved := filesConfig{Sources: []fileSourceDef{}}
+	for _, s := range srcs {
+		saved.Sources = append(saved.Sources, s.fileSourceDef)
+	}
+	if err := writeJSONFile(f.cfgPath, saved); err != nil {
+		return err
+	}
+	f.mu.Lock()
+	old := map[string]*fileSource{}
+	for _, s := range f.cfg {
+		old[s.ID] = s
+	}
+	for _, s := range srcs {
+		o := old[s.ID]
+		if o != nil && ((o.Enabled && !s.Enabled) || o.Path != s.Path) {
+			f.resets[s.ID] = true
+			delete(f.status, s.ID)
+		}
+		if o != nil && (o.Format != s.Format || o.Regex != s.Regex) {
+			f.redetect[s.ID] = true
+		}
+		delete(old, s.ID)
+	}
+	for id := range old { // removed
+		f.resets[id] = true
+		delete(f.status, id)
+	}
+	f.cfg = srcs
+	f.mu.Unlock()
+	select {
+	case f.wake <- struct{}{}:
+	default:
+	}
+	return nil
+}
+
+// Status is the state shown in Settings > Sources.
+func (f *Files) Status() map[string]any {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	list := make([]map[string]any, 0, len(f.cfg))
+	for _, s := range f.cfg {
+		m := map[string]any{
+			"id": s.ID, "name": s.Name, "path": s.Path, "format": s.Format, "regex": s.Regex,
+			"host": s.Host, "enabled": s.Enabled, "open": false, "read": 0, "detected": "", "behind": 0, "code": "", "error": "",
+		}
+		if st := f.status[s.ID]; st != nil && s.Enabled {
+			m["open"], m["read"], m["detected"], m["code"], m["error"] = st.Open, st.Read, st.Detected, st.Code, st.Error
+			m["behind"] = st.Behind
+		}
+		list = append(list, m)
+	}
+	return map[string]any{"sources": list, "defaultHost": f.hostName}
+}
+
+// Run polls the files every second.
+func (f *Files) Run(ctx context.Context) {
+	tick := time.NewTicker(time.Second)
+	defer tick.Stop()
+	n := 0
+	for {
+		select {
+		case <-ctx.Done():
+			now := time.Now()
+			for id := range f.tails {
+				f.flushPending(id, now)
+			}
+			f.checkpointAll()
+			f.saveState()
+			return
+		case <-tick.C:
+		case <-f.wake:
+		}
+		f.poll(time.Now())
+		if n++; n%5 == 0 {
+			f.saveState()
+		}
+	}
+}
+
+func (f *Files) saveState() {
+	if !f.dirty {
+		return
+	}
+	f.dirty = false
+	if err := writeJSONFile(f.statePath, f.st); err != nil {
+		logEvent("files", 4, "saving positions: %v", err)
+	}
+}
+
+// poll reads what was added to every file that is on.
+func (f *Files) poll(now time.Time) {
+	f.mu.Lock()
+	srcs := f.cfg
+	resets, redetect := f.resets, f.redetect
+	f.resets, f.redetect = map[string]bool{}, map[string]bool{}
+	f.mu.Unlock()
+	for id := range resets {
+		delete(f.st.Pos, id)
+		delete(f.st.Started, id)
+		delete(f.tails, id)
+		f.dirty = true
+	}
+	for id := range redetect {
+		if tl := f.tails[id]; tl != nil {
+			tl.auto = nil
+		}
+	}
+	active := map[string]bool{}
+	for _, s := range srcs {
+		if !s.Enabled {
+			continue
+		}
+		active[s.ID] = true
+		f.setStatus(s.ID, f.pollFile(s, now))
+	}
+	// Files turned off: their position is kept until the next poll forgets it.
+	for id := range f.tails {
+		if !active[id] {
+			delete(f.tails, id)
+		}
+	}
+	f.checkpointAll()
+}
+
+const (
+	maxFileRead       = 4 << 20         // per read
+	maxFileReads      = 16              // reads per file and per poll (64 MB)
+	fileContinueDelay = 2 * time.Second // an entry waits this long for continuation lines
+)
+
+func (f *Files) setStatus(id string, err error) {
+	code, detail := "", ""
+	if err != nil {
+		code, detail = errCode(err), err.Error()
+		if errors.Is(err, fs.ErrNotExist) {
+			code = "not_found"
+		}
+	}
+	detected := ""
+	if tl := f.tails[id]; tl != nil && tl.auto != nil {
+		detected = tl.auto.format
+	}
+	f.mu.Lock()
+	st := f.status[id]
+	if st == nil {
+		st = &fileSourceStatus{}
+		f.status[id] = st
+	}
+	changed := code != st.Code
+	st.Open, st.Detected, st.Code, st.Error = err == nil, detected, code, detail
+	st.Behind = 0
+	if tl := f.tails[id]; tl != nil && err == nil {
+		st.Behind = tl.behind
+	}
+	name := id
+	for _, s := range f.cfg {
+		if s.ID == id {
+			name = s.Name
+		}
+	}
+	f.mu.Unlock()
+	if changed && code != "" && code != "not_found" {
+		logEvent("files", 4, "cannot read the file %s: %v", name, err)
+	}
+}
+
+func (f *Files) count(id string) {
+	f.mu.Lock()
+	st := f.status[id]
+	if st == nil { // first poll of the file: its status comes after
+		st = &fileSourceStatus{}
+		f.status[id] = st
+	}
+	st.Read++
+	f.mu.Unlock()
+}
+
+// parserOf is the parser of a file: the chosen format, or the one detected
+// from its last lines (nil while the file has none).
+func (f *Files) parserOf(s *fileSource, tl *fileTail, path string) *fileParser {
+	if s.parser != nil {
+		return s.parser
+	}
+	if tl.auto == nil {
+		lines, err := sampleLines(path)
+		if err != nil || len(lines) == 0 {
+			return nil
+		}
+		tl.auto, _ = newFileParser(detectFormat(lines), "")
+	}
+	return tl.auto
+}
+
+func (f *Files) pollFile(s *fileSource, now time.Time) error {
+	path := s.Path
+	first := false
+	if _, ok := f.st.Started[s.ID]; !ok {
+		f.st.Started[s.ID], f.dirty, first = now, true, true
+	}
+	fi, err := os.Stat(path)
+	if err != nil {
+		return err
+	}
+	if !fi.Mode().IsRegular() {
+		return fmt.Errorf("%s is not a regular file", path)
+	}
+	tl := f.tails[s.ID]
+	if tl == nil {
+		tl = &fileTail{}
+		f.tails[s.ID] = tl
+	}
+	ino := inodeOf(fi)
+	pos, known := f.st.Pos[s.ID]
+	if !known {
+		pos = filePos{Ino: ino}
+		if first {
+			// Turned on: the last FILES_BACKFILL of what is already there
+			// (0: only new lines), found by date rather than read from the
+			// start of a file that can hold months of logs.
+			pos.Off = fi.Size()
+			if p := f.parserOf(s, tl, path); f.backfill > 0 && p != nil {
+				pos.Off = backfillStart(path, p, fi.Size(), f.st.Started[s.ID].Add(-f.backfill), now)
+				pos.Until = fi.Size()
+			}
+		}
+		f.st.Pos[s.ID], f.dirty = pos, true
+	}
+	if !tl.known {
+		tl.readOff, tl.known = pos.Off, true
+	}
+	if pos.Ino != ino {
+		// Rotated (access.log -> access.log.1): finish the old file, then
+		// read the new one from its start.
+		if old, err := os.Stat(path + ".1"); err == nil && inodeOf(old) == pos.Ino {
+			if p := f.parserOf(s, tl, path+".1"); p != nil {
+				for {
+					next, more, err := f.readFrom(s, tl, p, path+".1", tl.readOff, pos.Until, now)
+					if err != nil || !more {
+						break
+					}
+					tl.readOff = next
+				}
+			}
+		}
+		f.flushPending(s.ID, now)
+		pos, tl.readOff = filePos{Ino: ino}, 0
+		f.st.Pos[s.ID], f.dirty = pos, true
+	} else if fi.Size() < tl.readOff {
+		f.flushPending(s.ID, now) // truncated (copytruncate)
+		pos.Until, tl.readOff = 0, 0
+		f.st.Pos[s.ID], f.dirty = pos, true
+	}
+	if fi.Size() > tl.readOff {
+		p := f.parserOf(s, tl, path)
+		if p == nil {
+			return nil // nothing complete to detect the format from yet
+		}
+		for i := 0; i < maxFileReads; i++ {
+			next, more, err := f.readFrom(s, tl, p, path, tl.readOff, pos.Until, now)
+			if err != nil {
+				return err
+			}
+			tl.readOff = next
+			if !more {
+				break
+			}
+		}
+	}
+	tl.behind = max(fi.Size()-tl.readOff, 0)
+	if pe := tl.pending; pe != nil && now.Sub(pe.seen) >= fileContinueDelay {
+		f.flushPending(s.ID, now)
+	}
+	return nil
+}
+
+// backfillStart finds where the lines dated notBefore or later start, by
+// bisection on the dates of the lines (logs are written in order). Undated
+// lines are left out of the history anyway: then only new lines are read.
+func backfillStart(path string, p *fileParser, size int64, notBefore, now time.Time) int64 {
+	fh, err := os.Open(path)
+	if err != nil {
+		return size
+	}
+	defer fh.Close()
+	buf := make([]byte, 64<<10)
+	// lineAt reads the first line starting at or after off (off itself when
+	// it is 0): its start and its date.
+	lineAt := func(off int64) (int64, time.Time, bool) {
+		n, _ := fh.ReadAt(buf, off)
+		b := buf[:n]
+		start := off
+		if off > 0 {
+			nl := bytes.IndexByte(b, '\n')
+			if nl < 0 {
+				return size, time.Time{}, false
+			}
+			b, start = b[nl+1:], off+int64(nl)+1
+		}
+		end := bytes.IndexByte(b, '\n')
+		if end < 0 {
+			return size, time.Time{}, false
+		}
+		r := p.parse(string(b[:end]), now)
+		return start, r.t, r.dated
+	}
+	if _, t, ok := lineAt(0); !ok || !t.Before(notBefore) {
+		if !ok {
+			// No date on the first line: try the last ones.
+			if _, _, ok := lineAt(max(size-int64(len(buf)), 0)); !ok {
+				return size
+			}
+		} else {
+			return 0 // all of it is recent
+		}
+	}
+	lo, hi := int64(0), size // line at lo: before notBefore; at hi: not
+	for hi-lo > int64(len(buf)) {
+		mid := lo + (hi-lo)/2
+		_, t, ok := lineAt(mid)
+		if ok && t.Before(notBefore) {
+			lo = mid
+		} else {
+			hi = mid
+		}
+	}
+	start, _, _ := lineAt(lo)
+	return min(start, size)
+}
+
+// readFrom parses the complete lines written after off and returns the new
+// offset; more tells whether the read window was full.
+func (f *Files) readFrom(s *fileSource, tl *fileTail, p *fileParser, path string, off, until int64, now time.Time) (int64, bool, error) {
+	fh, err := os.Open(path)
+	if err != nil {
+		return off, false, err
+	}
+	defer fh.Close()
+	buf, err := io.ReadAll(io.NewSectionReader(fh, off, maxFileRead))
+	if err != nil {
+		return off, false, err
+	}
+	full := len(buf) == maxFileRead
+	end := bytes.LastIndexByte(buf, '\n')
+	if end < 0 {
+		if !full {
+			return off, false, nil // partial line: wait for its end
+		}
+		end = len(buf) - 1 // line longer than the read window: cut it
+	}
+	start := off
+	for _, b := range bytes.Split(buf[:end], []byte{'\n'}) {
+		f.line(s, tl, p, string(b), start, start < until, now)
+		start += int64(len(b)) + 1
+	}
+	return off + int64(end) + 1, full, nil
+}
+
+// line handles one line: a continuation of the pending entry, or a new one.
+func (f *Files) line(s *fileSource, tl *fileTail, p *fileParser, line string, off int64, history bool, now time.Time) {
+	if strings.TrimSpace(line) == "" {
+		return
+	}
+	if pe := tl.pending; p.multiline() && pe != nil && p.continuation(line) {
+		if pe.lines < maxContinuation && len(pe.rec.msg) < maxFileMsg {
+			pe.rec.msg += "\n" + strings.TrimRight(cleanLine(line), " \t")
+			pe.lines++
+		}
+		pe.seen = now
+		return
+	}
+	f.flushPending(s.ID, now)
+	pe := &pendingEntry{rec: p.parse(line, now), off: off, seen: now, history: history}
+	if p.multiline() {
+		tl.pending = pe
+		return
+	}
+	f.emit(s, pe, now)
+}
+
+func (f *Files) flushPending(id string, now time.Time) {
+	tl := f.tails[id]
+	if tl == nil || tl.pending == nil {
+		return
+	}
+	pe := tl.pending
+	tl.pending = nil
+	f.mu.Lock()
+	var src *fileSource
+	for _, s := range f.cfg {
+		if s.ID == id {
+			src = s
+		}
+	}
+	f.mu.Unlock()
+	if src != nil {
+		f.emit(src, pe, now)
+	}
+}
+
+// emit sends an entry; history entries are kept only when dated within
+// FILES_BACKFILL of when the file was turned on.
+func (f *Files) emit(s *fileSource, pe *pendingEntry, now time.Time) {
+	r := pe.rec
+	if pe.history {
+		notBefore := f.st.Started[s.ID].Add(-f.backfill)
+		if !r.dated || r.t.Before(notBefore) {
+			return
+		}
+	}
+	if len(r.msg) > maxFileMsg {
+		r.msg = strings.ToValidUTF8(r.msg[:maxFileMsg], "") + "…"
+	}
+	extra := map[string]string{"log_file": s.Path}
+	for k, v := range r.fields {
+		extra[k] = v
+	}
+	f.sink(&Entry{
+		Time:       r.t,
+		Received:   now,
+		Host:       firstNonEmpty(r.host, s.Host, f.hostName),
+		App:        firstNonEmpty(r.app, s.Name),
+		SevNum:     r.sev,
+		Severity:   severityNames[r.sev],
+		Message:    r.msg,
+		Proto:      "file",
+		SourceType: "file",
+		Extra:      extra,
+		Wait:       true, // the file keeps the lines: wait for room in the queue
+	})
+	f.count(s.ID)
+}
+
+// checkpointAll saves, for each file, an offset before which every entry is
+// sent: the start of its pending entry, or the read offset.
+func (f *Files) checkpointAll() {
+	for id, tl := range f.tails {
+		pos, ok := f.st.Pos[id]
+		if !ok || !tl.known {
+			continue
+		}
+		off := tl.readOff
+		if pe := tl.pending; pe != nil && pe.off < off {
+			off = pe.off
+		}
+		if off != pos.Off {
+			pos.Off, f.dirty = off, true
+			f.st.Pos[id] = pos
+		}
+	}
+}
+
+// DetectFile tells, for the settings, whether a file can be read and its
+// format, with a line as example.
+func DetectFile(path string) map[string]any {
+	res := map[string]any{"format": "", "sample": "", "code": "", "error": ""}
+	p, err := checkFilePath(path)
+	if err != nil {
+		res["code"], res["error"] = "invalid", err.Error()
+		return res
+	}
+	var lines []string
+	if fi, err2 := os.Stat(p); err2 != nil {
+		err = err2
+	} else if !fi.Mode().IsRegular() {
+		err = fmt.Errorf("%s is not a regular file", p)
+	} else {
+		lines, err = sampleLines(p)
+	}
+	switch {
+	case errors.Is(err, fs.ErrNotExist):
+		res["code"], res["error"] = "not_found", err.Error()
+	case err != nil:
+		res["code"], res["error"] = errCode(err), err.Error()
+	case len(lines) == 0:
+		res["code"] = "empty"
+	default:
+		res["format"] = detectFormat(lines)
+		sample := lines[len(lines)-1]
+		if len(sample) > 300 {
+			sample = strings.ToValidUTF8(sample[:300], "") + "…"
+		}
+		res["sample"] = cleanLine(sample)
+	}
+	return res
+}

filesource_test.go

@@ -0,0 +1,427 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+)
+
+func mustParser(t *testing.T, format, expr string) *fileParser {
+	t.Helper()
+	p, err := newFileParser(format, expr)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return p
+}
+
+func TestParseLogTime(t *testing.T) {
+	now := time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC)
+	utc := func(s string) time.Time { t, _ := time.Parse(time.RFC3339Nano, s); return t }
+	tests := []struct {
+		in   string
+		want time.Time
+	}{
+		{"2026-10-05T10:00:00.123Z", utc("2026-10-05T10:00:00.123Z")},
+		{"2026-10-05T10:00:00+02:00", utc("2026-10-05T08:00:00Z")},
+		{"2026-10-05 10:00:00,250 +0200", utc("2026-10-05T08:00:00.25Z")},
+		{"2026/10/05 10:00:00Z", utc("2026-10-05T10:00:00Z")},
+		{"05/Oct/2026:10:00:00 +0200", utc("2026-10-05T08:00:00Z")},
+		{"1759572000", time.Unix(1759572000, 0)},
+		{"1759572000123", time.UnixMilli(1759572000123)},
+	}
+	for _, tt := range tests {
+		got, ok := parseLogTime(tt.in, now)
+		if !ok || !got.Equal(tt.want) {
+			t.Errorf("%q: got %v %v, want %v", tt.in, got, ok, tt.want)
+		}
+	}
+	// Syslog timestamps: no year, the last one before tomorrow.
+	if got, ok := parseLogTime("Dec 31 23:00:00", now); !ok || got.Year() != 2025 {
+		t.Errorf("Dec 31: %v %v", got, ok)
+	}
+	if got, ok := parseLogTime("Oct  5 11:00:00", now); !ok || got.Year() != 2026 || got.Day() != 5 {
+		t.Errorf("Oct 5: %v %v", got, ok)
+	}
+	for _, bad := range []string{"", "hello", "200", "12345"} {
+		if _, ok := parseLogTime(bad, now); ok {
+			t.Errorf("%q parsed", bad)
+		}
+	}
+}
+
+func TestParseText(t *testing.T) {
+	now := time.Now()
+	p := mustParser(t, fmtText, "")
+	r := p.parse("2026-10-05 10:00:00.5 ERROR db: connection refused", now)
+	if !r.dated || r.msg != "ERROR db: connection refused" || r.sev != 3 {
+		t.Errorf("got %+v", r)
+	}
+	r = p.parse("[2026-10-05T10:00:00Z] - started", now)
+	if !r.dated || r.msg != "started" || r.sev != 6 {
+		t.Errorf("got %+v", r)
+	}
+	r = p.parse("\x1b[33mWARN\x1b[0m no timestamp here", now)
+	if r.dated || r.msg != "WARN no timestamp here" || r.sev != 4 || !r.t.Equal(now) {
+		t.Errorf("got %+v", r)
+	}
+}
+
+func TestParseJSON(t *testing.T) {
+	now := time.Now()
+	p := mustParser(t, fmtJSON, "")
+	r := p.parse(`{"time":"2026-10-05T10:00:00Z","level":"warn","msg":"slow query","host":"db.example","req":{"id":7,"tags":["a"]},"ok":true}`, now)
+	if !r.dated || r.sev != 4 || r.msg != "slow query" {
+		t.Errorf("got %+v", r)
+	}
+	for k, want := range map[string]string{"field_host": "db.example", "req.id": "7", "req.tags": `["a"]`, "ok": "true"} {
+		if r.fields[k] != want {
+			t.Errorf("%s = %q, want %q", k, r.fields[k], want)
+		}
+	}
+	// pino: numeric level, epoch in ms.
+	r = p.parse(`{"level":50,"time":1759572000123,"msg":"boom"}`, now)
+	if r.sev != 3 || !r.t.Equal(time.UnixMilli(1759572000123)) {
+		t.Errorf("got %+v", r)
+	}
+	// No message key: the whole line; not JSON: text.
+	line := `{"event":"login","user":"bob"}`
+	if r = p.parse(line, now); r.msg != line || r.fields["user"] != "bob" {
+		t.Errorf("got %+v", r)
+	}
+	if r = p.parse("plain ERROR line", now); r.msg != "plain ERROR line" || r.sev != 3 {
+		t.Errorf("got %+v", r)
+	}
+}
+
+func TestParseLogfmt(t *testing.T) {
+	p := mustParser(t, fmtLogfmt, "")
+	r := p.parse(`time="2026-10-05T10:00:00Z" level=error msg="Cannot connect: \"docker\" down" providerName=docker flag`, time.Now())
+	if !r.dated || r.sev != 3 || r.msg != `Cannot connect: "docker" down` || r.fields["providerName"] != "docker" {
+		t.Errorf("got %+v", r)
+	}
+}
+
+func TestParseRegex(t *testing.T) {
+	if _, err := newFileParser(fmtRegex, "("); err == nil {
+		t.Error("invalid regex accepted")
+	}
+	if _, err := newFileParser(fmtRegex, ""); err == nil {
+		t.Error("empty regex accepted")
+	}
+	p := mustParser(t, fmtRegex, `^(?P<time>\S+ \S+) \[(?P<level>\w+)\] (?P<module>[\w.]+): (?P<msg>.*)$`)
+	r := p.parse("2026-10-05 10:00:00 [warning] app.db: pool exhausted", time.Now())
+	if !r.dated || r.sev != 4 || r.msg != "pool exhausted" || r.fields["module"] != "app.db" {
+		t.Errorf("got %+v", r)
+	}
+	if !p.continuation("Traceback (most recent call last):") || p.continuation("2026-10-05 10:00:01 [info] a: b") {
+		t.Error("continuation")
+	}
+}
+
+func TestParseCLF(t *testing.T) {
+	p := mustParser(t, fmtCLF, "")
+	r := p.parse(`203.0.113.9 - - [05/Oct/2026:10:00:00 +0200] "GET /api/x?y=1 HTTP/1.1" 502 157 "-" "curl/8.0" 4211 "api@docker" "http://172.18.0.5:8080" 31ms`, time.Now())
+	if !r.dated || r.sev != 3 || !strings.HasPrefix(r.msg, "203.0.113.9 - - [") {
+		t.Errorf("got %+v", r)
+	}
+	for k, want := range map[string]string{"client_ip": "203.0.113.9", "http_method": "GET", "http_path": "/api/x?y=1", "http_status": "502",
+		"user_agent": "curl/8.0", "router": "api@docker", "backend": "http://172.18.0.5:8080", "duration_ms": "31"} {
+		if r.fields[k] != want {
+			t.Errorf("%s = %q, want %q", k, r.fields[k], want)
+		}
+	}
+	if _, ok := r.fields["http_referer"]; ok {
+		t.Error("empty referer kept")
+	}
+	// nginx combined, 404.
+	r = p.parse(`10.0.0.1 - alice [05/Oct/2026:10:00:00 +0000] "POST /login HTTP/2.0" 404 0 "https://example.com/" "Mozilla/5.0"`, time.Now())
+	if r.sev != 4 || r.fields["http_user"] != "alice" || r.fields["http_referer"] != "https://example.com/" {
+		t.Errorf("got %+v", r)
+	}
+}
+
+func TestParseTraefikJSON(t *testing.T) {
+	p := mustParser(t, fmtTraefik, "")
+	r := p.parse(`{"ClientHost":"203.0.113.9","DownstreamStatus":200,"Duration":12500000,"RequestHost":"app.example.com","RequestMethod":"GET","RequestPath":"/","RequestProtocol":"HTTP/2.0","RouterName":"app@docker","StartUTC":"2026-10-05T10:00:00.1Z","entryPointName":"websecure","level":"info","msg":"","time":"2026-10-05T10:00:00Z","TLSVersion":"1.3"}`, time.Now())
+	if !r.dated || r.sev != 6 || r.msg != "GET app.example.com/ 200 · 12.5 ms · app@docker · 203.0.113.9" {
+		t.Errorf("got %+v", r)
+	}
+	for k, want := range map[string]string{"entrypoint": "websecure", "http_status": "200", "TLSVersion": "1.3", "duration_ms": "12.5"} {
+		if r.fields[k] != want {
+			t.Errorf("%s = %q, want %q", k, r.fields[k], want)
+		}
+	}
+	// Traefik's own logs in the same file: generic JSON.
+	r = p.parse(`{"level":"error","msg":"Provider error","time":"2026-10-05T10:00:00Z"}`, time.Now())
+	if r.sev != 3 || r.msg != "Provider error" {
+		t.Errorf("got %+v", r)
+	}
+}
+
+func TestDetectFormat(t *testing.T) {
+	tests := map[string][]string{
+		fmtJSON:    {`{"level":"info","msg":"a"}`, `{"level":"warn","msg":"b"}`},
+		fmtTraefik: {`{"RequestMethod":"GET","DownstreamStatus":200}`, `{"level":"info","msg":"Configuration loaded"}`},
+		fmtCLF:     {`10.0.0.1 - - [05/Oct/2026:10:00:00 +0000] "GET / HTTP/1.1" 200 12 "-" "curl"`},
+		fmtLogfmt:  {`time="2026-10-05T10:00:00Z" level=info msg="Starting provider" providerName=docker`},
+		fmtText:    {`2026-10-05 10:00:00 INFO started`, `user=bob logged in`},
+	}
+	for want, lines := range tests {
+		if got := detectFormat(lines); got != want {
+			t.Errorf("%q: got %s, want %s", lines, got, want)
+		}
+	}
+	// A few lines of another format do not change the answer (80 %).
+	lines := []string{`{"a":1}`, `{"a":2}`, `{"a":3}`, `{"a":4}`, `not json`}
+	if got := detectFormat(lines); got != fmtJSON {
+		t.Errorf("got %s", got)
+	}
+	lines = append(lines, `still not json`)
+	if got := detectFormat(lines); got != fmtText {
+		t.Errorf("got %s", got)
+	}
+}
+
+func TestDetectFile(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "access.log")
+	appendFile(t, path, `10.0.0.1 - - [05/Oct/2026:10:00:00 +0000] "GET / HTTP/1.1" 200 12 "-" "curl"`+"\n"+`partial`)
+	res := DetectFile(path)
+	if res["format"] != fmtCLF || !strings.HasPrefix(res["sample"].(string), "10.0.0.1 - -") || res["code"] != "" {
+		t.Errorf("got %+v", res)
+	}
+	for p, code := range map[string]string{
+		filepath.Join(dir, "nope.log"): "not_found", "relative.log": "invalid", filepath.Join(dir, "*.log"): "invalid", dir: "read",
+	} {
+		if res := DetectFile(p); res["code"] != code {
+			t.Errorf("%s: got %+v, want %s", p, res, code)
+		}
+	}
+	empty := filepath.Join(dir, "empty.log")
+	appendFile(t, empty, "")
+	if res := DetectFile(empty); res["code"] != "empty" {
+		t.Errorf("got %+v", res)
+	}
+}
+
+func TestFileSourceValidation(t *testing.T) {
+	f := NewFiles(t.TempDir(), t.TempDir(), time.Hour, func(*Entry) {})
+	bad := []fileSourceDef{
+		{Name: "", Path: "/logs/a.log"},
+		{Name: "a", Path: "relative.log"},
+		{Name: "a", Path: "/logs/*.log"},
+		{Name: "a", Path: "/logs/a.log", Format: "xml"},
+		{Name: "a", Path: "/logs/a.log", Format: fmtRegex, Regex: "(?P<x"},
+	}
+	for _, d := range bad {
+		if err := f.Configure(filesConfig{Sources: []fileSourceDef{d}}); err == nil {
+			t.Errorf("%+v accepted", d)
+		}
+	}
+	if err := f.Configure(filesConfig{Sources: []fileSourceDef{{Name: " traefik ", Path: "/logs/traefik/access.log"}}}); err != nil {
+		t.Fatal(err)
+	}
+	src := f.Status()["sources"].([]map[string]any)
+	if len(src) != 1 || src[0]["name"] != "traefik" || src[0]["id"] == "" || src[0]["format"] != fmtAuto {
+		t.Errorf("got %+v", src)
+	}
+}
+
+type fileSink struct {
+	mu sync.Mutex
+	e  []*Entry
+}
+
+func (s *fileSink) add(e *Entry) { s.mu.Lock(); s.e = append(s.e, e); s.mu.Unlock() }
+func (s *fileSink) take() []*Entry {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	e := s.e
+	s.e = nil
+	return e
+}
+
+func appendFile(t *testing.T, path, s string) {
+	t.Helper()
+	fh, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, err := fh.WriteString(s); err != nil {
+		t.Fatal(err)
+	}
+	fh.Close()
+}
+
+func msgs(es []*Entry) []string {
+	var out []string
+	for _, e := range es {
+		out = append(out, e.Message)
+	}
+	return out
+}
+
+func TestFilesFollow(t *testing.T) {
+	dir, data := t.TempDir(), t.TempDir()
+	now := time.Now()
+	old := now.Add(-3 * time.Hour).UTC().Format(time.RFC3339)
+	recent := now.Add(-10 * time.Minute).UTC().Format(time.RFC3339)
+	app := filepath.Join(dir, "app.log")
+	appendFile(t, app, old+" INFO too old\n"+recent+" ERROR recent\n  at line 1\nundated history line\n")
+
+	sink := &fileSink{}
+	f := NewFiles(dir, data, time.Hour, sink.add)
+	if err := f.Configure(filesConfig{Sources: []fileSourceDef{{Name: "myapp", Path: app, Host: "web1", Enabled: true}}}); err != nil {
+		t.Fatal(err)
+	}
+	f.poll(now)
+	f.poll(now.Add(3 * time.Second)) // the pending entry waited for continuation lines
+	got := sink.take()
+	if len(got) != 1 || got[0].Message != "ERROR recent\n  at line 1" || got[0].SevNum != 3 || got[0].App != "myapp" ||
+		got[0].Host != "web1" || got[0].SourceType != "file" || got[0].Extra["log_file"] != app {
+		t.Fatalf("history: got %q %+v", msgs(got), got)
+	}
+	if st := f.Status()["sources"].([]map[string]any)[0]; st["detected"] != fmtText || st["open"] != true {
+		t.Errorf("status %+v", st)
+	}
+
+	appendFile(t, app, "plain new line\n\tcontinued\n")
+	f.poll(now.Add(4 * time.Second))
+	appendFile(t, app, "after\n")
+	f.poll(now.Add(5 * time.Second))
+	f.poll(now.Add(8 * time.Second))
+	if got = sink.take(); strings.Join(msgs(got), "|") != "plain new line\n\tcontinued|after" {
+		t.Fatalf("new lines: got %q", msgs(got))
+	}
+
+	appendFile(t, app, "before rotation\n")
+	if err := os.Rename(app, app+".1"); err != nil {
+		t.Fatal(err)
+	}
+	appendFile(t, app, "after rotation\n")
+	f.poll(now.Add(9 * time.Second))
+	f.poll(now.Add(12 * time.Second))
+	if got = sink.take(); strings.Join(msgs(got), "|") != "before rotation|after rotation" {
+		t.Fatalf("rotation: got %q", msgs(got))
+	}
+
+	// Restart: resumes where it stopped, nothing sent twice.
+	f.checkpointAll()
+	f.dirty = true
+	f.saveState()
+	appendFile(t, app, "while stopped\n")
+	f2 := NewFiles(dir, data, time.Hour, sink.add)
+	f2.poll(now.Add(13 * time.Second))
+	f2.poll(now.Add(16 * time.Second))
+	if got = sink.take(); strings.Join(msgs(got), "|") != "while stopped" {
+		t.Fatalf("restart: got %q", msgs(got))
+	}
+
+	// Truncated in place (copytruncate).
+	os.Truncate(app, 0)
+	appendFile(t, app, "x\n")
+	f2.poll(now.Add(17 * time.Second))
+	f2.poll(now.Add(20 * time.Second))
+	if got = sink.take(); strings.Join(msgs(got), "|") != "x" {
+		t.Fatalf("truncate: got %q", msgs(got))
+	}
+}
+
+func TestFilesAutoAndErrors(t *testing.T) {
+	dir, data := t.TempDir(), t.TempDir()
+	path := filepath.Join(dir, "access.json")
+	sink := &fileSink{}
+	f := NewFiles(dir, data, 0, sink.add) // FILES_BACKFILL=0: only new lines
+	cfg := filesConfig{Sources: []fileSourceDef{
+		{ID: "a1", Name: "api", Path: path, Enabled: true}, // auto
+		{ID: "b2", Name: "missing", Path: filepath.Join(dir, "nope.log"), Format: fmtText, Enabled: true},
+	}}
+	if err := f.Configure(cfg); err != nil {
+		t.Fatal(err)
+	}
+	now := time.Now()
+	appendFile(t, path, "")
+	f.poll(now) // empty file: format not known yet
+	appendFile(t, path, `{"msg":"one","level":"error"}`+"\n"+`{"msg":"two"}`+"\n"+`{"msg":"partial`)
+	f.poll(now.Add(time.Second))
+	got := sink.take()
+	if strings.Join(msgs(got), "|") != "one|two" || got[0].SevNum != 3 {
+		t.Fatalf("got %q", msgs(got))
+	}
+	src := f.Status()["sources"].([]map[string]any)
+	if src[0]["detected"] != fmtJSON || src[0]["open"] != true || src[1]["code"] != "not_found" {
+		t.Errorf("status %+v", src)
+	}
+
+	// Forced format: the JSON lines are read as text.
+	cfg.Sources[0].Format = fmtText
+	f.Configure(cfg)
+	appendFile(t, path, `"}`+"\n")
+	f.poll(now.Add(2 * time.Second))
+	f.poll(now.Add(5 * time.Second))
+	if got = sink.take(); strings.Join(msgs(got), "|") != `{"msg":"partial"}` {
+		t.Fatalf("forced: got %q", msgs(got))
+	}
+
+	// Turned off then on: starts over (only new lines here).
+	cfg.Sources[0].Enabled = false
+	f.Configure(cfg)
+	f.poll(now.Add(6 * time.Second))
+	appendFile(t, path, "while off\n")
+	cfg.Sources[0].Enabled = true
+	f.Configure(cfg)
+	f.poll(now.Add(7 * time.Second))
+	appendFile(t, path, "three\n")
+	f.poll(now.Add(8 * time.Second))
+	f.poll(now.Add(11 * time.Second))
+	if got = sink.take(); strings.Join(msgs(got), "|") != "three" {
+		t.Fatalf("re-enabled: got %q", msgs(got))
+	}
+}
+
+// A large access log: the last hour is found by date, not read from the
+// start, and every entry is counted from the first poll.
+func TestFilesBackfillLargeFile(t *testing.T) {
+	dir, data := t.TempDir(), t.TempDir()
+	path := filepath.Join(dir, "access.log")
+	now := time.Now()
+	line := func(at time.Time, n int) string {
+		return fmt.Sprintf(`86.246.223.62 - - [%s] "GET /wp-content/x.png HTTP/2.0" 200 640365 "-" "-" %d "vblog-io-router@file" "http://wordpress" 3ms`+"\n",
+			at.UTC().Format("02/Jan/2006:15:04:05 -0700"), n)
+	}
+	var b strings.Builder
+	const n = 200000 // one line every 3 s for a week: ~32 MB
+	for i := 0; i < n; i++ {
+		b.WriteString(line(now.Add(-time.Duration(n-i)*3*time.Second), i))
+	}
+	appendFile(t, path, b.String())
+	sink := &fileSink{}
+	f := NewFiles(dir, data, time.Hour, sink.add)
+	f.Configure(filesConfig{Sources: []fileSourceDef{{ID: "a1", Name: "traefik-log", Path: path, Enabled: true}}})
+	f.poll(now)
+	if pos := f.st.Pos["a1"]; pos.Off < int64(b.Len())*99/100 {
+		t.Errorf("history read from offset %d of %d", pos.Off, b.Len())
+	}
+	appendFile(t, path, line(now, n))
+	f.poll(now.Add(time.Second))
+	got := sink.take()
+	// The last hour (1200 lines, the oldest one on the edge) and the new line.
+	if len(got) < 1200 || len(got) > 1201 || got[0].App != "traefik-log" || got[len(got)-1].Extra["http_status"] != "200" {
+		t.Fatalf("got %d entries", len(got))
+	}
+	if st := f.Status()["sources"].([]map[string]any)[0]; st["read"] != uint64(len(got)) || st["behind"] != int64(0) || st["detected"] != fmtCLF {
+		t.Errorf("status %+v", st)
+	}
+	// Undated plain text: only new lines.
+	txt := filepath.Join(dir, "app.log")
+	appendFile(t, txt, strings.Repeat("no date here\n", 1000))
+	p := mustParser(t, fmtText, "")
+	if off := backfillStart(txt, p, 13000, now.Add(-time.Hour), now); off != 13000 {
+		t.Errorf("undated: %d", off)
+	}
+}

main.go

@@ -43,6 +43,7 @@ type config struct {
 	auditLog   string
 	auditHost  string
 	auditFill  time.Duration
+	filesFill  time.Duration
 	batchSize  int
 	queueSize  int
 	flushEvery time.Duration
@@ -125,6 +126,7 @@ func main() {
 		auditLog:   getenv("AUDIT_LOG", "/var/log/audit/audit.log"),
 		auditHost:  os.Getenv("AUDIT_HOST"),
 		auditFill:  getenvDuration("AUDIT_BACKFILL", time.Hour),
+		filesFill:  getenvDuration("FILES_BACKFILL", time.Hour),
 		batchSize:  getenvInt("BATCH_SIZE", 1000),
 		queueSize:  getenvInt("QUEUE_SIZE", 100000),
 		flushEvery: time.Duration(getenvInt("FLUSH_MS", 1000)) * time.Millisecond,
@@ -218,6 +220,10 @@ func main() {
 	// Linux audit log of the host (auditd), off until enabled in Settings > Sources.
 	api.audit = NewAudit(cfg.auditLog, cfg.hostRoot, cfg.auditHost, cfg.dataDir, cfg.auditFill, sink)
 	go api.audit.Run(ctx)
+	// Log files of other programs (Traefik, nginx…) mounted in the container,
+	// declared in Settings > Sources.
+	api.files = NewFiles(cfg.hostRoot, cfg.dataDir, cfg.filesFill, sink)
+	go api.files.Run(ctx)
 	api.selfLog = selfLog
 	api.Routes(mux)
 	mux.Handle("GET /", http.FileServer(http.FS(static)))

query.go

@@ -119,10 +119,10 @@ func (f Filter) filterExpr() string {
 		parts = append(parts, "app:="+strconv.Quote(f.App))
 	}
 	switch f.Source {
-	case "docker", "host", "kernel", "audit", "logstream":
+	case "docker", "host", "kernel", "audit", "logstream", "file":
 		parts = append(parts, `source_type:=`+strconv.Quote(f.Source))
 	case "syslog": // also matches logs stored before source_type existed
-		parts = append(parts, `!(source_type:in("docker","host","kernel","audit","logstream"))`)
+		parts = append(parts, `!(source_type:in("docker","host","kernel","audit","logstream","file"))`)
 	}
 	if f.Severity >= 0 && f.Severity < 7 {
 		names := make([]string, 0, 8)

syslog.go

@@ -36,7 +36,7 @@ type Entry struct {
 	Proto    string // udp or tcp
 	HostIP   string // original host value when it was an IP resolved through DNS
 
-	SourceType string            // "syslog", "docker", "host", "kernel", "audit" or "logstream"
+	SourceType string            // "syslog", "docker", "host", "kernel", "audit", "file" or "logstream"
 	Extra      map[string]string // additional fields (docker: container, image, …)
 
 	// Not stored. Wait: the producer can be slowed down when the queue is full

tags.go

@@ -217,6 +217,51 @@ func (s *TagStore) Delete(id string) error {
 	return errTagNotFound
 }
 
+// tagListError tells which tag of a list (PUT /api/tags) is invalid.
+type tagListError struct {
+	index int
+	err   error
+}
+
+func (e *tagListError) Error() string { return e.err.Error() }
+func (e *tagListError) Unwrap() error { return e.err }
+
+// Replace saves a whole list at once (the Save button of the settings). A tag
+// whose id is known keeps its code; the others get a new id and a free code.
+// Nothing is saved when a tag is invalid: the error gives its position.
+func (s *TagStore) Replace(in []Tag) ([]Tag, error) {
+	if len(in) > 99 {
+		return nil, errTooManyTags
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	codes := map[string]string{}
+	for _, t := range s.tags {
+		codes[t.ID] = t.Code
+	}
+	out := make([]Tag, len(in))
+	seen := map[string]bool{}
+	for i, t := range in {
+		if err := t.validate(); err != nil {
+			return nil, &tagListError{index: i, err: err}
+		}
+		if c, ok := codes[t.ID]; ok && !seen[t.ID] {
+			t.Code = c
+		} else {
+			t.ID, t.Code = newID(), ""
+		}
+		seen[t.ID] = true
+		out[i] = t
+	}
+	for i := range out {
+		if out[i].Code == "" {
+			out[i].Code = freeCode(out)
+		}
+	}
+	s.tags = out
+	return append([]Tag(nil), out...), s.save()
+}
+
 func (s *TagStore) Reset() ([]Tag, error) {
 	s.mu.Lock()
 	defer s.mu.Unlock()

tags_test.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"errors"
 	"os"
 	"path/filepath"
 	"slices"
@@ -52,3 +53,37 @@ func TestTagCodes(t *testing.T) {
 		t.Fatalf("reloaded code = %q, want 07", got)
 	}
 }
+
+func TestTagReplace(t *testing.T) {
+	s, err := LoadTagStore(filepath.Join(t.TempDir(), "tags.json"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	// Defaults: warning 01, error 02. Keep error, drop warning, add two tags
+	// (one with an unknown id, one repeating a known id).
+	got, err := s.Replace([]Tag{
+		{ID: "new1", Pattern: "a", Color: "#000000"},
+		{ID: "error", Code: "42", Pattern: "error", Color: "#111111", Enabled: true},
+		{ID: "error", Pattern: "b", Color: "#000000"},
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	codes := []string{got[0].Code, got[1].Code, got[2].Code}
+	if want := []string{"01", "02", "03"}; !slices.Equal(codes, want) {
+		t.Fatalf("codes = %v, want %v", codes, want)
+	}
+	if got[1].ID != "error" || got[0].ID == "new1" || got[2].ID == "error" || got[1].Color != "#111111" {
+		t.Fatalf("ids/fields = %+v", got)
+	}
+
+	// An invalid tag saves nothing and gives its position.
+	_, err = s.Replace([]Tag{{Pattern: "ok", Color: "#000000"}, {Pattern: "(", Regex: true, Color: "#000000"}})
+	var le *tagListError
+	if !errors.As(err, &le) || le.index != 1 {
+		t.Fatalf("err = %v, want tagListError at 1", err)
+	}
+	if n := len(s.List()); n != 3 {
+		t.Fatalf("list after a failed Replace has %d tags, want 3", n)
+	}
+}

web/app.js

@@ -40,13 +40,16 @@ const I18N = {
     copyMsg: 'Copy message', copyJson: 'Copy as JSON', copied: 'Copied',
     language: 'Language',
     tagsTitle: 'Color tags',
-    tagsHelp: 'Keywords are highlighted in messages. Changes are saved automatically. When tags overlap, the one highest in the list wins.',
+    tagsHelp: 'Keywords are highlighted in messages. Changes apply with the Save button. When tags overlap, the one highest in the list wins.',
     addTag: '+ Add tag', resetTags: 'Restore default tags',
     optWhole: 'whole word', optWholeTitle: 'Only color whole words (avoids "ok" in "token")',
     optCaseTitle: 'Match case', optRegexTitle: 'Regular expression',
     optActive: 'active', optActiveTitle: 'Enable this tag',
     colorAria: 'Color', keyword: 'keyword', preview: 'preview', del: 'Delete', noTags: 'No tags.',
     newTag: 'new',
+    saveSettings: 'Save', cancelSettings: 'Cancel',
+    settingsClean: 'No pending changes', settingsDirty: 'Unsaved changes', settingsSaved: 'Settings saved',
+    unsavedConfirm: 'Some changes have not been saved. Discard them?',
     confirmDelete: (p) => `Delete tag "${p}"?`,
     confirmReset: 'Replace all tags with the defaults (warning, error)?',
     presetAria: 'Add a preset', presetPick: '+ Preset…',
@@ -78,6 +81,35 @@ const I18N = {
     sourceAria: 'Source', srcAll: 'All sources', tabSources: 'Log sources',
     srcHost: 'Host system',
     srcKernel: 'Kernel (dmesg)',
+    srcFile: 'Files',
+    filesTitle: 'Log files',
+    addFile: '+ Add a file',
+    filesNone: 'No file yet: add one to follow the logs of Traefik, nginx or any other program.',
+    fileName: 'Name (app)',
+    filePath: '/logs/traefik/access.log',
+    fileHost: 'Host',
+    fileRegex: '(?P<time>\\S+ \\S+) \\[(?P<level>\\w+)\\] (?P<msg>.*)',
+    fileEnabledAria: 'Follow this file',
+    fmt_auto: 'Automatic',
+    fmt_text: 'Plain text',
+    fmt_json: 'JSON lines',
+    fmt_logfmt: 'logfmt (key=value)',
+    fmt_clf: 'Access log (CLF)',
+    fmt_traefik: 'Traefik access (JSON)',
+    fmt_regex: 'Regular expression',
+    fileOff: 'Off.',
+    fileDraft: 'Fill in the name and the path to save this file.',
+    fileOk: (r) => `Followed: ${r} entries since startup.`,
+    fileDetected: (f) => ` Format detected: ${f}.`,
+    fileBehind: (mb) => ` Reading the history: ${mb} MB left.`,
+    probeFound: (f) => `File found, format detected: ${f}.`,
+    probeSample: 'Last line: ',
+    file_empty: 'The file is empty for now: its format will be detected from its first lines.',
+    file_invalid: '',
+    file_not_found: 'No file at this path: mount it in the container (see docker-compose.yml) and check the path.',
+    file_permission: 'Permission denied: the file must be readable by UID 10001 or the adm group (see the README). ',
+    file_read: 'Cannot read the file: ',
+    filesHelp: 'Follows the log file of another program, mounted read-only in the container (e.g. /var/log/traefik:/logs/traefik:ro, then /logs/traefik/access.log here). Once the path is filled in, the file is checked and its format detected from its last lines (JSON lines, logfmt, access logs of nginx, Apache or Traefik, else plain text); choose a format to force it, or a regular expression whose named groups time, level, msg, host and app fill these fields, the other groups giving extra fields. Each line gives an entry (badge FILE at the start of the line, app = the name); indented lines are added to the previous entry; for access logs 5xx are errors and 4xx warnings. When turned on, the last hour of dated lines is read first (FILES_BACKFILL); rotations are followed and reading resumes where it stopped.',
     auditTitle: 'Linux audit (auditd)',
     auditEnabled: 'Collect the audit events of this machine',
     auditPath: 'File',
@@ -196,7 +228,7 @@ const I18N = {
     hCappedTitle: 'The chosen division would exceed 300 intervals over this range: it has been enlarged.',
     hUnzoom: '× Back to stream', unitDay: 'd',
     toTop: 'Back to top',
-    colRcv: 'Received', colMt: 'Message time', colSev: 'Severity', colHost: 'Host', colApp: 'App', colCodes: 'Pattern', colMsg: 'Message',
+    colSrc: 'Source', colRcv: 'Received', colMt: 'Message time', colSev: 'Severity', colHost: 'Host', colApp: 'App', colCodes: 'Pattern', colMsg: 'Message',
     codesTitle: 'Codes of the color tags found in the message',
     tagCodeTitle: 'Code shown on the log lines this tag matches',
     colGrip: 'Drag to resize, double-click for the automatic width',
@@ -240,13 +272,16 @@ const I18N = {
     copyMsg: 'Copier le message', copyJson: 'Copier en JSON', copied: 'Copié',
     language: 'Langue',
     tagsTitle: 'Tags de couleur',
-    tagsHelp: 'Les mots-clés sont surlignés dans les messages. Les modifications sont enregistrées automatiquement. En cas de chevauchement, le tag le plus haut dans la liste l\'emporte.',
+    tagsHelp: 'Les mots-clés sont surlignés dans les messages. Les modifications s\'appliquent avec le bouton Valider. En cas de chevauchement, le tag le plus haut dans la liste l\'emporte.',
     addTag: '+ Ajouter un tag', resetTags: 'Rétablir les tags par défaut',
     optWhole: 'mot entier', optWholeTitle: 'Ne colorer que les mots entiers (évite « ok » dans « token »)',
     optCaseTitle: 'Respecter la casse', optRegexTitle: 'Expression régulière',
     optActive: 'actif', optActiveTitle: 'Activer ce tag',
     colorAria: 'Couleur', keyword: 'mot-clé', preview: 'aperçu', del: 'Supprimer', noTags: 'Aucun tag.',
     newTag: 'nouveau',
+    saveSettings: 'Valider', cancelSettings: 'Annuler',
+    settingsClean: 'Aucune modification en attente', settingsDirty: 'Modifications non validées', settingsSaved: 'Paramètres enregistrés',
+    unsavedConfirm: 'Des modifications n\'ont pas été validées. Les abandonner ?',
     confirmDelete: (p) => `Supprimer le tag « ${p} » ?`,
     confirmReset: 'Remplacer tous les tags par les tags par défaut (warning, error) ?',
     presetAria: 'Ajouter un préréglage', presetPick: '+ Préréglage…',
@@ -278,6 +313,35 @@ const I18N = {
     sourceAria: 'Source', srcAll: 'Toutes les sources', tabSources: 'Sources de logs',
     srcHost: 'Système hôte',
     srcKernel: 'Noyau (dmesg)',
+    srcFile: 'Fichiers',
+    filesTitle: 'Fichiers de logs',
+    addFile: '+ Ajouter un fichier',
+    filesNone: 'Aucun fichier pour l\'instant : ajoutez-en un pour suivre les logs de Traefik, nginx ou de tout autre programme.',
+    fileName: 'Nom (app)',
+    filePath: '/logs/traefik/access.log',
+    fileHost: 'Hôte',
+    fileRegex: '(?P<time>\\S+ \\S+) \\[(?P<level>\\w+)\\] (?P<msg>.*)',
+    fileEnabledAria: 'Suivre ce fichier',
+    fmt_auto: 'Automatique',
+    fmt_text: 'Texte brut',
+    fmt_json: 'JSON lines',
+    fmt_logfmt: 'logfmt (clé=valeur)',
+    fmt_clf: 'Journal d\'accès (CLF)',
+    fmt_traefik: 'Accès Traefik (JSON)',
+    fmt_regex: 'Expression régulière',
+    fileOff: 'Désactivé.',
+    fileDraft: 'Renseignez le nom et le chemin pour enregistrer ce fichier.',
+    fileOk: (r) => `Suivi : ${r} entrées depuis le démarrage.`,
+    fileDetected: (f) => ` Format détecté : ${f}.`,
+    fileBehind: (mb) => ` Lecture de l\'historique : ${mb} Mo restants.`,
+    probeFound: (f) => `Fichier trouvé, format détecté : ${f}.`,
+    probeSample: 'Dernière ligne : ',
+    file_empty: 'Le fichier est vide pour l\'instant : son format sera détecté à partir de ses premières lignes.',
+    file_invalid: '',
+    file_not_found: 'Aucun fichier à ce chemin : montez-le dans le conteneur (voir docker-compose.yml) et vérifiez le chemin.',
+    file_permission: 'Accès refusé : le fichier doit être lisible par l\'UID 10001 ou le groupe adm (voir le README). ',
+    file_read: 'Lecture du fichier impossible : ',
+    filesHelp: 'Suit le fichier de logs d\'un autre programme, monté en lecture seule dans le conteneur (par ex. /var/log/traefik:/logs/traefik:ro, puis /logs/traefik/access.log ici). Une fois le chemin saisi, le fichier est vérifié et son format détecté à partir de ses dernières lignes (JSON lines, logfmt, journaux d\'accès de nginx, Apache ou Traefik, sinon texte brut) ; choisissez un format pour le forcer, ou une expression régulière dont les groupes nommés time, level, msg, host et app remplissent ces champs, les autres groupes donnant des champs supplémentaires. Chaque ligne donne une entrée (badge FILE en début de ligne, app = le nom) ; les lignes indentées sont rattachées à l\'entrée précédente ; pour les journaux d\'accès, 5xx en erreur et 4xx en warning. À l\'activation, la dernière heure de lignes datées est lue d\'abord (FILES_BACKFILL) ; les rotations sont suivies et la lecture reprend où elle s\'était arrêtée.',
     auditTitle: 'Audit Linux (auditd)',
     auditEnabled: 'Collecter les événements d\'audit de cette machine',
     auditPath: 'Fichier',
@@ -396,7 +460,7 @@ const I18N = {
     hCappedTitle: 'La division choisie dépasserait 300 intervalles sur cette plage : elle a été élargie.',
     hUnzoom: '× Revenir au flux', unitDay: 'j',
     toTop: 'Revenir en haut',
-    colRcv: 'Réception', colMt: 'Heure message', colSev: 'Sévérité', colHost: 'Hôte', colApp: 'App', colCodes: 'Motif de recherche', colMsg: 'Message',
+    colSrc: 'Source', colRcv: 'Réception', colMt: 'Heure message', colSev: 'Sévérité', colHost: 'Hôte', colApp: 'App', colCodes: 'Motif de recherche', colMsg: 'Message',
     codesTitle: 'Codes des tags de couleur trouvés dans le message',
     tagCodeTitle: 'Code affiché sur les lignes de log où ce tag est trouvé',
     colGrip: 'Glisser pour redimensionner, double-clic pour la largeur automatique',
@@ -529,7 +593,7 @@ function apiError(res, text, data) {
   if (data && data.code && I18N[lang]['err_' + data.code]) {
     msg = t('err_' + data.code) + (data.detail ? (lang === 'fr' ? ' : ' : ': ') + data.detail : '');
   }
-  return new Error(msg);
+  return Object.assign(new Error(msg), { data });
 }
 
 function toast(msg) {
@@ -562,6 +626,26 @@ const state = {
 };
 const recOf = new WeakMap(); // .row element -> record
 
+// Settings window: changes are shown at once but only saved by the Save button.
+const settings = {
+  saved: null,  // browser preferences as saved (see localPrefs)
+  tags: null,   // tags as saved on the server
+  pending: {},  // source changes waiting for Save, by source
+  orig: {},     // server values of the pending fields, to drop a change set back
+  saving: false,
+};
+
+// Records a source change for Save; a field set back to its saved value is dropped.
+function pend(key, change, current) {
+  const orig = settings.orig[key] ??= { ...current };
+  const p = { ...settings.pending[key], ...change };
+  for (const k of Object.keys(p)) if (p[k] === orig[k]) delete p[k];
+  if (Object.keys(p).length) settings.pending[key] = p;
+  else { delete settings.pending[key]; delete settings.orig[key]; }
+}
+// A source state refreshed from the server, with the unsaved changes over it.
+const withPending = (key, s) => (s && settings.pending[key] ? { ...s, ...settings.pending[key] } : s);
+
 const list = $('#list');
 
 /* ================= Language ================= */
@@ -578,7 +662,6 @@ function applyLang() {
 function setLang(next) {
   if (next === lang || !I18N[next]) return;
   lang = next;
-  store.set('lang', lang);
   applyLang();
   syncControls();
   setConn(...state.conn);
@@ -598,6 +681,7 @@ function setLang(next) {
   renderDmesg();
   renderAudit();
   renderSelfLog();
+  renderFiles(true);
   renderDocker();
 }
 
@@ -613,11 +697,12 @@ function effectiveTheme() {
   if (th) return th;
   return matchMedia('(prefers-color-scheme: dark)').matches ? 'dark' : 'light';
 }
-// mode: 'system' (follow the OS), 'light' or 'dark'.
-function setTheme(mode) {
+// mode: 'system' (follow the OS), 'light' or 'dark'. In Settings the theme is
+// only stored by the Save button (persist = false).
+function setTheme(mode, persist = true) {
   if (mode === 'light' || mode === 'dark') document.documentElement.dataset.theme = mode;
   else { delete document.documentElement.dataset.theme; mode = 'system'; }
-  store.set('theme', mode);
+  if (persist) store.set('theme', mode);
   renderThemeSwitch();
 }
 
@@ -632,7 +717,7 @@ function renderThemeSwitch() {
 $('#themeBtn').addEventListener('click', () => setTheme(effectiveTheme() === 'dark' ? 'light' : 'dark'));
 $('#themeSwitch').addEventListener('click', (ev) => {
   const b = ev.target.closest('[data-theme-opt]');
-  if (b) setTheme(b.dataset.themeOpt);
+  if (b) setTheme(b.dataset.themeOpt, false);
 });
 
 /* ================= Log font and size ================= */
@@ -723,20 +808,15 @@ $('#sizeSwitch').addEventListener('click', (ev) => {
   const b = ev.target.closest('[data-size]');
   if (!b) return;
   applyLogSize(b.dataset.size);
-  store.set('logSize', ui.size);
   renderInterface();
 });
 $('#densitySwitch').addEventListener('click', (ev) => {
   const b = ev.target.closest('[data-density]');
   if (!b) return;
   applyLogDensity(b.dataset.density);
-  store.set('logDensity', ui.density);
   renderInterface();
 });
-$('#fontSelect').addEventListener('change', (ev) => {
-  applyLogFont(ev.target.value);
-  store.set('logFont', ui.font);
-});
+$('#fontSelect').addEventListener('change', (ev) => applyLogFont(ev.target.value));
 
 /* ================= Settings tabs ================= */
 
@@ -905,15 +985,18 @@ function fmtTime(s) {
   return `${day} ${p.h}:${p.m}:${p.s}<span class="ms">.${p.ms}</span>`;
 }
 
-// Sources with their own look in the list: a colored edge and a badge
-// before the app name (the severity badge stays the same).
-const SRC_BADGE = { audit: 'AUDIT', logstream: 'LS' };
+// Every row starts with a badge naming its source, one color per source
+// (--src-<type> in style.css). Audit events, log files and LogStream's own
+// logs also get a colored edge and a light tint (the severity badge stays the same).
+const SRC_BADGE = { syslog: 'SYSLOG', docker: 'DOCKER', kernel: 'KERNEL', host: 'HOST', audit: 'AUDIT', file: 'FILE', logstream: 'LS' };
+const srcOf = (r) => (SRC_BADGE[r.source_type] ? r.source_type : 'syslog'); // older entries have no source_type
 
 function rowHTML(r, isNew) {
   const sev = r.severity || '';
   const cls = SEV_CLASS[sev] || 'info';
-  const badge = SRC_BADGE[r.source_type];
-  return `<article class="row sev-${cls}${badge ? ' src-' + r.source_type : ''}${isNew ? ' new' : ''}" tabindex="0">`
+  const src = srcOf(r);
+  return `<article class="row sev-${cls} src-${src}${isNew ? ' new' : ''}" tabindex="0">`
+    + `<b class="src-badge">${SRC_BADGE[src]}</b>`
     + `<time class="rcv" title="${esc(t('rcvTitle'))}">${r.received ? esc(fmtFull(r.received)) : '—'}</time>`
     + `<time class="mt" title="${esc(t('msgTimeTitle'))}">${fmtTime(r.msg_time || r._time)}</time>`
     + `<span class="sev">${esc(sev || '—')}</span>`
@@ -921,7 +1004,7 @@ function rowHTML(r, isNew) {
       ? `<span class="host" data-act="host" title="${esc(hostTitle(r) + ' · ' + t('clickHost'))}">${esc(r.host_name || r.host)}</span>`
       : '<span class="host"></span>')
     + (r.app
-      ? `<span class="app${r.source_type === 'docker' ? ' proj' : ''}" data-act="app"${r.source_type === 'docker' ? ` style="--h:${hueOf(r.compose_project || r.container || r.app)}"` : ''} title="${esc((r.compose_project ? r.compose_project + '/' : '') + (r.container ? r.container + ' · ' : '') + r.app + ' · ' + t('clickApp'))}">${r.source_type === 'docker' ? DOCKER_ICON : ''}${badge ? `<b class="src-badge">${badge}</b>` : ''}${esc(r.app)}</span>`
+      ? `<span class="app${r.source_type === 'docker' ? ' proj' : ''}" data-act="app"${r.source_type === 'docker' ? ` style="--h:${hueOf(r.compose_project || r.container || r.app)}"` : ''} title="${esc((r.compose_project ? r.compose_project + '/' : '') + (r.container ? r.container + ' · ' : '') + r.app + ' · ' + t('clickApp'))}">${r.source_type === 'docker' ? DOCKER_ICON : ''}${esc(r.app)}</span>`
       : '<span class="app"></span>')
     + `<span class="codes">${codesHTML(r._msg)}</span>`
     + `<div class="msg">${msgHTML(r)}</div>`
@@ -1025,7 +1108,7 @@ function updateCount(tookMs) {
 }
 
 function detailsHTML(r) {
-  const order = ['received', 'msg_time', '_time', 'host', 'host_name', 'host_ip', 'source_type', 'audit_type', 'audit_serial', 'auid', 'user', 'exe', 'res', 'key', 'container', 'image', 'compose_project', 'compose_service', 'stream', 'unit', 'log_file', 'subsystem', 'device', 'app', 'procid', 'severity', 'facility', 'source', 'proto', 'sd', '_msg'];
+  const order = ['received', 'msg_time', '_time', 'host', 'host_name', 'host_ip', 'source_type', 'audit_type', 'audit_serial', 'auid', 'user', 'exe', 'res', 'key', 'container', 'image', 'compose_project', 'compose_service', 'stream', 'unit', 'log_file', 'client_ip', 'http_method', 'http_host', 'http_path', 'http_status', 'duration_ms', 'router', 'subsystem', 'device', 'app', 'procid', 'severity', 'facility', 'source', 'proto', 'sd', '_msg'];
   const hidden = new Set(['_stream_id', '_stream', 'sevnum', 'audit_raw']); // audit_raw: shown under the message
   if (r.msg_time) hidden.add('_time'); // same as the reception time
   const keys = order.filter((k) => r[k] != null && r[k] !== '' && !hidden.has(k))
@@ -1690,7 +1773,6 @@ function renderHistoSettings() {
 function setHistoPref(k, v) {
   if (!HISTO_OPTS[k] || !HISTO_OPTS[k].includes(v)) return;
   histoPrefs[k] = v;
-  store.set('histo.' + k, v);
   renderHistoSettings();
   if (k === 'step') refreshHisto();
   else if (k === 'refresh') { scheduleHisto(); if (v !== 'off') refreshHisto(); }
@@ -1901,8 +1983,6 @@ function updateTimePreview() {
 function onTimePrefsChange() {
   timePrefs.tz = $('#tzSelect').value;
   timePrefs.fmt = $('#fmtSelect').value;
-  store.set('tz', timePrefs.tz);
-  store.set('timefmt', timePrefs.fmt);
   updateTimePreview();
   renderList();
   renderPeriod();
@@ -1970,7 +2050,7 @@ document.addEventListener('keydown', (ev) => { if (ev.key === 'Escape') setExpor
 let syslogState = null;
 
 async function loadSyslog() {
-  try { syslogState = await api('/api/syslog'); } catch { syslogState = null; }
+  try { syslogState = withPending('syslog', await api('/api/syslog')); } catch { syslogState = null; }
   renderSyslog();
 }
 
@@ -2009,12 +2089,10 @@ function renderSyslog() {
   }
 }
 
-async function configureSyslog(change) {
+function configureSyslog(change) {
   const s = syslogState || { enabled: true, udp: true, tcp: true };
-  const body = { enabled: s.enabled, udp: s.udp, tcp: s.tcp, ...change };
-  syslogState = { ...s, ...body };
-  renderSyslog();
-  try { syslogState = await api('/api/syslog', { method: 'PUT', body }); } catch (e) { toast(e.message); }
+  pend('syslog', change, s);
+  syslogState = { ...s, ...change };
   renderSyslog();
 }
 
@@ -2030,7 +2108,7 @@ $('#syslogProtos').addEventListener('click', (ev) => {
 let hostState = null;
 
 async function loadHost() {
-  try { hostState = await api('/api/hostlogs'); } catch { hostState = null; }
+  try { hostState = withPending('host', await api('/api/hostlogs')); } catch { hostState = null; }
   renderHost();
 }
 
@@ -2057,12 +2135,10 @@ function renderHost() {
   status.className = 'docker-status ' + cls;
 }
 
-async function configureHost(enabled) {
+function configureHost(enabled) {
+  pend('host', { enabled }, hostState || {});
   hostState = { ...(hostState || {}), enabled };
   renderHost();
-  try { hostState = await api('/api/hostlogs', { method: 'PUT', body: { enabled } }); } catch (e) { toast(e.message); }
-  renderHost();
-  setTimeout(loadHost, 1500); // the first scan runs in the background
 }
 
 $('#hostEnabled').addEventListener('change', (ev) => configureHost(ev.target.checked));
@@ -2072,7 +2148,7 @@ $('#hostEnabled').addEventListener('change', (ev) => configureHost(ev.target.che
 let dmesgState = null;
 
 async function loadDmesg() {
-  try { dmesgState = await api('/api/dmesg'); } catch { dmesgState = null; }
+  try { dmesgState = withPending('dmesg', await api('/api/dmesg')); } catch { dmesgState = null; }
   renderDmesg();
 }
 
@@ -2096,12 +2172,10 @@ function renderDmesg() {
   status.className = 'docker-status ' + cls;
 }
 
-async function configureDmesg(enabled) {
+function configureDmesg(enabled) {
+  pend('dmesg', { enabled }, dmesgState || {});
   dmesgState = { ...(dmesgState || {}), enabled };
   renderDmesg();
-  try { dmesgState = await api('/api/dmesg', { method: 'PUT', body: { enabled } }); } catch (e) { toast(e.message); }
-  renderDmesg();
-  setTimeout(loadDmesg, 1500); // the device is opened in the background
 }
 
 $('#dmesgEnabled').addEventListener('change', (ev) => configureDmesg(ev.target.checked));
@@ -2111,7 +2185,7 @@ $('#dmesgEnabled').addEventListener('change', (ev) => configureDmesg(ev.target.c
 let auditState = null;
 
 async function loadAudit() {
-  try { auditState = await api('/api/audit'); } catch { auditState = null; }
+  try { auditState = withPending('audit', await api('/api/audit')); } catch { auditState = null; }
   renderAudit();
 }
 
@@ -2138,13 +2212,10 @@ function renderAudit() {
   status.className = 'docker-status ' + cls;
 }
 
-async function configureAudit(change) {
-  const body = { enabled: auditState?.enabled || false, path: auditState?.path || '', ...change };
-  auditState = { ...(auditState || {}), ...body };
+function configureAudit(change) {
+  pend('audit', change, { enabled: false, path: '', ...auditState });
+  auditState = { ...(auditState || {}), ...change };
   renderAudit();
-  try { auditState = await api('/api/audit', { method: 'PUT', body }); } catch (e) { toast(e.message); }
-  renderAudit();
-  setTimeout(loadAudit, 1500); // the file is read in the background
 }
 
 $('#auditEnabled').addEventListener('change', (ev) => configureAudit({ enabled: ev.target.checked }));
@@ -2155,7 +2226,7 @@ $('#auditPath').addEventListener('change', (ev) => configureAudit({ path: ev.tar
 let selfState = null;
 
 async function loadSelfLog() {
-  try { selfState = await api('/api/selflog'); } catch { selfState = null; }
+  try { selfState = withPending('selflog', await api('/api/selflog')); } catch { selfState = null; }
   renderSelfLog();
 }
 
@@ -2171,17 +2242,166 @@ function renderSelfLog() {
   status.className = 'docker-status ' + (s.enabled ? 'good' : 'muted');
 }
 
-async function configureSelfLog(change) {
-  const body = { enabled: selfState?.enabled ?? true, min_level: selfState?.min_level || 'warning', ...change };
-  selfState = { ...(selfState || {}), ...body };
-  renderSelfLog();
-  try { selfState = await api('/api/selflog', { method: 'PUT', body }); } catch (e) { toast(e.message); }
+function configureSelfLog(change) {
+  pend('selflog', change, { enabled: true, min_level: 'warning', ...selfState });
+  selfState = { ...(selfState || {}), ...change };
   renderSelfLog();
 }
 
 $('#selfEnabled').addEventListener('change', (ev) => configureSelfLog({ enabled: ev.target.checked }));
 $('#selfLevel').addEventListener('change', (ev) => configureSelfLog({ min_level: ev.target.value }));
 
+/* ================= Log files ================= */
+
+const FILE_FORMATS = ['auto', 'text', 'json', 'logfmt', 'clf', 'traefik', 'regex'];
+let filesState = null; // {sources, defaultHost} from the server
+let fileDrafts = [];   // rows added here, saved once they have a name and a path
+let fileSeq = 0;
+const fileProbes = new Map(); // path -> result of /api/files/detect (format found, last line)
+
+const FILE_FIELDS = ['id', 'name', 'path', 'format', 'regex', 'host', 'enabled'];
+
+// full: rebuild the rows (after Save or Cancel).
+async function loadFiles(full) {
+  let fresh;
+  try { fresh = await api('/api/files'); } catch { filesState = null; renderFiles(full); return; }
+  if (settings.pending.files && filesState) {
+    // Unsaved edits: keep the rows as edited, with the server status of each file.
+    const byId = new Map(fresh.sources.map((x) => [x.id, x]));
+    filesState = { ...fresh, sources: filesState.sources.map((x) => ({ ...byId.get(x.id), ...Object.fromEntries(FILE_FIELDS.map((k) => [k, x[k]])) })) };
+  } else filesState = fresh;
+  renderFiles(full);
+}
+
+// Status line of a row: the check of the path just typed, else the state
+// of the file once saved.
+function fileStatus(s) {
+  const pr = fileProbes.get(s.path);
+  if (pr && (s.draft || (s.enabled && !s.open && !s.code))) {
+    if (pr.code) return [t('file_' + pr.code) + (pr.code === 'not_found' || pr.code === 'empty' ? '' : (pr.code === 'invalid' ? '' : ' ') + (pr.error || '')), pr.code === 'empty' ? 'muted' : 'bad'];
+    return [t('probeFound', t('fmt_' + pr.format)), 'good'];
+  }
+  if (s.draft) return [t('fileDraft'), 'muted'];
+  if (!s.enabled) return [t('fileOff'), 'muted'];
+  if (s.code) return [t('file_' + s.code) + (s.code === 'not_found' ? '' : s.error || ''), 'bad'];
+  if (s.open) {
+    return [t('fileOk', fmtNum(s.read)) + (s.format === 'auto' && s.detected ? t('fileDetected', t('fmt_' + s.detected)) : '')
+      + (s.behind > 1 << 20 ? t('fileBehind', fmtNum(Math.round(s.behind / (1 << 20)))) : ''), 'good'];
+  }
+  return [t('hostWaiting'), 'muted'];
+}
+
+function fileSample(s) {
+  const pr = fileProbes.get(s.path);
+  return pr && pr.sample ? t('probeSample') + pr.sample : '';
+}
+
+// Checks the path typed in a row: found or not, detected format, last line.
+async function probeFile(path) {
+  if (!path) return;
+  try {
+    fileProbes.set(path, await api('/api/files/detect', { method: 'POST', body: { path } }));
+  } catch { fileProbes.delete(path); }
+  renderFiles();
+}
+
+function fileRowHTML(s) {
+  const key = s.id || s.draft;
+  const [text, cls] = fileStatus(s);
+  return `<div class="file-row${s.enabled ? '' : ' off'}" data-key="${esc(key)}">
+    <input type="checkbox" class="switch" data-f="enabled"${s.enabled ? ' checked' : ''} aria-label="${esc(t('fileEnabledAria'))}">
+    <input type="text" data-f="name" value="${esc(s.name)}" placeholder="${esc(t('fileName'))}" aria-label="${esc(t('fileName'))}" spellcheck="false" maxlength="64">
+    <input type="text" data-f="path" class="mono" value="${esc(s.path)}" placeholder="${esc(t('filePath'))}" aria-label="${esc(t('filePath'))}" spellcheck="false">
+    <select class="field" data-f="format">${FILE_FORMATS.map((f) => `<option value="${f}"${s.format === f ? ' selected' : ''}>${esc(t('fmt_' + f))}</option>`).join('')}</select>
+    <input type="text" data-f="host" value="${esc(s.host)}" placeholder="${esc(filesState?.defaultHost || t('fileHost'))}" aria-label="${esc(t('fileHost'))}" title="${esc(t('fileHost'))}" spellcheck="false">
+    <button class="icon-btn danger" type="button" data-del title="${esc(t('del'))}" aria-label="${esc(t('del'))}">
+      <svg viewBox="0 0 24 24"><path d="M4 7h16M10 11v6M14 11v6M6 7l1 13h10l1-13M9 7V4h6v3"/></svg>
+    </button>
+    <input type="text" data-f="regex" class="mono fr-regex" value="${esc(s.regex)}" placeholder="${esc(t('fileRegex'))}" aria-label="${esc(t('fmt_regex'))}" spellcheck="false"${s.format === 'regex' ? '' : ' hidden'}>
+    <p class="fr-status docker-status ${cls}">${esc(text)}</p>
+    <p class="fr-sample"${fileSample(s) ? '' : ' hidden'}>${esc(fileSample(s))}</p>
+  </div>`;
+}
+
+// full: rebuild the rows even while one is being edited (language change).
+function renderFiles(full) {
+  const list = $('#fileList');
+  if (!filesState) { list.innerHTML = ''; return; }
+  const rows = filesState.sources.concat(fileDrafts);
+  const keys = rows.map((s) => s.id || s.draft).join(',');
+  if (!full && list.dataset.keys === keys && list.contains(document.activeElement)) {
+    // Being edited: only refresh the status lines.
+    for (const s of rows) {
+      const row = list.querySelector(`.file-row[data-key="${CSS.escape(s.id || s.draft)}"]`);
+      if (!row) continue;
+      const [text, cls] = fileStatus(s);
+      const p = row.querySelector('.fr-status');
+      p.textContent = text;
+      p.className = 'fr-status docker-status ' + cls;
+      const sample = fileSample(s);
+      row.querySelector('.fr-sample').textContent = sample;
+      row.querySelector('.fr-sample').hidden = !sample;
+    }
+    return;
+  }
+  list.dataset.keys = keys;
+  list.innerHTML = rows.map(fileRowHTML).join('') || `<p class="muted small">${esc(t('filesNone'))}</p>`;
+}
+
+function fileRowOf(key) {
+  return filesState?.sources.find((s) => s.id === key) || fileDrafts.find((s) => s.draft === key);
+}
+
+async function saveFiles() {
+  const ready = fileDrafts.filter((d) => d.name.trim() && d.path.trim());
+  const sources = filesState.sources.concat(ready).map(({ id, name, path, format, regex, host, enabled }) => ({ id: id || '', name, path, format, regex, host, enabled }));
+  filesState = await api('/api/files', { method: 'PUT', body: { sources } });
+  fileDrafts = fileDrafts.filter((d) => !ready.includes(d));
+  renderFiles(true);
+  setTimeout(loadFiles, 1500); // the files are read in the background
+}
+
+$('#fileList').addEventListener('change', (ev) => {
+  const row = ev.target.closest('.file-row');
+  const s = row && fileRowOf(row.dataset.key);
+  const f = ev.target.dataset.f;
+  if (!s || !f) return;
+  s[f] = f === 'enabled' ? ev.target.checked : ev.target.value.trim();
+  if (f === 'path') {
+    // A name by default: the file name without its extension.
+    if (!s.name && s.path) {
+      s.name = s.path.split('/').pop().replace(/\.(log|txt|json)$/i, '');
+      row.querySelector('[data-f="name"]').value = s.name;
+    }
+    probeFile(s.path);
+  }
+  if (f === 'format') row.querySelector('.fr-regex').hidden = s.format !== 'regex';
+  if (f === 'format' && s.format === 'regex' && !s.regex) { row.querySelector('.fr-regex').focus(); return; }
+  if (s.draft && !(s.name && s.path)) return;
+  settings.pending.files = true;
+});
+
+$('#fileList').addEventListener('click', (ev) => {
+  const btn = ev.target.closest('[data-del]');
+  if (!btn) return;
+  const key = btn.closest('.file-row').dataset.key;
+  if (fileDrafts.some((d) => d.draft === key)) {
+    fileDrafts = fileDrafts.filter((d) => d.draft !== key);
+    renderFiles(true);
+    return;
+  }
+  filesState.sources = filesState.sources.filter((s) => s.id !== key);
+  settings.pending.files = true;
+  renderFiles(true);
+});
+
+$('#addFile').addEventListener('click', () => {
+  if (!filesState) return;
+  fileDrafts.push({ draft: 'new' + (++fileSeq), name: '', path: '', format: 'auto', regex: '', host: '', enabled: true });
+  renderFiles(true);
+  $('#fileList').lastElementChild.querySelector('[data-f="path"]').focus();
+});
+
 /* ================= Docker source ================= */
 
 let dockerState = null;
@@ -2190,9 +2410,18 @@ const LOCK_ICON = '<svg viewBox="0 0 24 24" aria-hidden="true"><rect x="5" y="11
 
 async function loadDocker() {
   try { dockerState = await api('/api/docker'); } catch (e) { dockerState = { enabled: true, connected: false, error: e.message }; }
+  applyDockerPending();
   renderDocker();
 }
 
+// Unsaved choices stay shown over the state refreshed from the server.
+function applyDockerPending() {
+  const p = settings.pending.docker;
+  if (!p || !dockerState) return;
+  if ('defaultEnabled' in p) dockerState.defaultEnabled = p.defaultEnabled;
+  for (const c of dockerState.containers || []) if (c.key in p.containers) c.enabled = p.containers[c.key];
+}
+
 // One label per choice key ("project/service"): replicas of a service share it.
 function dockerChips() {
   const byKey = new Map();
@@ -2255,12 +2484,28 @@ function renderDocker() {
     : `<span class="chip-empty">${esc(t('dockerNoMatch'))}</span>`;
 }
 
-async function configureDocker(body) {
-  try {
-    dockerState = await api('/api/docker', { method: 'PUT', body });
-    renderDocker();
-    setTimeout(loadDocker, 1500); // followers start or stop in the background
-  } catch (e) { toast(e.message); loadDocker(); }
+// Kept until Save; a choice set back to its saved value is dropped.
+function configureDocker(change) {
+  if (!dockerState) return;
+  const orig = settings.orig.docker ??= {
+    defaultEnabled: !!dockerState.defaultEnabled,
+    containers: Object.fromEntries(dockerChips().map((c) => [c.key, !!c.enabled])),
+  };
+  const p = settings.pending.docker || { containers: {} };
+  if ('defaultEnabled' in change) {
+    if (change.defaultEnabled === orig.defaultEnabled) delete p.defaultEnabled;
+    else p.defaultEnabled = change.defaultEnabled;
+  }
+  for (const [k, on] of Object.entries(change.containers || {})) {
+    if (orig.containers[k] === on) delete p.containers[k];
+    else p.containers[k] = on;
+  }
+  if ('defaultEnabled' in p || Object.keys(p.containers).length) settings.pending.docker = p;
+  else { delete settings.pending.docker; delete settings.orig.docker; }
+  // Shown at once: the change itself, even when it brings back the saved value.
+  if ('defaultEnabled' in change) dockerState.defaultEnabled = change.defaultEnabled;
+  for (const c of dockerState.containers || []) if (c.key in (change.containers || {})) c.enabled = change.containers[c.key];
+  renderDocker();
 }
 
 $('#dockerDefault').addEventListener('change', (ev) => configureDocker({ defaultEnabled: ev.target.checked }));
@@ -2273,12 +2518,7 @@ $('#dockerFilter').addEventListener('input', (ev) => { dockerView.filter = ev.ta
 $('#dockerList').addEventListener('click', (ev) => {
   const chip = ev.target.closest('.chip[data-key]');
   if (!chip || chip.disabled) return;
-  const key = chip.dataset.key;
-  const on = !chip.classList.contains('on');
-  // Immediate feedback, confirmed by the server answer.
-  for (const c of (dockerState && dockerState.containers) || []) if (c.key === key) c.enabled = on;
-  renderDocker();
-  configureDocker({ containers: { [key]: on } });
+  configureDocker({ containers: { [chip.dataset.key]: !chip.classList.contains('on') } });
 });
 for (const [id, on] of [['#dockerAll', true], ['#dockerNone', false]]) {
   $(id).addEventListener('click', () => {
@@ -2301,9 +2541,11 @@ setInterval(() => {
     loadDmesg();
     loadAudit();
     loadSelfLog();
+    loadFiles();
   }
 }, 4000);
 
+
 /* ================= Database statistics ================= */
 
 // 1536 -> "1,5 Ko" / "1.5 KB" (binary multiples, the usual reading for disk sizes).
@@ -2499,19 +2741,10 @@ function filterTagList() {
 }
 $('#tagFilter').addEventListener('input', filterTagList);
 
-const saveTimers = {};
-function scheduleSave(tag, rowEl) {
-  clearTimeout(saveTimers[tag.id]);
-  saveTimers[tag.id] = setTimeout(async () => {
-    const errEl = rowEl.querySelector('.err');
-    try {
-      await api('/api/tags/' + encodeURIComponent(tag.id), { method: 'PUT', body: tag });
-      errEl.textContent = '';
-    } catch (e) {
-      errEl.textContent = e.message;
-    }
-  }, 400);
-}
+// Tags are edited here and sent as a whole list by Save (PUT /api/tags); a new
+// tag gets its id and code from the server then.
+let tagSeq = 0;
+const newTag = (fields) => ({ code: '', label: '', wholeWord: false, caseSensitive: false, regex: false, enabled: true, ...fields, id: 'new-' + (++tagSeq) });
 
 const applyTags = debounce(() => { compileMatchers(); rehighlight(); }, 150);
 
@@ -2526,52 +2759,43 @@ $('#tagList').addEventListener('input', (ev) => {
   const mark = rowEl.querySelector('.preview mark');
   mark.textContent = tag.label || tag.pattern || t('preview');
   mark.setAttribute('style', tagStyle(tag.color));
-  scheduleSave(tag, rowEl);
+  rowEl.querySelector('.err').textContent = '';
   applyTags();
 });
 
-$('#tagList').addEventListener('click', async (ev) => {
+$('#tagList').addEventListener('click', (ev) => {
   const del = ev.target.closest('[data-del]');
   if (!del) return;
   const rowEl = del.closest('.tag-row');
   const tag = state.tags.find((x) => x.id === rowEl.dataset.id);
   if (!tag || !confirm(t('confirmDelete', tag.pattern))) return;
-  try {
-    await api('/api/tags/' + encodeURIComponent(tag.id), { method: 'DELETE' });
-    state.tags = state.tags.filter((x) => x !== tag);
-    renderTagList();
-    applyTags();
-  } catch (e) { toast(e.message); }
+  state.tags = state.tags.filter((x) => x !== tag);
+  renderTagList();
+  applyTags();
+  renderSaveState();
 });
 
-$('#addTag').addEventListener('click', async () => {
-  const color = PALETTE[state.tags.length % PALETTE.length];
-  try {
-    const tag = await api('/api/tags', { method: 'POST', body: { pattern: t('newTag'), color, wholeWord: true, enabled: true } });
-    state.tags.push(tag);
-    $('#tagFilter').value = '';
-    renderTagList();
-    applyTags();
-    const input = $('#tagList').lastElementChild.querySelector('input[type="text"]');
-    input.focus();
-    input.select();
-  } catch (e) { toast(e.message); }
+$('#addTag').addEventListener('click', () => {
+  state.tags.push(newTag({ pattern: t('newTag'), color: PALETTE[state.tags.length % PALETTE.length], wholeWord: true }));
+  $('#tagFilter').value = '';
+  renderTagList();
+  applyTags();
+  const input = $('#tagList').lastElementChild.querySelector('input[type="text"]');
+  input.focus();
+  input.select();
 });
 
 // Adds a preset group, skipping tags whose pattern is already in the list.
-$('#presetTags').addEventListener('change', async (ev) => {
+$('#presetTags').addEventListener('change', (ev) => {
   const preset = state.presets.flatMap((g) => g.presets).find((p) => p.id === ev.target.value);
   ev.target.value = '';
   if (!preset) return;
   let added = 0;
-  try {
-    for (const p of preset.tags) {
-      if (state.tags.some((x) => x.pattern === p.pattern)) continue;
-      const tag = { ...p, label: pickText(p.label), regex: p.regex ?? true, enabled: true };
-      state.tags.push(await api('/api/tags', { method: 'POST', body: tag }));
-      added++;
-    }
-  } catch (e) { toast(e.message); }
+  for (const p of preset.tags) {
+    if (state.tags.some((x) => x.pattern === p.pattern)) continue;
+    state.tags.push(newTag({ ...p, label: pickText(p.label), regex: p.regex ?? true, enabled: true }));
+    added++;
+  }
   renderTagList();
   applyTags();
   toast(t('presetAdded', added));
@@ -2580,13 +2804,16 @@ $('#presetTags').addEventListener('change', async (ev) => {
 $('#resetTags').addEventListener('click', async () => {
   if (!confirm(t('confirmReset'))) return;
   try {
-    state.tags = await api('/api/tags/reset', { method: 'POST' });
+    state.tags = await api('/api/tags/defaults');
     renderTagList();
     applyTags();
   } catch (e) { toast(e.message); }
+  renderSaveState();
 });
 
 $('#settingsBtn').addEventListener('click', () => {
+  settingsBaseline();
+  renderSaveState();
   renderTimeSettings();
   renderTagList();
   loadPresets();
@@ -2597,14 +2824,163 @@ $('#settingsBtn').addEventListener('click', () => {
   loadDmesg();
   loadAudit();
   loadSelfLog();
+  loadFiles();
   loadDocker();
   renderDbStats();
   loadDbStats();
   showSettingsTab(store.get('settingsTab', 'locale'));
   $('#settingsDlg').showModal();
 });
-$('#closeSettings').addEventListener('click', () => $('#settingsDlg').close());
-$('#settingsDlg').addEventListener('click', (ev) => { if (ev.target === ev.currentTarget) ev.currentTarget.close(); });
+/* ================= Settings: Save / Cancel ================= */
+
+// Browser preferences of the Settings window (stored in localStorage by Save).
+function localPrefs() {
+  return {
+    lang, theme: document.documentElement.dataset.theme || 'system',
+    font: ui.font, size: ui.size, density: ui.density, tz: timePrefs.tz, fmt: timePrefs.fmt, histo: { ...histoPrefs },
+  };
+}
+
+const tagFields = ({ id, pattern, label, color, wholeWord, caseSensitive, regex, enabled }) =>
+  ({ id, pattern, label: label || '', color, wholeWord: !!wholeWord, caseSensitive: !!caseSensitive, regex: !!regex, enabled: !!enabled });
+const tagsJSON = (tags) => JSON.stringify(tags.map(tagFields));
+
+// What Cancel goes back to: the state when the window was opened or last saved.
+function settingsBaseline() {
+  settings.saved = localPrefs();
+  settings.tags = structuredClone(state.tags);
+  settings.pending = {};
+  settings.orig = {};
+}
+
+function settingsDirty() {
+  return JSON.stringify(localPrefs()) !== JSON.stringify(settings.saved)
+    || tagsJSON(state.tags) !== tagsJSON(settings.tags)
+    || Object.keys(settings.pending).length > 0;
+}
+
+function renderSaveState() {
+  if (!settings.saved) return;
+  const dirty = settingsDirty();
+  $('#setActions').classList.toggle('dirty', dirty);
+  $('#setSave').disabled = !dirty || settings.saving;
+  $('#setCancel').disabled = !dirty || settings.saving;
+  $('#setState').textContent = t(dirty ? 'settingsDirty' : 'settingsClean');
+}
+
+// Each source sends its whole setting, as edited.
+const SOURCE_SAVE = {
+  syslog: async () => { syslogState = await api('/api/syslog', { method: 'PUT', body: { enabled: syslogState.enabled, udp: syslogState.udp, tcp: syslogState.tcp } }); },
+  host: async () => { hostState = await api('/api/hostlogs', { method: 'PUT', body: { enabled: hostState.enabled } }); },
+  dmesg: async () => { dmesgState = await api('/api/dmesg', { method: 'PUT', body: { enabled: dmesgState.enabled } }); },
+  audit: async () => { auditState = await api('/api/audit', { method: 'PUT', body: { enabled: !!auditState.enabled, path: auditState.path || '' } }); },
+  selflog: async () => { selfState = await api('/api/selflog', { method: 'PUT', body: { enabled: selfState.enabled, min_level: selfState.min_level || 'warning' } }); },
+  docker: async () => { dockerState = await api('/api/docker', { method: 'PUT', body: settings.pending.docker }); },
+  files: saveFiles,
+};
+
+async function saveSettings() {
+  settings.saving = true;
+  renderSaveState();
+  const p = localPrefs();
+  for (const [k, v] of [['lang', p.lang], ['theme', p.theme], ['logFont', p.font], ['logSize', p.size], ['logDensity', p.density], ['tz', p.tz], ['timefmt', p.fmt]]) store.set(k, v);
+  for (const [k, v] of Object.entries(p.histo)) store.set('histo.' + k, v);
+  settings.saved = p;
+
+  const errors = [];
+  if (tagsJSON(state.tags) !== tagsJSON(settings.tags)) {
+    try {
+      state.tags = await api('/api/tags', { method: 'PUT', body: state.tags.map(tagFields) });
+      settings.tags = structuredClone(state.tags);
+      renderTagList();
+      applyTags();
+    } catch (e) {
+      errors.push(e.message);
+      const row = $('#tagList').querySelectorAll('.tag-row')[e.data?.index];
+      if (row) {
+        showSettingsTab('filters');
+        row.hidden = false;
+        row.querySelector('.err').textContent = e.message;
+        row.scrollIntoView({ block: 'nearest' });
+      }
+    }
+  }
+  for (const key of Object.keys(settings.pending)) {
+    try {
+      await SOURCE_SAVE[key]();
+      delete settings.pending[key];
+      delete settings.orig[key];
+    } catch (e) { errors.push(e.message); }
+  }
+  // Sources start or stop in the background: show their state a bit later.
+  setTimeout(reloadSources, 1500);
+  reloadSources();
+  settings.saving = false;
+  renderSaveState();
+  toast(errors.length ? errors.join(' · ') : t('settingsSaved'));
+}
+
+function reloadSources() {
+  loadSyslog();
+  loadHost();
+  loadDmesg();
+  loadAudit();
+  loadSelfLog();
+  loadFiles(true);
+  loadDocker();
+}
+
+function cancelSettings() {
+  const p = settings.saved;
+  setLang(p.lang);
+  setTheme(p.theme, false);
+  applyLogFont(p.font);
+  applyLogSize(p.size);
+  applyLogDensity(p.density);
+  if (p.tz !== timePrefs.tz || p.fmt !== timePrefs.fmt) {
+    Object.assign(timePrefs, { tz: p.tz, fmt: p.fmt });
+    renderList();
+    renderPeriod();
+    refreshHisto();
+  }
+  for (const [k, v] of Object.entries(p.histo)) if (histoPrefs[k] !== v) setHistoPref(k, v);
+  renderInterface();
+  renderTimeSettings();
+
+  state.tags = structuredClone(settings.tags);
+  renderTagList();
+  applyTags();
+
+  settings.pending = {};
+  settings.orig = {};
+  fileDrafts = [];
+  reloadSources();
+  renderSaveState();
+}
+
+// Closing with unsaved changes asks first, then drops them.
+function closeSettings() {
+  if (settingsDirty()) {
+    if (!confirm(t('unsavedConfirm'))) return;
+    cancelSettings();
+  }
+  $('#settingsDlg').close();
+}
+
+$('#setSave').addEventListener('click', saveSettings);
+$('#setCancel').addEventListener('click', cancelSettings);
+$('#closeSettings').addEventListener('click', closeSettings);
+$('#settingsDlg').addEventListener('click', (ev) => { if (ev.target === ev.currentTarget) closeSettings(); });
+$('#settingsDlg').addEventListener('cancel', (ev) => { ev.preventDefault(); closeSettings(); }); // Escape
+// Closed anyway (a second Escape in some browsers): the changes are not kept.
+$('#settingsDlg').addEventListener('close', (ev) => { if (!ev.currentTarget.open && settingsDirty()) cancelSettings(); });
+// Every control of the window: refresh the Save button once the change is applied.
+for (const type of ['input', 'change', 'click']) {
+  $('#settingsDlg').addEventListener(type, () => setTimeout(renderSaveState));
+}
+addEventListener('beforeunload', (ev) => {
+  if ($('#settingsDlg').open && settingsDirty()) ev.preventDefault();
+});
 
 /* ================= Startup ================= */
 

web/index.html

@@ -105,6 +105,7 @@
       <option value="host" data-i18n="srcHost">Host system</option>
       <option value="kernel" data-i18n="srcKernel">Kernel (dmesg)</option>
       <option value="audit">Audit</option>
+      <option value="file" data-i18n="srcFile">Files</option>
       <option value="logstream">LogStream</option>
     </select>
     <span class="spacer"></span>
@@ -139,6 +140,7 @@
 
   <div id="table" class="table" hidden>
     <div id="listHead" class="list-head" role="presentation">
+      <span data-col="src"><span class="lbl" data-i18n="colSrc">Source</span></span>
       <span data-col="rcv"><span class="lbl" data-i18n="colRcv">Received</span><i class="grip" data-i18n-title="colGrip"></i></span>
       <span data-col="mt"><span class="lbl" data-i18n="colMt">Message time</span><i class="grip" data-i18n-title="colGrip"></i></span>
       <span data-col="sev"><span class="lbl" data-i18n="colSev">Severity</span><i class="grip" data-i18n-title="colGrip"></i></span>
@@ -173,6 +175,7 @@
       </header>
 
       <div class="set-body">
+        <div class="set-side">
         <nav id="setNav" class="set-nav" role="tablist" aria-orientation="vertical">
           <button type="button" role="tab" data-tab="locale">
             <svg viewBox="0 0 24 24"><circle cx="12" cy="12" r="9"/><path d="M3 12h18M12 3a14 14 0 0 1 0 18M12 3a14 14 0 0 0 0 18"/></svg>
@@ -195,6 +198,15 @@
             <span data-i18n="tabData">Database</span>
           </button>
         </nav>
+        <div id="setActions" class="set-actions">
+          <button id="setSave" class="btn save" type="button" disabled>
+            <svg viewBox="0 0 24 24" aria-hidden="true"><path d="M5 12.5 10 17l9-10"/></svg>
+            <span data-i18n="saveSettings">Save</span>
+          </button>
+          <button id="setCancel" class="btn ghost" type="button" disabled data-i18n="cancelSettings">Cancel</button>
+          <p id="setState" class="set-state" role="status"></p>
+        </div>
+        </div>
 
         <div class="set-panels">
           <!-- Localization -->
@@ -312,6 +324,15 @@
               <p class="muted small src-help" data-i18n="selfHelp"></p>
             </section>
 
+            <section class="set-section span">
+              <h3 data-i18n="filesTitle">Log files</h3>
+              <div id="fileList" class="file-list"></div>
+              <div class="file-toolbar">
+                <button id="addFile" class="btn primary" type="button" data-i18n="addFile">+ Add a file</button>
+              </div>
+              <p class="muted small src-help" data-i18n="filesHelp"></p>
+            </section>
+
             <section class="set-section span">
               <h3 data-i18n="dockerTitle">Docker containers</h3>
               <p id="dockerStatus" class="docker-status"></p>

web/style.css

@@ -22,8 +22,13 @@
   --hit: rgba(99, 102, 241, .18);
   --shadow: 0 10px 30px rgba(15, 23, 42, .12);
   --chip-l: 38%;            /* lightness of project-colored text (labels, Docker apps) */
+  --src-syslog: #64748b;    /* source badges: syslog received over the network */
+  --src-docker: #1d6fd6;    /* Docker containers */
+  --src-kernel: #b4540a;    /* kernel ring buffer (dmesg) */
+  --src-host: #b0407a;      /* /var/log files of the host system */
   --src-audit: #7c4dbd;     /* audit events (auditd) */
   --src-ls: #1f9aa3;        /* LogStream internal logs */
+  --src-file: #5f8a1f;      /* log files (Settings > Sources > Log files) */
   --src-tint: 6%;           /* background tint of their rows */
 
   --sev-crit: #be123c;
@@ -73,8 +78,13 @@
   --hit: rgba(129, 140, 248, .28);
   --shadow: 0 10px 30px rgba(0, 0, 0, .5);
   --chip-l: 70%;
+  --src-syslog: #94a3b8;
+  --src-docker: #5aa2f5;
+  --src-kernel: #e7a25a;
+  --src-host: #e58bbb;
   --src-audit: #a983e6;
   --src-ls: #3cc3cc;
+  --src-file: #9ccc54;
   --src-tint: 9%;
   --sev-crit: #fb7185;
   --sev-err: #f87171;
@@ -108,8 +118,13 @@
     --hit: rgba(129, 140, 248, .28);
     --shadow: 0 10px 30px rgba(0, 0, 0, .5);
   --chip-l: 70%;
+    --src-syslog: #94a3b8;
+    --src-docker: #5aa2f5;
+    --src-kernel: #e7a25a;
+    --src-host: #e58bbb;
     --src-audit: #a983e6;
     --src-ls: #3cc3cc;
+    --src-file: #9ccc54;
     --src-tint: 9%;
     --sev-crit: #fb7185;
     --sev-err: #f87171;
@@ -340,12 +355,12 @@ body.busy .progress::after {
 .list { margin: 6px 20px 0; background: var(--panel); border: 1px solid var(--border); border-radius: var(--radius); overflow: hidden; }
 .list:empty { display: none; }
 
-/* Columns: received, message time, severity, host, app, tag codes, message. Widths come from
+/* Columns: source, received, message time, severity, host, app, tag codes, message. Widths come from
    --col-* (set on #table when a column has been resized, see app.js), shared by
    the header and every row through subgrid so that the columns line up. */
 .table {
   display: grid;
-  grid-template-columns: var(--col-rcv, max-content) var(--col-mt, max-content) var(--col-sev, 4.6rem)
+  grid-template-columns: minmax(3.9rem, max-content) var(--col-rcv, max-content) var(--col-mt, max-content) var(--col-sev, 4.6rem)
     var(--col-host, minmax(5rem, 9rem)) var(--col-app, minmax(4rem, 8rem)) var(--codes-w) minmax(0, 1fr);
   column-gap: 12px;
   margin: 6px 20px 0; background: var(--panel); border: 1px solid var(--border); border-radius: var(--radius);
@@ -375,7 +390,7 @@ body.col-resizing, body.col-resizing * { cursor: col-resize !important; user-sel
 
 .row {
   display: grid;
-  grid-template-columns: max-content max-content 4.6rem minmax(5rem, 9rem) minmax(4rem, 8rem) var(--codes-w) 1fr;
+  grid-template-columns: max-content max-content max-content 4.6rem minmax(5rem, 9rem) minmax(4rem, 8rem) var(--codes-w) 1fr;
   gap: 0 12px; align-items: baseline;
   padding: 5px 14px 5px 11px;
   border-left: 3px solid transparent;
@@ -420,18 +435,25 @@ body.col-resizing, body.col-resizing * { cursor: col-resize !important; user-sel
 .row.sev-crit, .row.sev-err, .row.sev-warning { border-left-color: var(--sev); }
 .row.sev-crit, .row.sev-err { background: color-mix(in srgb, var(--sev) 7%, transparent); }
 .row .host[data-act], .row .app[data-act] { cursor: pointer; }
-/* Audit events and LogStream internal logs: colored edge (over the severity
-   one), light tint, badge before the app name. */
+/* Source badge at the start of every row, fixed width, one color per source.
+   Audit events, log files and LogStream internal logs also get a colored edge
+   (over the severity one) and a light tint. */
+.row.src-syslog { --src: var(--src-syslog); }
+.row.src-docker { --src: var(--src-docker); }
+.row.src-kernel { --src: var(--src-kernel); }
+.row.src-host { --src: var(--src-host); }
 .row.src-audit { --src: var(--src-audit); }
 .row.src-logstream { --src: var(--src-ls); }
-.row.src-audit, .row.src-logstream { border-left-color: var(--src); background: color-mix(in srgb, var(--src) var(--src-tint), transparent); }
-.row.src-audit:hover, .row.src-logstream:hover { background: color-mix(in srgb, var(--src) 13%, var(--row-hover)); }
+.row.src-file { --src: var(--src-file); }
+.row.src-audit, .row.src-logstream, .row.src-file { border-left-color: var(--src); background: color-mix(in srgb, var(--src) var(--src-tint), transparent); }
+.row.src-audit:hover, .row.src-logstream:hover, .row.src-file:hover { background: color-mix(in srgb, var(--src) 13%, var(--row-hover)); }
 .src-badge {
-  display: inline-block; margin-right: 5px; padding: 0 4px; border-radius: 4px; vertical-align: 1px;
-  font: 700 9.5px/1.45 var(--sans); letter-spacing: .04em;
+  justify-self: start; box-sizing: content-box; width: 6ch; padding: 1px 4px; border-radius: 4px; text-align: center;
+  font-family: var(--log-font, var(--mono)); font-size: 10px; font-weight: 700; letter-spacing: .02em; line-height: 1.35;
   color: var(--src); background: color-mix(in srgb, var(--src) 16%, transparent);
   box-shadow: inset 0 0 0 1px color-mix(in srgb, var(--src) 40%, transparent);
 }
+:root[data-density="compact"] .row .src-badge { padding-top: 0; padding-bottom: 0; line-height: 1.3; }
 .row .raw {
   margin-top: 2px; font-size: .85em; line-height: 1.35; color: var(--muted);
   white-space: pre-wrap; word-break: break-all;
@@ -539,11 +561,24 @@ dialog.settings {
 dialog.settings .dlg { display: flex; flex-direction: column; height: 100%; padding: 0; }
 dialog.settings .dlg > header { padding: 14px 16px 14px 22px; border-bottom: 1px solid var(--border); }
 .set-body { flex: 1; min-height: 0; display: grid; grid-template-columns: minmax(176px, max-content) minmax(0, 1fr); }
-.set-nav {
-  display: flex; flex-direction: column; gap: 2px;
-  padding: 12px 10px; border-right: 1px solid var(--border);
+.set-side {
+  display: flex; flex-direction: column; min-height: 0; overflow-y: auto;
+  border-right: 1px solid var(--border);
   background: color-mix(in srgb, var(--panel-2) 50%, transparent);
 }
+.set-nav { display: flex; flex-direction: column; gap: 2px; padding: 12px 10px; }
+/* Save / Cancel, under the section labels: Save takes the accent color while changes wait. */
+.set-actions {
+  display: flex; flex-wrap: wrap; gap: 6px; margin: 4px 10px 12px; padding-top: 12px;
+  border-top: 1px solid var(--border);
+}
+.set-actions .btn { flex: 1; justify-content: center; }
+.set-actions .btn.save { color: var(--muted); }
+.set-actions .btn svg { width: 16px; height: 16px; }
+.set-actions.dirty .btn.save { background: var(--accent); border-color: var(--accent); color: #fff; box-shadow: 0 0 0 3px var(--accent-soft); }
+.set-actions.dirty .btn.save:hover { filter: brightness(1.08); }
+.set-state { flex-basis: 100%; margin: 2px 2px 0; font-size: 12px; color: var(--muted); }
+.set-actions.dirty .set-state { color: var(--accent); font-weight: 550; }
 .set-nav button {
   display: flex; align-items: center; gap: 10px;
   height: 38px; padding: 0 12px; border: 0; border-radius: 9px;
@@ -571,10 +606,11 @@ dialog.settings .dlg > header { padding: 14px 16px 14px 22px; border-bottom: 1px
 .set-panel .preview-list { margin: 14px 0 10px; pointer-events: none; }
 .preview-list .row {
   cursor: default;
-  grid-template-columns: max-content max-content minmax(0, max-content) minmax(0, 1fr);
-  grid-template-areas: "rcv sev host app" "msg msg msg msg";
+  grid-template-columns: max-content max-content max-content minmax(0, max-content) minmax(0, 1fr);
+  grid-template-areas: "src rcv sev host app" "msg msg msg msg msg";
   gap: 3px 10px;
 }
+.preview-list .row .src-badge { grid-area: src; }
 .preview-list .row time.rcv { grid-area: rcv; }
 .preview-list .row time.mt { display: none; }
 .preview-list .row .sev { grid-area: sev; }
@@ -751,11 +787,40 @@ input.switch:focus-visible { outline: 2px solid var(--accent); outline-offset: 2
 .tag-row .icon-btn svg { width: 15px; height: 15px; }
 .opt input { display: none; }
 .opt:has(input:checked) { color: var(--accent); border-color: var(--accent); background: var(--accent-soft); }
+/* Log files (Settings > Sources) */
+.file-list { display: flex; flex-direction: column; gap: 4px; }
+.file-row {
+  display: grid; align-items: center; gap: 6px;
+  grid-template-columns: auto minmax(7rem, 1fr) minmax(10rem, 2fr) minmax(9rem, 1.4fr) minmax(6rem, .8fr) 26px;
+  padding: 6px 6px 6px 10px; border: 1px solid var(--border); border-radius: 8px; background: var(--panel);
+}
+.file-row.off > input[type="text"], .file-row.off > select { opacity: .6; }
+.file-row input[type="text"] {
+  height: 28px; padding: 0 8px; min-width: 0;
+  border: 1px solid var(--border); border-radius: 6px; background: var(--panel); font-size: 12.5px; outline: 0;
+}
+.file-row input.mono { font-family: var(--mono); }
+.file-row input[type="text"]:focus { border-color: var(--accent); box-shadow: 0 0 0 3px var(--accent-soft); }
+.file-row select.field { height: 28px; }
+.file-row .icon-btn { width: 26px; height: 26px; border-radius: 6px; }
+.file-row .icon-btn svg { width: 15px; height: 15px; }
+.file-row .fr-regex, .file-row .fr-status, .file-row .fr-sample { grid-column: 2 / -1; }
+.file-row .fr-sample {
+  margin: 0; font: 11.5px var(--mono); color: var(--muted);
+  white-space: nowrap; overflow: hidden; text-overflow: ellipsis;
+}
+.file-row .fr-sample[hidden] { display: none; }
+.file-row .fr-regex[hidden] { display: none; }
+.file-row .fr-status { margin: 0 !important; font-size: 12px; }
+.file-toolbar { margin-top: 10px; }
 .tag-row .err { grid-column: 1 / -1; color: var(--sev-err); font-size: 12px; }
 .tag-row .err:empty { display: none; }
 
 /* ---------- Mobile ---------- */
 @media (max-width: 760px) {
+  .file-row { grid-template-columns: auto minmax(0, 1fr) 26px; }
+  .file-row [data-del] { grid-column: 3; grid-row: 1; }
+  .file-row > [data-f="path"], .file-row > [data-f="format"], .file-row > [data-f="host"] { grid-column: 2 / -1; }
   body { font-size: 13.5px; }
   .topbar { flex-wrap: wrap; padding: 8px 16px; gap: 8px 10px; }
   .search { order: 3; flex-basis: 100%; max-width: none; }
@@ -780,10 +845,11 @@ input.switch:focus-visible { outline: 2px solid var(--accent); outline-offset: 2
   .table .list { display: block; margin: 0; }
   .list-head { display: none; }
   .row, .table .row {
-    grid-template-columns: auto auto 1fr auto;
-    grid-template-areas: "rcv sev host codes" "msg msg msg msg";
+    grid-template-columns: auto auto auto 1fr auto;
+    grid-template-areas: "src rcv sev host codes" "msg msg msg msg msg";
     gap: 3px 8px; padding: 8px 12px 8px 10px;
   }
+  .row .src-badge { grid-area: src; }
   .row time.rcv { grid-area: rcv; }
   .row time.mt { display: none; } /* still shown in the details panel */
   .row .sev { grid-area: sev; }
@@ -797,10 +863,11 @@ input.switch:focus-visible { outline: 2px solid var(--accent); outline-offset: 2
   .field-grid { grid-template-columns: 1fr; gap: 4px; }
   dialog.settings { height: calc(100vh - 24px); width: calc(100vw - 16px); }
   .set-body { grid-template-columns: 1fr; grid-template-rows: auto minmax(0, 1fr); }
-  .set-nav {
-    display: grid; grid-template-columns: repeat(5, minmax(0, 1fr)); padding: 6px;
-    border-right: 0; border-bottom: 1px solid var(--border);
-  }
+  .set-side { border-right: 0; border-bottom: 1px solid var(--border); overflow: visible; }
+  .set-nav { display: grid; grid-template-columns: repeat(5, minmax(0, 1fr)); padding: 6px; }
+  .set-actions { margin: 0 6px 6px; padding-top: 6px; align-items: center; }
+  .set-actions .btn { flex: 0 0 auto; height: 30px; }
+  .set-state { flex: 1; flex-basis: auto; margin: 0; text-align: right; }
   .set-nav button {
     flex-direction: column; justify-content: center; height: auto; padding: 6px 2px; gap: 3px;
     font-size: 11.5px; text-align: center;