Log files: one file per entry, format detection

- a path is one file (wildcards refused for now); the name defaults to
  the file name
- format "auto" (the default): detected from the last 50 lines of the
  file, the structured format followed by at least 80 % of them (Traefik
  access JSON, JSON lines, CLF, logfmt), else plain text; done again when
  the format changes, shown in the status ("Format detected: …"); any
  other format forces it
- POST /api/files/detect {path}: found / permission / empty, detected
  format and last line, shown under the row as soon as the path is typed
  (POST so that read-only accounts cannot read files through it)
- README (EN/FR) updated

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.env.example

@@ -1,21 +1,73 @@
 # Copy this file to .env and adjust it.
-SYSLOG_PORT=514
+SYSLOG_PORT=5514
-HTTP_PORT=8080
+HTTP_PORT=8989
 TZ=Europe/Paris
 # How long logs are kept (e.g. 7d, 30d, 12w, 1y)
 RETENTION=30d
-# Web UI authentication (empty = disabled)
+# Web UI authentication: local (login page with the account below, or none) or oidc (OpenID Connect provider)
+AUTH_MODE=local
+# local mode: user and password (empty = no authentication)
 AUTH_USER=
 AUTH_PASS=
+# local mode: optional read-only account (can search and export, cannot change tags, sources or purge)
+AUTH_VIEWER_USER=
+AUTH_VIEWER_PASS=
+# local mode: PNG logo shown on the login page, path inside the container (empty = no logo).
+# Mount the file in docker-compose.yml, e.g. ./logo.png:/config/logo.png:ro, then LOGIN_LOGO=/config/logo.png
+LOGIN_LOGO=
+
+# Ready-made color tags offered in Settings > Filters (see docs/presets.md).
+# Empty: /data/presets.json if present, else the built-in list. To use your own file,
+# mount it in docker-compose.yml, e.g. ./presets.json:/config/presets.json:ro
+PRESETS_FILE=
+# Session lifetime, both modes (e.g. 8h, 24h)
+SESSION_TTL=12h
+# oidc mode: issuer URL exactly as the provider announces it
+# (Keycloak: https://sso.example.org/realms/<realm>, Authentik: https://auth.example.org/application/o/<slug>/)
+OIDC_ISSUER=
+OIDC_CLIENT_ID=
+OIDC_CLIENT_SECRET=
+# Callback URL of logstream, to register in the provider (path free, /auth/callback recommended)
+OIDC_REDIRECT_URL=https://logs.example.org/auth/callback
+# Requested scopes (openid is always added)
+OIDC_SCOPES=openid profile email
+# oidc mode: only members of this group are admins, the others are read-only (empty = everyone is admin).
+# The groups come from the ID token claim OIDC_GROUPS_CLAIM (default groups)
+OIDC_ADMIN_GROUP=
+OIDC_GROUPS_CLAIM=groups
 # Reverse DNS: show host names instead of IP addresses (on/off)
 RDNS=on
 # DNS server used for reverse lookups (e.g. your router: 192.168.1.1). Empty = system resolver
 DNS_SERVER=
 # Allow "Delete all logs" in Settings (true/false)
-ALLOW_PURGE=true
+ALLOW_PURGE=false
 # Maximum number of rows in a CSV export
 EXPORT_MAX=100000
 # Collect the logs of the Docker containers of this machine (on/off)
 DOCKER_LOGS=on
 # History read from a container seen for the first time (e.g. 30m, 1h, 24h; 0 = only new lines)
 DOCKER_BACKFILL=1h
+# Host system logs (enable them in Settings > Sources)
+# Group allowed to read the host logs: 4 = adm on Debian/Ubuntu; or the systemd-journal group
+# (getent group systemd-journal | cut -d: -f3)
+HOST_LOGS_GID=4
+# History read when the source is turned on (e.g. 30m, 1h, 24h; 0 = only new entries)
+HOST_LOGS_BACKFILL=1h
+# Kernel messages of the host (dmesg, enable them in Settings > Sources). The container needs
+# /dev/kmsg: uncomment "devices" in docker-compose.yml. When the host has
+# kernel.dmesg_restrict=1 (default on Debian/Ubuntu), either run on the host
+#   sysctl -w kernel.dmesg_restrict=0   (persist it in /etc/sysctl.d/)
+# or run the container as root with the SYSLOG capability ("user" and "cap_add" in docker-compose.yml).
+# History read when the source is turned on (e.g. 30m, 1h, 24h; 0 = only new messages)
+DMESG_BACKFILL=1h
+# Linux audit log of the host (auditd, enable it in Settings > Sources). Uncomment the
+# /var/log/audit volume in docker-compose.yml and let the container read the file:
+# log_group = adm in /etc/audit/auditd.conf, and chgrp adm + chmod 750 on /var/log/audit.
+# History read when the source is turned on (e.g. 30m, 1h, 24h; 0 = only new events)
+AUDIT_BACKFILL=1h
+# Host name of the audit events (empty: node= of the records, else /host/etc/hostname, else localhost)
+AUDIT_HOST=
+# Log files of other programs (Traefik, nginx, applications…), declared in Settings > Log sources >
+# Log files: mount their directory read-only in docker-compose.yml (e.g. /var/log/traefik:/logs/traefik:ro).
+# History read when a file is turned on (e.g. 30m, 1h, 24h; 0 = only new lines)
+FILES_BACKFILL=1h

Dockerfile

@@ -4,7 +4,7 @@
 FROM golang:1.27.1-alpine3.24 AS build
 WORKDIR /src
 COPY go.mod ./
-COPY *.go ./
+COPY *.go presets.json ./
 COPY web ./web
 RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/logstream .
 

README.fr.md

@@ -0,0 +1,639 @@
+[English](README.md) | Français
+
+# Logstream
+
+Un collecteur syslog simple : il reçoit les logs en UDP/TCP sur le port 514, les stocke dans
+[VictoriaLogs](https://docs.victoriametrics.com/victorialogs/) et sert une interface web
+soignée (thème clair/sombre, recherche instantanée, direct, tags de couleur, interface en
+anglais et en français).
+
+```
+devices ──514 udp/tcp──▶ logstream (Go) ──HTTP batches──▶ VictoriaLogs
+                              ▲    └── SSE (live) ──▶ browser
+                              └──── API / UI ◀─────────┘
+```
+
+## Architecture
+
+![Schéma logique de Logstream](docs/architecture.png)
+
+- **Ingestion** : les messages syslog (UDP/TCP) et les logs des conteneurs Docker passent tous
+  par `sink()` (résolution DNS inverse des hôtes donnés par leur IP, sans bloquer la
+  réception), puis par la file du `Store`, qui les envoie par lots à VictoriaLogs. Quand
+  VictoriaLogs est injoignable, les lots sont gardés sur disque (`/data/spool`, jusqu'à
+  `SPOOL_MAX_MB`) et renvoyés, les plus anciens d'abord, dès son retour.
+- **Direct** : `sink()` publie aussi chaque message dans le `Hub`, qui le diffuse aux
+  navigateurs en SSE.
+- **Recherche** : l'API HTTP traduit les filtres de l'interface en requêtes LogsQL envoyées à
+  VictoriaLogs.
+- **Frise** : `/api/histogram` (`histogram.go`) compte les logs par intervalle et par sévérité,
+  alignés sur l'heure locale ; les bornes du zoom (`from`/`to`) s'appliquent aussi à la liste
+  et à l'export.
+- **État** : les tags et les réglages des sources sont dans `/data` (volume `logstream-data`) ;
+  les logs eux-mêmes dans le volume `vlogs-data`.
+
+La source modifiable du schéma est
+[`docs/architecture.excalidraw`](docs/architecture.excalidraw)
+(à ouvrir sur [excalidraw.com](https://excalidraw.com)).
+
+## Démarrage
+
+```bash
+cp .env.example .env      # optional
+docker compose up -d --build
+./tools/send-test-logs.sh # sends 100 test messages
+```
+
+Ouvrez ensuite <http://localhost:8080>.
+
+## Envoyer des logs
+
+- **rsyslog** (Linux) : ajoutez `*.* @SERVER_IP:514` (UDP) ou `*.* @@SERVER_IP:514` (TCP)
+  dans `/etc/rsyslog.d/90-logstream.conf`, puis lancez `systemctl restart rsyslog`.
+- **Équipements réseau, NAS, pare-feu** : indiquez l'IP du serveur et le port 514 dans leurs
+  réglages « syslog distant » (remote syslog).
+- **Test manuel** : `logger -n 127.0.0.1 -P 514 -d "hello error"` (util-linux) ou
+  `echo "<14>test ok" | nc -u -w1 127.0.0.1 514`.
+
+**Paramètres > Sources de logs > Syslog** active ou désactive la réception syslog et choisit les
+protocoles (UDP, TCP) sans redémarrage ; le choix est enregistré dans `/data/syslog.json`. Cet
+onglet affiche aussi l'état d'écoute (et l'erreur si le port est déjà utilisé). Le port
+lui-même est publié par docker-compose : modifiez `SYSLOG_PORT` dans `.env`, puis lancez
+`docker compose up -d`.
+
+Formats pris en charge : RFC 3164 (BSD) et RFC 5424. En TCP, les deux découpages sont acceptés :
+« un message par ligne » et « octet counting » (RFC 6587).
+
+## Recherche
+
+**Mode simple** (par défaut) : chaque mot est cherché comme sous-chaîne, sans tenir compte de
+la casse, dans le message, l'hôte et l'application. Les mots sont combinés avec ET.
+
+| Saisie | Signification |
+|---|---|
+| `error disk` | contient « error » **et** « disk » |
+| `"disk full"` | contient la phrase exacte |
+| `error -timeout` | contient « error » mais pas « timeout » |
+
+**Mode LogsQL** (bouton `Simple` / `LogsQL`) : le langage de requête complet de VictoriaLogs,
+par exemple `error AND host:web-01`, `app:~"ssh|nginx"` ou `* | stats by (host) count()`.
+Le direct est désactivé dans ce mode.
+
+Chaque ligne affiche, de gauche à droite : l'**heure de réception** (horloge du serveur),
+l'horodatage trouvé dans le message lui-même (`msg_time`), la sévérité, l'hôte, l'application,
+les codes des tags trouvés et le message. Un clic sur un hôte ou une application filtre dessus.
+
+Les logs sont indexés, recherchés et triés par **heure de réception** : les équipements dont
+l'horloge est fausse (par exemple des points d'accès dont le NTP échoue) apparaissent quand même
+dans la bonne plage de temps. Les logs stockés par les versions antérieures à ce changement sont
+indexés par l'horodatage de leur message ; purgez-les depuis les Paramètres pour repartir sur
+une base propre.
+
+Les badges de sévérité `err`/`crit` et `warning` reprennent les couleurs des tags `error` et
+`warning`.
+
+Raccourcis : `/` place le curseur dans la recherche, `Esc` la vide. Un clic sur une ligne
+affiche tous ses champs.
+
+Les en-têtes de colonnes restent visibles pendant le défilement. Faites glisser le bord d'un
+en-tête (réception, heure message, sévérité, hôte, app) pour redimensionner la colonne, et
+double-cliquez dessus pour revenir à la largeur automatique ; le message occupe l'espace
+restant. Les largeurs sont mémorisées par le navigateur (**Paramètres > Affichage >
+Réinitialiser les colonnes** les rétablit toutes). Sur téléphone, la liste garde sa
+présentation sur deux lignes, sans colonnes.
+
+## Modes Flux et Période
+
+Le premier contrôle de la barre de filtres bascule entre deux modes d'affichage :
+
+- **Flux** : les derniers logs sur une durée glissante (5 min à 30 jours, ou tout), avec le direct.
+- **Période** : les logs entre une date de début et une date de fin, saisies dans le fuseau choisi
+  dans les Paramètres. ◀ et ▶ passent à la période précédente ou suivante de même durée, la loupe
+  la double autour de son milieu. Le direct se met en pause ; le mode et la période sont conservés
+  au rechargement.
+
+## Frise
+
+La frise au-dessus de la liste montre le volume de logs par intervalle, compté selon l'**heure
+de réception** sur l'horloge du serveur (elle correspond donc aux heures affichées dans les
+lignes).
+
+- Au survol d'un intervalle : ses bornes, son total et le détail par sévérité.
+- Un clic sur une barre affiche cet intervalle en mode Période ; un glisser sur plusieurs barres
+  affiche la sélection. « × Revenir au flux » repasse en mode Flux. La liste, les compteurs et
+  l'export CSV suivent la période.
+- En direct, le dernier intervalle grandit à l'arrivée des messages, et la frise se recharge à
+  chaque nouvel intervalle. Rien n'est rafraîchi tant que l'onglet du navigateur est masqué ;
+  la frise se met à jour dès qu'il redevient visible.
+
+**Paramètres > Affichage > Frise** (mémorisé par navigateur) : échelle (linéaire, √ par défaut,
+log), hauteur (S/M/L : 40/80/120 px), couleur (empilement par sévérité, intensité comparée à
+la médiane de la fenêtre : calme, rafale au-delà de 3×, anomalie au-delà de 10×, ou aucune),
+barres ou aire, division (automatique, environ 100 intervalles, ou fixe : 1 s, 10 s, 1 min,
+5 min, 1 h, 1 jour) et rafraîchissement (désactivé, 5 s, 15 s, 30 s, 1 min, ou à chaque nouvel
+intervalle). Une division fixe qui dépasserait 300 intervalles sur la plage est élargie
+(signalé par « élargi »). Les intervalles sont alignés sur l'heure locale du fuseau horaire
+choisi dans les Paramètres (les jours commencent à minuit, heure locale).
+
+API : `curl 'localhost:8080/api/histogram?range=24h&step=auto&tz=Europe/Paris'` renvoie
+`step` (ms), `start` (ms), `count`, `now` (horloge du serveur) et les `buckets` non vides
+(`i` = numéro d'intervalle, `n` = total, `sev` = nombre par sévérité). Toutes les routes de
+logs acceptent aussi `from` / `to` (millisecondes Unix) à la place de `range`.
+
+## Logs des conteneurs Docker
+
+Logstream collecte aussi les logs des conteneurs Docker qui tournent sur la machine où il est
+installé (`DOCKER_LOGS=on`, le défaut dans `docker-compose.yml`). Ils sont recherchés, filtrés,
+colorés et exportés comme les messages syslog :
+
+- **host** est le nom de l'hôte Docker, **app** le service compose (ou le nom du conteneur), et
+  chaque log porte aussi `container`, `container_id`, `image`, `compose_project`,
+  `compose_service` et `stream` (stdout/stderr), visibles dans le détail de la ligne.
+- Le filtre **Source** n'affiche que les logs syslog ou que les logs Docker ; les lignes Docker
+  ont un petit cube devant le nom de l'application, de la couleur de son projet compose.
+- La sévérité vient de la ligne elle-même quand l'application l'écrit : JSON
+  (`"level":"error"`), logfmt (`level=warn`), `[ERROR]`, ou un niveau en majuscules au début de
+  la ligne (`ERROR`, `WARN`…). Sinon, elle vaut `info`. Les codes de couleur du terminal sont
+  supprimés.
+- **Paramètres > Sources de logs** affiche une étiquette par conteneur (`project/service`) dans un seul
+  champ : les conteneurs suivis en couleur, puis les non suivis en gris ; un clic bascule une
+  étiquette. La couleur identifie le projet compose, et les noms d'applications Docker de la
+  liste utilisent la même couleur. Les conteneurs arrêtés sont masqués par défaut (« Afficher
+  les conteneurs arrêtés » les montre, en pointillés, et ils restent modifiables). Une zone de
+  filtre et « Tout activer » / « Tout désactiver » (appliqués aux étiquettes affichées) aident
+  quand les conteneurs sont nombreux. Les nouveaux conteneurs sont suivis automatiquement, sauf
+  si cette option est désactivée. Les choix sont enregistrés par service compose (ou nom de
+  conteneur) dans `/data/docker.json` : ils survivent aux recréations.
+- Logstream mémorise la position de la dernière ligne stockée pour chaque conteneur
+  (`/data/docker-state.json`) : après un redémarrage, il reprend sans perdre de lignes. La
+  position n'avance qu'une fois la ligne dans VictoriaLogs ou dans le tampon disque, et une file
+  pleine ralentit la lecture au lieu de perdre des lignes. Un conteneur vu pour la
+  première fois est lu à partir de `DOCKER_BACKFILL` en arrière (1 heure par défaut).
+- Logstream lui-même et le proxy ci-dessous ne sont jamais collectés ; ajoutez l'étiquette
+  `logstream.exclude=true` à tout autre conteneur pour l'exclure définitivement.
+
+**Sécurité** : l'accès au socket Docker équivaut à un accès root sur la machine. Logstream passe
+donc par [docker-socket-proxy](https://github.com/Tecnativa/docker-socket-proxy), qui ne laisse
+passer que la liste des conteneurs, la lecture des logs, les événements et les informations du
+moteur (`GET` uniquement).
+
+## Logs système de l'hôte
+
+Logstream peut aussi collecter les logs système de la machine qui héberge la stack, sans rien
+configurer sur l'hôte. La source est **désactivée par défaut** : activez-la dans
+**Paramètres > Sources de logs > Logs système de l'hôte** (enregistré dans `/data/hostlogs.json`).
+
+- `docker-compose.yml` monte `/var/log` et `/run/log/journal` en lecture seule sous `/host`.
+- Si l'hôte utilise systemd, Logstream lit directement les fichiers du **journal systemd**
+  (pas besoin de `journalctl` dans l'image) : **host** est le nom de la machine, **app** le
+  programme (`SYSLOG_IDENTIFIER`), la sévérité et la facility viennent du journal, et l'unité
+  systemd est conservée dans `unit`. Sinon, il suit les fichiers texte de `/var/log` (`syslog`,
+  `messages`, `*.log`), analysés comme des lignes syslog, avec le nom du fichier dans `log_file`.
+- Ces logs ont la source `host` : le filtre **Source** permet de les afficher seuls.
+- À l'activation, la dernière heure est lue d'abord (`HOST_LOGS_BACKFILL`) ; la position
+  atteinte est enregistrée dans `/data/hostlogs-state.json`, un redémarrage ne perd donc ni ne
+  duplique d'entrées.
+- **Droits** : le conteneur tourne avec un utilisateur sans privilège et reçoit le groupe `adm`
+  (gid 4), qui peut lire le journal et `/var/log` sur Debian et Ubuntu. Sur d'autres systèmes,
+  réglez `HOST_LOGS_GID` dans `.env` sur le gid de `systemd-journal`
+  (`getent group systemd-journal | cut -d: -f3`). Paramètres > Sources de logs affiche un message clair
+  si l'accès est refusé.
+- Limites : les champs du journal compressés par journald (messages de plus de 512 octets,
+  compressés en zstd/lz4/xz) ne peuvent pas être décodés sans bibliothèque supplémentaire ; ils
+  sont comptés dans les Paramètres et affichés comme « (compressed journal entry) ». Les fichiers
+  texte tournés ou compressés (`*.1`, `*.gz`) ne sont pas lus.
+
+## Messages du noyau (dmesg)
+
+Logstream peut collecter le buffer circulaire du noyau de l'hôte (ce qu'affiche `dmesg` :
+matériel, pilotes, OOM killer, pare-feu…) en lisant `/dev/kmsg`. La source est **désactivée par
+défaut** : activez-la dans **Paramètres > Sources de logs > Messages du noyau (dmesg)** (enregistré dans
+`/data/dmesg.json`).
+
+- **Périphérique** : le conteneur ne voit pas `/dev/kmsg` par défaut. Décommentez la section
+  `devices` de `docker-compose.yml` (`/dev/kmsg:/dev/kmsg:r`, en lecture seule), puis
+  `docker compose up -d`.
+- **Droits** : si l'hôte a `kernel.dmesg_restrict=1` (le défaut sur Debian et Ubuntu), seuls
+  les processus qui ont la capacité `CAP_SYSLOG` peuvent lire le log du noyau, et le conteneur
+  tourne avec un utilisateur sans privilège. Au choix :
+  - l'autoriser sur l'hôte : `sysctl -w kernel.dmesg_restrict=0`, conservé au redémarrage avec
+    `echo kernel.dmesg_restrict=0 | sudo tee /etc/sysctl.d/50-dmesg.conf` (tous les
+    utilisateurs de l'hôte peuvent alors lancer `dmesg`) ; ce réglage ne peut pas être changé
+    depuis un conteneur ;
+  - ou lancer Logstream en root avec la capacité `SYSLOG` : décommentez `user: "0:0"` et
+    `cap_add: [SYSLOG]` dans `docker-compose.yml`.
+
+  Paramètres > Sources de logs indique si le périphérique est absent ou l'accès refusé ; le reste de
+  Logstream continue de fonctionner et le périphérique est réessayé toutes les 10 secondes.
+- **Champs** : **app** vaut `kernel` (ou le nom du programme pour les messages écrits par des
+  programmes, comme `systemd[1]: …`), la sévérité et la facility viennent de l'enregistrement,
+  **host** vaut `localhost`, `subsystem` et `device` sont conservés quand le noyau les fournit.
+  Ces logs ont la source `kernel` (`proto` `kmsg`) : le filtre **Source** permet de les afficher
+  seuls.
+- **Heure** : les enregistrements portent le temps écoulé depuis le démarrage ; Logstream le
+  convertit en date grâce à l'heure de démarrage (`/proc/uptime`). Comme avec `dmesg -T`,
+  l'heure peut être trop tôt après une mise en veille de l'hôte.
+- À l'activation, la dernière heure est lue d'abord (`DMESG_BACKFILL`, `0` = seulement les
+  nouveaux messages). Le numéro de séquence du dernier message et l'identifiant de démarrage
+  sont enregistrés dans `/data/dmesg-state.json` : après un redémarrage de Logstream, la lecture
+  reprend après le dernier message (sans doublon) ; après un redémarrage de l'hôte, le nouveau
+  buffer est lu depuis `DMESG_BACKFILL`. Les messages écrasés dans le buffer du noyau avant
+  d'être lus sont comptés dans Paramètres.
+
+## Audit Linux (auditd)
+
+Logstream peut suivre le journal du démon d'audit Linux (`/var/log/audit/audit.log`) :
+connexions, `sudo`, fichiers surveillés, appels système visés par vos règles `auditctl`… La
+source est **désactivée par défaut** : activez-la dans **Réglages > Sources > Audit Linux
+(auditd)**, où le chemin du fichier peut aussi être changé (enregistré dans `/data/audit.json` ;
+`AUDIT_LOG` fixe le chemin par défaut).
+
+- **Montage** : décommentez `/var/log/audit:/var/log/audit:ro` dans `docker-compose.yml`, puis
+  `docker compose up -d`. (`/var/log` est déjà monté sous `/host` pour les logs système de
+  l'hôte : `/host/var/log/audit/audit.log` fonctionne donc aussi comme chemin, sans nouveau
+  volume.)
+- **Droits** (côté hôte) : auditd écrit son journal en `root`, mode `0600`, dans un répertoire
+  `0700`, et Logstream tourne avec l'UID `10001`. Le plus fiable est le groupe que le conteneur
+  reçoit déjà pour les logs de l'hôte (`HOST_LOGS_GID`, `adm` par défaut) :
+  ```bash
+  sudo sed -i 's/^log_group.*/log_group = adm/' /etc/audit/auditd.conf
+  sudo chgrp adm /var/log/audit && sudo chmod 750 /var/log/audit
+  sudo service auditd restart        # systemctl refuse de redémarrer auditd sur certaines distributions
+  ```
+  auditd crée alors chaque fichier, rotations comprises, en mode `0640` avec le groupe `adm`.
+  Une ACL fonctionne aussi
+  (`sudo setfacl -m u:10001:rx /var/log/audit && sudo setfacl -m u:10001:r /var/log/audit/audit.log`)
+  mais doit être réappliquée à chaque nouveau fichier après une rotation. Réglages > Sources
+  indique si le fichier est absent ou l'accès refusé ; il est réessayé chaque seconde.
+- **Événements** : les enregistrements d'un même événement (`SYSCALL`, `EXECVE`, `CWD`, `PATH`,
+  `PROCTITLE`…) partagent le même `msg=audit(heure:numéro)` et deviennent **un seul** log, dès
+  qu'aucun enregistrement n'est arrivé depuis 2 secondes. Le message est un résumé — qui, quoi,
+  résultat — comme `cedric (as root): exec apt install nginx [exec] → success` ou
+  `bob: PAM:authentication via /usr/sbin/sshd from 203.0.113.9 on ssh → failed` ; les
+  enregistrements bruts sont gardés dans `audit_raw` et affichés sous le résumé (repliés en
+  densité compacte tant que la ligne n'est pas ouverte). Les valeurs hexadécimales
+  (`proctitle`, `cmd`, arguments avec espaces) sont décodées.
+- **Champs** : `audit_type` (type du premier enregistrement, aussi utilisé comme **app**),
+  `audit_serial`, `auid` (identifiant de l'utilisateur connecté), `user` (son nom, ou l'uid s'il
+  ne peut pas être résolu), `exe`, `res` (`success` / `failed`, d'après `res=` ou `success=`) et
+  `key` (clé de la règle). La sévérité est `warning` en cas d'échec, `info` sinon ; facility
+  `authpriv` ; source `audit`. Dans la liste, les lignes d'audit ont un liseré gauche violet,
+  un fond légèrement teinté et un badge **AUDIT** devant l'app ; le filtre **Source** permet de
+  les afficher seules.
+- **Noms** : `user` vient du format enrichi (`log_format = ENRICHED`, le défaut des versions
+  récentes d'auditd), sinon du `/etc/passwd` de l'hôte s'il est monté en `/host/etc/passwd`.
+  **host** est le `node=` des enregistrements (`name_format = hostname` dans `auditd.conf`),
+  sinon `AUDIT_HOST`, sinon `/host/etc/hostname`, sinon `localhost`.
+- À l'activation, la dernière heure est lue d'abord (`AUDIT_BACKFILL`, `0` = seulement les
+  nouveaux événements). L'inode et une position avant laquelle tout événement a été envoyé sont
+  enregistrés dans `/data/audit-state.json` : après un redémarrage, la lecture reprend là ;
+  après une rotation par auditd, la fin de `audit.log.1` est lue avant le nouveau fichier.
+
+## Fichiers de logs (Traefik, nginx, applications…)
+
+Logstream peut suivre les fichiers de logs d'autres programmes : Traefik, nginx, Apache, ou
+toute application qui écrit dans un fichier. Déclarez chaque fichier dans **Paramètres > Sources
+de logs > Fichiers de logs** (enregistré dans `/data/files.json`, 50 au plus) : un **nom** (l'**app**
+de ses entrées, par défaut le nom du fichier), le **chemin** du fichier tel que monté dans le
+conteneur, un **format** (*Automatique* par défaut), un **hôte** facultatif (par défaut
+`/host/etc/hostname`, sinon `localhost`) et un interrupteur.
+
+- **Montage** du dossier en lecture seule, par ex. dans `docker-compose.yml` :
+  `- /var/log/traefik:/logs/traefik:ro`, puis `docker compose up -d`. Les fichiers de `/var/log`
+  sont déjà accessibles sous `/host/var/log/…`. Les fichiers doivent être lisibles par l'UID
+  `10001` ou le groupe de `HOST_LOGS_GID` (`adm` par défaut) ; Paramètres > Sources de logs
+  indique quand aucun fichier ne correspond ou que l'accès est refusé.
+- **Chemin** : un fichier, par ex. `/logs/traefik/access.log` (pas de jokers). Dès la saisie, le
+  fichier est vérifié : trouvé ou non, lisible ou non, son format détecté et sa dernière ligne.
+- **Détection du format** (*Automatique*) : à partir des dernières lignes du fichier (50 au
+  plus), le format structuré que suivent au moins 80 % d'entre elles — accès Traefik JSON, JSON
+  lines, journal d'accès CLF, logfmt — sinon texte brut. Elle est refaite quand le format est
+  changé ; le format trouvé s'affiche dans l'état du fichier. Choisissez un autre format pour
+  le forcer (une expression régulière n'est jamais détectée).
+- **Formats** :
+  - *Texte brut* : une entrée par ligne ; le niveau est détecté comme pour les logs Docker, un
+    horodatage en début de ligne (ISO 8601, `2006/01/02 15:04:05`, `[02/Jan/2006:15:04:05 +0200]`,
+    `Jan  2 15:04:05`) devient l'heure du message et est retiré du message. Les lignes indentées
+    (stack traces) sont rattachées à l'entrée précédente.
+  - *JSON lines* : `msg` / `message`, `time` / `ts` / `@timestamp` (texte ou heure Unix) et
+    `level` (mot ou nombre, pino/bunyan compris) sont reconnus ; les autres clés deviennent des
+    champs, les objets imbriqués avec des noms pointés (`req.id`).
+  - *logfmt* : paires `clé=valeur`, comme les logs par défaut de Traefik ou des programmes Go ;
+    mêmes clés que le JSON.
+  - *Journal d'accès (CLF)* : Common / Combined Log Format de nginx et Apache, et le journal
+    d'accès de Traefik dans ce format (routeur, URL du serveur et durée compris). Champs
+    `client_ip`, `http_method`, `http_path`, `http_status`, `http_size`, `user_agent`,
+    `http_referer`, `router`, `backend`, `duration_ms`.
+  - *Accès Traefik (JSON)* : le journal d'accès JSON de Traefik (`accessLog.format: json`),
+    résumé en `GET app.example.com/ 200 · 12.5 ms · app@docker · 203.0.113.9`, toutes les clés
+    dans les champs. Les logs de Traefik lui-même dans le même fichier sont lus comme du JSON.
+  - *Expression régulière* (syntaxe RE2 / Go) : les groupes nommés `time`, `level`, `msg`,
+    `host` et `app` remplissent ces champs, `status` donne le niveau d'un journal d'accès, les
+    autres groupes deviennent des champs, par ex.
+    `^(?P<time>\S+ \S+) \[(?P<level>\w+)\] (?P<module>\S+): (?P<msg>.*)$`. Une ligne qui ne
+    correspond pas est rattachée à l'entrée précédente.
+
+  Pour les journaux d'accès, les statuts 5xx sont en `err`, les 4xx en `warning`, les autres en
+  `info`. Une ligne qui ne suit pas le format choisi est gardée comme du texte brut. Les champs
+  qui portent le nom d'un champ de LogStream (`host`, `app`…) sont renommés `field_host`,
+  `field_app`…
+- **Affichage** : source `file`, le chemin dans `log_file`. Dans la liste, ces lignes ont un
+  liseré gauche vert, une légère teinte et un badge **FILE** avant l'app ; le filtre **Source**
+  les affiche seules.
+- À l'activation d'un fichier, les lignes de la dernière heure déjà présentes sont lues d'abord
+  (`FILES_BACKFILL`, `0` = seulement les nouvelles lignes ; les lignes sans horodatage sont alors
+  ignorées) ; un fichier créé ensuite est lu depuis son début. L'inode et une
+  position sont enregistrés par fichier dans `/data/files-state.json` : après un redémarrage la
+  lecture reprend là ; après une rotation (`access.log` renommé `access.log.1`), la fin de
+  l'ancien fichier est lue avant le nouveau ; un fichier vidé sur place (`copytruncate`) est
+  relu depuis son début.
+
+## Logs internes de LogStream
+
+Logstream enregistre ses propres événements comme des logs, avec l'**hôte** `logstream`,
+l'**app** du composant (`store`, `syslog`, `docker`, `host`, `dmesg`, `audit`, `files`, `main`) et la
+source `logstream` : VictoriaLogs injoignable puis de retour, lots perdus, lots refusés depuis
+le tampon disque, erreurs du listener syslog, API Docker ou flux d'un conteneur décrochés,
+sources qui ne peuvent pas lire, démarrage et arrêt. Ils sont actifs par défaut à partir du
+niveau **warning** ; **Réglages > Sources > Logs internes de LogStream** permet de les couper
+ou de changer le niveau minimal (enregistré dans `/data/selflog.json`). Un même événement
+(composant et message) est enregistré au plus une fois toutes les 30 secondes, avec le nombre
+de répétitions écartées ; les événements passent par une file bornée et n'attendent jamais le
+stockage : un VictoriaLogs injoignable ne peut ni les accumuler ni les faire boucler. Tout
+continue d'aller dans `docker compose logs logstream` comme avant. Dans la liste, ces lignes
+ont un liseré gauche turquoise et un badge **LS** devant l'app ; le filtre **Source** permet de
+les afficher seules.
+
+## Export CSV
+
+Le bouton **Exporter** (à côté du nombre de logs) télécharge tous les logs stockés qui
+correspondent aux filtres en cours (recherche, plage de temps, sévérité, hôte, application),
+du plus récent au plus ancien, jusqu'à `EXPORT_MAX` lignes (100 000 par défaut), et pas
+seulement les lignes à l'écran. Deux variantes :
+
+- **CSV** : séparateur virgule, UTF-8.
+- **CSV pour Excel** : séparateur point-virgule avec un BOM UTF-8, pour qu'Excel en français
+  l'ouvre directement avec les accents. Les cellules qui commencent par `=`, `+`, `-` ou `@`
+  sont préfixées par `'` pour qu'un message de log forgé ne puisse pas s'exécuter comme une
+  formule.
+
+Colonnes : `received`, `message_time` (toutes deux au format `YYYY-MM-DD HH:MM:SS.mmm` dans le
+fuseau horaire choisi dans les Paramètres), `severity`, `facility`, `host`, `host_ip`, `app`,
+`pid`, `source_ip`, `proto`, `message`. En mode LogsQL, seule la partie filtre est prise en
+charge (pas de `| pipes`).
+
+En ligne de commande : `curl -o logs.csv 'localhost:8080/api/export.csv?q=error&range=24h&tz=Europe/Paris'`.
+
+## Paramètres
+
+L'icône en forme d'engrenage ouvre les paramètres, organisés en onglets. Tout, sauf les tags de
+couleur, est mémorisé par navigateur.
+
+- **Localisation et langue**
+  - *Langue* : anglais ou français.
+  - *Date et heure* : fuseau horaire (celui du navigateur, UTC ou environ 80 fuseaux courants)
+    et format d'affichage de l'heure de réception : `DD/MM/YYYY HH:MM:SS` (par défaut,
+    l'affichage français habituel), avec millisecondes, `YYYY-MM-DD`, horloge sur 12 heures,
+    ISO 8601 ou epoch Unix. Le fuseau horaire s'applique à toutes les dates affichées.
+- **Motifs de recherche** : tags de couleur. Chaque tag a un mot-clé, une couleur de fond (le texte passe
+  automatiquement en noir ou en blanc pour rester lisible) et des options : mot entier, respect
+  de la casse, expression régulière, actif. Les tags sont stockés sur le serveur dans
+  `/data/tags.json` (volume `logstream-data`) : ils sont donc partagés par tous les navigateurs.
+  Tags par défaut (pastel) : `warning` (orange) et `error` (rouge) ; `ok` (vert) est dans le
+  préréglage *Niveaux de log*. Les tags par
+  défaut qui utilisent encore les couleurs des versions précédentes passent automatiquement aux
+  couleurs pastel.
+  Le menu *+ Préréglage…* ajoute des tags tout faits : logs d'accès HTTP/HTTPS (codes de statut,
+  méthodes, sondes, robots, erreurs TLS et proxy), logs système (SSH, sudo, noyau, systemd,
+  pare-feu), applications (Docker, bases de données) et motifs généraux (niveaux de log,
+  adresses IPv4). Les tags déjà présents ne sont pas ajoutés en double, et les tags ajoutés se
+  modifient comme les autres. La liste vient d'un fichier texte modifiable (`PRESETS_FILE`) :
+  voir [docs/presets.fr.md](docs/presets.fr.md) pour le détail de chaque préréglage et le
+  format du fichier. Dans une expression
+  régulière, un groupe nommé `hl` (`(?<hl>…)`) ne colore que cette partie de la correspondance :
+  les préréglages s'en servent pour colorer le code de statut ou la méthode, pas le texte autour.
+  Chaque tag reçoit un code à deux chiffres (`01`, `02`…) attribué par le serveur : il reste
+  attaché au tag jusqu'à sa suppression (les tags créés par les versions précédentes en reçoivent
+  un aussi). La colonne *Motif de recherche* de la liste affiche, en badges gris, les codes des tags actifs
+  trouvés dans chaque message ; elle a la place pour 3, au-delà elle en affiche 2 et `+N`, et
+  l'infobulle les liste tous.
+- **Affichage**
+  - *Thème* : Système (suit la préférence de l'ordinateur ou du téléphone), Clair ou Sombre. Le
+    bouton soleil/lune de l'en-tête bascule entre clair et sombre.
+  - *Affichage des logs* : taille du texte (très petite, petite, moyenne, grande), densité
+    (normale, ou compacte pour afficher environ 50 % de lignes en plus à l'écran) et police :
+    la police monospace du système, l'une des 3 polices étroites intégrées, servies par
+    LogStream lui-même et utilisables hors ligne (Inconsolata Condensed, la plus étroite,
+    Iosevka et Ubuntu Mono), ou l'une des 11 polices libres conçues pour le texte dense
+    (JetBrains Mono, Fira Code, Source Code Pro, IBM Plex Mono, Cascadia Code, Roboto Mono,
+    Inconsolata, Red Hat Mono, Noto Sans Mono, Victor Mono, DM Mono). Ces 11 polices sont
+    chargées par le navigateur depuis [Bunny Fonts](https://fonts.bunny.net), un service
+    européen de polices respectueux de la vie privée ; sans accès à internet, la police du
+    système est utilisée. Les ligatures sont désactivées pour que `->` ou `!=` s'affichent tels
+    quels. Le réglage le plus dense est Très petite + Compacte + Inconsolata Condensed.
+- **Base de données** : chiffres de la base lus dans VictoriaLogs (actualisés au plus toutes les 30 s) :
+  lignes stockées, taille sur disque (index compris), taille brute et taux de compression,
+  période couverte avec la rétention, lignes des dernières 24 h et de la dernière heure, hôtes
+  et applications distincts, espace disque libre. Les tailles sont en Ko/Mo/Go (KB/MB/GB en
+  anglais). « Supprimer tous les logs » efface définitivement tous les logs stockés (il faut
+  taper `PURGE` pour confirmer). Les tags et les paramètres sont conservés. VictoriaLogs doit
+  être lancé avec `-delete.enable` (déjà présent dans `docker-compose.yml`). La fonction est
+  désactivée par défaut : mettez `ALLOW_PURGE=true` pour l'autoriser. Tout administrateur peut
+  alors purger : activez l'[authentification](#authentification) si l'interface est accessible
+  à d'autres.
+
+## Authentification
+
+`AUTH_MODE` choisit comment l'interface et l'API sont protégées (`/healthz` reste toujours ouvert) :
+
+- **`local`** (par défaut) : une page de connexion avec le compte `AUTH_USER` / `AUTH_PASS` ; laissez-les
+  vides pour n'avoir aucune authentification (par exemple derrière un reverse proxy qui contrôle déjà).
+  L'interface affiche alors un bandeau d'avertissement, que l'on peut fermer.
+- **`oidc`** : connexion par un fournisseur OpenID Connect (Keycloak, Authentik, Authelia, Zitadel…),
+  flux « authorization code » avec PKCE.
+
+En mode `local`, la page de connexion suit le thème et la langue de l'interface. La session dure
+`SESSION_TTL` (12 h par défaut), survit aux redémarrages (sa clé de signature est dans
+`/data/session.key`) et se termine quand `AUTH_USER` ou `AUTH_PASS` change ; le bouton de
+déconnexion (en haut à droite) y met fin. Les échecs de connexion sont écrits dans les logs avec
+l'adresse du client (`auth: failed login for "bob" from 192.0.2.7`). Les scripts peuvent toujours
+appeler l'API avec des identifiants HTTP Basic (`curl -u utilisateur:motdepasse`).
+
+Un **compte en lecture seule** optionnel, `AUTH_VIEWER_USER` / `AUTH_VIEWER_PASS`, peut chercher,
+suivre le direct et exporter, mais pas modifier les tags, les sources ni purger : ces réglages
+sont grisés dans son interface et l'API répond `403`.
+
+Pour afficher votre logo sur la page de connexion, montez un PNG dans le conteneur et indiquez
+son chemin dans `LOGIN_LOGO` :
+
+```yaml
+# docker-compose.yml, service logstream
+    volumes:
+      - ./logo.png:/config/logo.png:ro
+```
+```bash
+# .env
+LOGIN_LOGO=/config/logo.png
+```
+
+Pour utiliser OIDC :
+
+1. Dans le fournisseur, créez un client **confidentiel** (avec secret) pour logstream et déclarez
+   l'URL de retour `https://logs.example.org/auth/callback` (votre adresse).
+2. Dans `.env` :
+   ```bash
+   AUTH_MODE=oidc
+   OIDC_ISSUER=https://sso.example.org/realms/maison    # exactement l'« issuer » du fournisseur
+   OIDC_CLIENT_ID=logstream
+   OIDC_CLIENT_SECRET=...
+   OIDC_REDIRECT_URL=https://logs.example.org/auth/callback
+   ```
+3. `docker compose up -d`. Les logs affichent `oidc authentication enabled`, ou la raison pour
+   laquelle le fournisseur n'a pas pu être lu (issuer incorrect, injoignable…).
+
+Ouvrir l'interface renvoie vers la page de connexion du fournisseur, puis revient sur logstream.
+La session dure `SESSION_TTL` (12 h par défaut) et survit aux redémarrages (sa clé de
+signature est dans `/data/session.key`) ; à son expiration, la page repasse par la connexion. Le
+bouton de déconnexion (en haut à droite) termine la session logstream, puis ouvre la page de
+déconnexion du fournisseur s'il en a une.
+
+Tout utilisateur accepté par le fournisseur pour ce client peut se connecter : restreignez l'accès
+dans le fournisseur (Keycloak : rôles du client ou realm dédié ; Authentik : liaisons de
+l'application). Les connexions sont écrites dans les logs de logstream (`oidc: alice logged in`).
+Avec une URL de retour en `https`, les cookies ne sont envoyés qu'en HTTPS : logstream doit être
+joint à travers un reverse proxy TLS.
+
+Pour donner un accès en lecture seule à certains utilisateurs, définissez `OIDC_ADMIN_GROUP` (par
+exemple `logstream-admins`) : seuls ses membres sont administrateurs, les autres sont en lecture
+seule. Les groupes sont lus dans la revendication `groups` du jeton d'identité
+(`OIDC_GROUPS_CLAIM` pour en utiliser une autre) ; dans Keycloak, ajoutez au client un mapper
+« Group Membership » (le `/` initial est ignoré).
+
+Quel que soit le mode, chaque réponse porte des en-têtes de sécurité (Content-Security-Policy,
+X-Frame-Options…), et l'API refuse les modifications envoyées depuis un autre site (requêtes
+intersites).
+
+## Noms d'hôtes (DNS inverse)
+
+Quand un équipement envoie son adresse IP comme nom d'hôte (ou pas de nom d'hôte du tout),
+Logstream cherche son nom DNS (enregistrement PTR) et stocke le nom dans `host` et l'IP dans
+`host_ip`. Les résultats sont mis en cache (1 heure, 10 minutes quand il n'y a pas de nom). Les
+logs stockés auparavant avec une IP sont résolus à l'affichage, et le filtre d'hôte affiche
+`name (IP)`.
+
+Le conteneur utilise le DNS de Docker, qui relaie vers les résolveurs de l'hôte. Si vos noms
+locaux ne sont connus que de votre routeur ou d'un DNS local (Pi-hole, AdGuard, Unbound…),
+définissez `DNS_SERVER=192.168.1.1` (son adresse). Mettez `RDNS=off` pour désactiver les
+résolutions.
+
+## Configuration
+
+| Variable | Défaut | Rôle |
+|---|---|---|
+| `SYSLOG_PORT` | `514` | port syslog publié sur l'hôte |
+| `HTTP_PORT` | `8080` | port de l'interface web |
+| `RETENTION` | `30d` | durée de conservation des logs dans VictoriaLogs |
+| `AUTH_MODE` | `local` | `local` (page de connexion) ou `oidc`, voir [Authentification](#authentification) |
+| `AUTH_USER` / `AUTH_PASS` | vide | compte de la page de connexion (mode `local`) ; vide = pas d'authentification |
+| `AUTH_VIEWER_USER` / `AUTH_VIEWER_PASS` | vide | compte optionnel en lecture seule (mode `local`) |
+| `LOGIN_LOGO` | vide | PNG affiché sur la page de connexion, chemin dans le conteneur (mode `local`) |
+| `SESSION_TTL` | `12h` | durée de la session (les deux modes ; `OIDC_SESSION_TTL` fonctionne toujours) |
+| `OIDC_ISSUER` | vide | URL de l'issuer du fournisseur OpenID Connect (mode `oidc`) |
+| `OIDC_CLIENT_ID` / `OIDC_CLIENT_SECRET` | vide | client déclaré dans le fournisseur |
+| `OIDC_REDIRECT_URL` | vide | URL de retour de logstream, ex. `https://logs.example.org/auth/callback` |
+| `OIDC_SCOPES` | `openid profile email` | scopes demandés |
+| `OIDC_ADMIN_GROUP` | vide | seuls les membres de ce groupe sont administrateurs, les autres en lecture seule (vide = tout le monde est administrateur) |
+| `OIDC_GROUPS_CLAIM` | `groups` | revendication du jeton d'identité qui liste les groupes |
+| `RDNS` | `on` | résoudre les hôtes donnés par leur IP en noms DNS |
+| `DNS_SERVER` | vide | serveur DNS pour les résolutions inverses (`ip` ou `ip:port`) |
+| `ALLOW_PURGE` | `false` | autoriser « Supprimer tous les logs » dans les Paramètres |
+| `SYSLOG_TCP_MAX_CONNS` | `512` | connexions syslog TCP ouvertes en même temps ; au-delà, elles sont refusées |
+| `SYSLOG_TCP_IDLE` | `30m` | une connexion syslog TCP silencieuse pendant cette durée est fermée (les émetteurs se reconnectent) |
+| `EXPORT_MAX` | `100000` | nombre maximal de lignes dans un export CSV |
+| `PRESETS_FILE` | `/data/presets.json` | fichier des préréglages de tags ; liste intégrée s'il est absent (voir [docs/presets.fr.md](docs/presets.fr.md)) |
+| `DOCKER_LOGS` | `on` dans compose | collecter les logs des conteneurs Docker locaux |
+| `DOCKER_HOST` | `tcp://docker-proxy:2375` dans compose | adresse de l'API Docker (`unix:///var/run/docker.sock` hors compose) |
+| `DOCKER_BACKFILL` | `1h` | historique lu pour un conteneur vu pour la première fois |
+| `HOST_LOGS_GID` | `4` (adm) dans compose | groupe donné au conteneur pour lire les logs de l'hôte |
+| `HOST_LOGS_BACKFILL` | `1h` | historique lu à l'activation de la source « logs système de l'hôte » |
+| `HOST_LOGS_ROOT` | `/host` | emplacement de montage des répertoires de l'hôte |
+| `DMESG_BACKFILL` | `1h` | historique lu à l'activation de la source « messages du noyau » |
+| `DMESG_DEVICE` | `/dev/kmsg` | périphérique du log noyau lu par la source « messages du noyau » |
+| `AUDIT_LOG` | `/var/log/audit/audit.log` | chemin par défaut du journal d'audit Linux (modifiable dans Réglages > Sources) |
+| `AUDIT_BACKFILL` | `1h` | historique lu à l'activation de la source « audit Linux » |
+| `AUDIT_HOST` | vide | nom d'hôte des événements d'audit quand les enregistrements n'ont pas de `node=` |
+| `FILES_BACKFILL` | `1h` | historique lu à l'activation d'un fichier de logs (Paramètres > Sources de logs > Fichiers de logs) |
+| `TZ` | `Europe/Paris` | fuseau horaire des horodatages RFC 3164 (qui n'en portent pas) |
+| `BATCH_SIZE`, `FLUSH_MS`, `QUEUE_SIZE` | `1000`, `1000`, `100000` | réglage de l'ingestion |
+| `SPOOL_MAX_MB` | `1024` | taille du tampon disque des lots refusés par VictoriaLogs (`0` = pas de tampon : 15 s de tentatives, puis perte) |
+
+## Débogage
+
+- `docker compose logs -f logstream` : erreurs de réception et erreurs d'envoi vers
+  VictoriaLogs.
+- La barre du bas affiche les compteurs reçus / stockés / perdus, les messages en attente dans
+  le tampon disque et la dernière erreur de stockage.
+- <http://localhost:9428/select/vmui> : l'interface de VictoriaLogs, pour essayer des requêtes
+  LogsQL.
+- API :
+  ```bash
+  curl 'localhost:8080/api/logs?q=error&range=1h&limit=5'   # the response includes the generated LogsQL query
+  curl localhost:8080/api/stats
+  curl localhost:8080/api/tags
+  ```
+- Lancement hors Docker (Go 1.22+) : `VLOGS_URL=http://localhost:9428 DATA_DIR=./data SYSLOG_ADDR=:5514 go run .`
+
+## Mise à jour
+
+Toutes les versions d'images sont épinglées : un `docker compose pull` ou une reconstruction ne
+change jamais un composant à votre insu.
+
+| Emplacement | Image | Version |
+|---|---|---|
+| `docker-compose.yml` | `victoriametrics/victoria-logs` | `v1.52.0` |
+| `docker-compose.yml` | `tecnativa/docker-socket-proxy` | `v0.5.0` |
+| `Dockerfile` (build) | `golang` | `1.27.1-alpine3.24` |
+| `Dockerfile` (exécution) | `alpine` | `3.24.2` |
+
+Pour mettre à jour l'une d'elles :
+
+1. Lisez les notes de version : [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/changelog/),
+   [docker-socket-proxy](https://github.com/Tecnativa/docker-socket-proxy/releases),
+   [Go](https://go.dev/doc/devel/release), [Alpine](https://alpinelinux.org/releases/).
+   VictoriaLogs conserve son format de stockage entre versions mineures ; lisez le changelog
+   avant un changement de version majeure.
+2. Modifiez la version dans le fichier indiqué ci-dessus, puis lancez `docker compose up -d --build`.
+3. Vérifiez la barre du bas (reçus / stockés / perdus) et **Paramètres > Sources de logs**. Pour revenir
+   en arrière, remettez la version précédente et lancez la même commande.
+
+## Organisation du code
+
+| Fichier | Contenu |
+|---|---|
+| `main.go` | configuration, démarrage |
+| `auth.go` | authentification : OpenID Connect (découverte, PKCE, contrôle de l'ID token) et cookie de session signé |
+| `auth_local.go` | mode `local` : page de connexion (`web/login.html`), cookie de session, `LOGIN_LOGO` |
+| `syslog.go` | écoute UDP/TCP et analyse RFC 3164 / 5424 |
+| `store.go` | insertions par lots dans VictoriaLogs et requêtes LogsQL |
+| `query.go` | traduit les filtres de l'interface en LogsQL ; filtre du direct |
+| `histogram.go` | frise : division, alignement des intervalles, `/api/histogram` |
+| `hub.go` | envoie les nouveaux messages aux navigateurs (SSE) |
+| `rdns.go` | résolutions DNS inverses avec cache |
+| `export.go` | export CSV en flux |
+| `docker.go` | logs des conteneurs Docker (API, lecteurs, positions, détection du niveau) |
+| `syslogserver.go` | écoutes syslog ouvertes et fermées depuis Paramètres > Sources de logs |
+| `hostlogs.go`, `journal.go` | logs système de l'hôte : lecteur du journal systemd (sans `journalctl`) et suivi de `/var/log` |
+| `dmesg.go` | messages du noyau : lecture de `/dev/kmsg`, analyse des enregistrements, position par identifiant de démarrage et numéro de séquence |
+| `audit.go` | audit Linux : suivi de `audit.log` (rotation, position enregistrée), analyse des enregistrements, regroupement par événement, résumé |
+| `filesource.go`, `fileformats.go` | fichiers de logs : suivi (rotation, positions enregistrées, lignes de suite) et formats (détection, texte, JSON, logfmt, CLF, Traefik, expression régulière) |
+| `selflog.go` | logs internes de LogStream : `logEvent`, niveau minimal, limitation des répétitions, file bornée |
+| `tags.go` | stockage des tags de couleur |
+| `presets.go`, `presets.json` | préréglages de tags (`/api/presets`), liste intégrée |
+| `api.go` | routes HTTP `/api/*` |
+| `web/` | interface (HTML, CSS, JavaScript simple, sans étape de build), embarquée dans le binaire ; les traductions sont dans `web/app.js` (`I18N`), et dans `web/login.html` pour la page de connexion |
+
+## Remarque
+
+Avec Docker Desktop (macOS/Windows), l'IP source vue par le conteneur pour les paquets UDP est
+la passerelle Docker. Le champ `host` vient toujours de l'en-tête syslog, qui porte normalement
+le vrai nom de l'émetteur.

README.md

@@ -1,3 +1,5 @@
+English | [Français](README.fr.md)
+
 # Logstream
 
 A simple syslog sink: receives logs over UDP/TCP on port 514, stores them in
@@ -15,7 +17,9 @@ devices ──514 udp/tcp──▶ logstream (Go) ──HTTP batches──▶ Vi
 ![Schéma logique de Logstream](docs/architecture.png)
 
 - **Ingestion**: syslog (UDP/TCP) and Docker container logs both go through `sink()` (reverse DNS
-  on IP hosts), then the `Store` queue, which sends them in batches to VictoriaLogs.
+  on IP hosts, without holding up the listeners), then the `Store` queue, which sends them in
+  batches to VictoriaLogs. When VictoriaLogs is unreachable, batches are kept on disk
+  (`/data/spool`, up to `SPOOL_MAX_MB`) and sent again, oldest first, once it is back.
 - **Live view**: `sink()` also publishes each message to the `Hub`, which streams it to the
   browsers over SSE.
 - **Search**: the HTTP API turns the UI filters into LogsQL queries sent to VictoriaLogs.
@@ -46,7 +50,7 @@ Then open <http://localhost:8080>.
 - **Manual test**: `logger -n 127.0.0.1 -P 514 -d "hello error"` (util-linux) or
   `echo "<14>test ok" | nc -u -w1 127.0.0.1 514`.
 
-**Settings > Sources > Syslog** turns syslog reception on or off and chooses the protocols
+**Settings > Log sources > Syslog** turns syslog reception on or off and chooses the protocols
 (UDP, TCP) without restarting; the choice is saved in `/data/syslog.json`. It also shows the
 listening state (and the error if the port is already in use). The port itself is published
 by docker-compose: change `SYSLOG_PORT` in `.env`, then run `docker compose up -d`.
@@ -70,7 +74,7 @@ message, host and app. Words are combined with AND.
 The live view is disabled in this mode.
 
 Each row shows, from left to right: the **reception time** (server clock), the timestamp
-found in the message itself (`msg_time`), severity, host, app and message. Click a host or an
+found in the message itself (`msg_time`), severity, host, app, codes of the tags found and message. Click a host or an
 app to filter on it.
 
 Logs are indexed, searched and sorted by **reception time**: devices with a wrong clock
@@ -85,22 +89,32 @@ Shortcuts: `/` focuses the search box, `Esc` clears it. Clicking a row shows all
 The column headers stay visible while scrolling. Drag the edge of a header (received, message
 time, severity, host, app) to resize the column, double-click it to go back to the automatic
 width; the message takes the remaining space. Widths are remembered by the browser
-(**Settings > Interface > Reset column widths** restores them all). On phones the list keeps its
+(**Settings > Appearance > Reset column widths** restores them all). On phones the list keeps its
 two-line layout without columns.
 
+## Stream and Time range modes
+
+The first control of the filter bar switches between two display modes:
+
+- **Stream**: the latest logs over a sliding duration (5 min to 30 days, or all), with the live
+  view.
+- **Time range**: the logs between a start and an end date, typed in the time zone chosen in
+  Settings. ◀ and ▶ move to the previous or next range of the same length, the magnifier doubles
+  it around its middle. The live view pauses; the mode and the range are kept across reloads.
+
 ## Timeline
 
 The timeline above the list shows the volume of logs per interval, counted by **reception
 time** on the server clock (so it matches the times shown in the rows).
 
 - Hover an interval: its bounds, total and detail per severity.
-- Click a bar to zoom on that interval, or drag across several bars to zoom on the selection.
+- Click a bar to show that interval in Time range mode, or drag across several bars to show the
-  The time range then shows the zoomed period ("× Reset zoom" or any other range leaves it);
+  selection; "× Back to stream" returns to Stream mode. The list, the counters and the CSV export
-  the list, the counters and the CSV export follow the zoom, and the live view pauses.
+  follow the range.
 - In live mode the last interval grows as messages arrive, and the timeline reloads at each new
   interval. Nothing is refreshed while the browser tab is hidden; it catches up when shown again.
 
-**Settings > Interface > Timeline** (remembered per browser): scale (linear, √ by default, log),
+**Settings > Appearance > Timeline** (remembered per browser): scale (linear, √ by default, log),
 height (S/M/L: 40/80/120 px), color (stacked by severity, intensity compared with the median of
 the window: calm, burst above 3×, anomaly above 10×, or none), bars or area, division
 (automatic, about 100 intervals, or fixed: 1 s, 10 s, 1 min, 5 min, 1 h, 1 day) and refresh
@@ -127,7 +141,7 @@ colored and exported like syslog messages:
 - The severity comes from the line itself when the application writes it: JSON
   (`"level":"error"`), logfmt (`level=warn`), `[ERROR]`, or an upper-case level word at the
   start of the line (`ERROR`, `WARN`…). Otherwise it is `info`. Terminal color codes are removed.
-- **Settings > Sources** shows one label per container (`project/service`) in a single field:
+- **Settings > Log sources** shows one label per container (`project/service`) in a single field:
   followed containers in color, then not followed ones in grey; a click switches a label.
   The color identifies the compose project, and Docker app names in the log list use the
   same color. Stopped containers are hidden by default ("Show stopped containers" displays
@@ -135,8 +149,10 @@ colored and exported like syslog messages:
   to the labels shown) help with many containers. New containers are followed automatically
   unless that option is turned off. Choices are saved per compose service (or container name)
   in `/data/docker.json`, so they survive re-creations.
-- Logstream remembers the position read in each container (`/data/docker-state.json`): after a
+- Logstream remembers the position of the last line stored for each container
-  restart it resumes without losing or duplicating lines. A container seen for the first time
+  (`/data/docker-state.json`): after a restart it resumes without losing lines. The position
+  only moves once a line is in VictoriaLogs or in the disk buffer, and a full queue slows the
+  reading down instead of dropping lines. A container seen for the first time
   is read from `DOCKER_BACKFILL` ago (1 hour by default).
 - Logstream itself and the proxy below are never collected; add the label
   `logstream.exclude=true` to any other container to exclude it for good.
@@ -145,6 +161,177 @@ colored and exported like syslog messages:
 therefore goes through [docker-socket-proxy](https://github.com/Tecnativa/docker-socket-proxy),
 which only lets through listing containers, reading logs, events and engine info (`GET` only).
 
+## Host system logs
+
+Logstream can also collect the system logs of the machine hosting the stack, without
+configuring anything on the host. The source is **off by default**: turn it on in
+**Settings > Log sources > Host system logs** (saved in `/data/hostlogs.json`).
+
+- `docker-compose.yml` mounts `/var/log` and `/run/log/journal` read-only under `/host`.
+- When the host runs systemd, Logstream reads the **systemd journal** files directly (no
+  `journalctl` needed in the image): **host** is the machine name, **app** the program
+  (`SYSLOG_IDENTIFIER`), severity and facility come from the journal, and the systemd unit is
+  kept in `unit`. Otherwise it follows the text files of `/var/log` (`syslog`, `messages`,
+  `*.log`), parsed like syslog lines, with the file name in `log_file`.
+- These logs have the `host` source: the **Source** filter shows them alone.
+- When the source is turned on, the last hour is read first (`HOST_LOGS_BACKFILL`); the
+  position reached is saved in `/data/hostlogs-state.json`, so a restart neither loses nor
+  duplicates entries.
+- **Permissions**: the container runs as an unprivileged user and gets the `adm` group
+  (gid 4), which can read the journal and `/var/log` on Debian and Ubuntu. On other systems,
+  set `HOST_LOGS_GID` in `.env` to the gid of `systemd-journal`
+  (`getent group systemd-journal | cut -d: -f3`). Settings > Log sources shows a clear message when
+  access is denied.
+- Limits: journal fields compressed by journald (messages longer than 512 bytes, compressed
+  with zstd/lz4/xz) cannot be decoded without extra libraries; they are counted in Settings and
+  shown as "(compressed journal entry)". Rotated or compressed text files (`*.1`, `*.gz`) are
+  not read.
+
+## Kernel messages (dmesg)
+
+Logstream can collect the kernel ring buffer of the host (what `dmesg` shows: hardware,
+drivers, OOM killer, firewall…) by reading `/dev/kmsg`. The source is **off by default**: turn it
+on in **Settings > Log sources > Kernel messages (dmesg)** (saved in `/data/dmesg.json`).
+
+- **Device**: the container does not see `/dev/kmsg` by default. Uncomment the `devices`
+  section of `docker-compose.yml` (`/dev/kmsg:/dev/kmsg:r`, read-only), then
+  `docker compose up -d`.
+- **Permissions**: when the host has `kernel.dmesg_restrict=1` (the default on Debian and
+  Ubuntu), only processes with the `CAP_SYSLOG` capability can read the kernel log, and the
+  container runs as an unprivileged user. Either:
+  - allow it on the host: `sysctl -w kernel.dmesg_restrict=0`, kept across reboots with
+    `echo kernel.dmesg_restrict=0 | sudo tee /etc/sysctl.d/50-dmesg.conf` (every user of the
+    host can then run `dmesg`); this setting cannot be changed from a container;
+  - or run Logstream as root with the `SYSLOG` capability: uncomment `user: "0:0"` and
+    `cap_add: [SYSLOG]` in `docker-compose.yml`.
+
+  Settings > Log sources says when the device is missing or access is denied; the rest of
+  Logstream keeps working and the device is tried again every 10 seconds.
+- **Fields**: **app** is `kernel` (or the program name for messages written by programs such
+  as `systemd[1]: …`), severity and facility come from the record, **host** is `localhost`,
+  `subsystem` and `device` are kept when the kernel gives them. These logs have the `kernel`
+  source (`proto` `kmsg`): the **Source** filter shows them alone.
+- **Time**: records carry the time since boot; Logstream turns it into a date using the boot
+  time (`/proc/uptime`). As with `dmesg -T`, the time can be early after the host was suspended.
+- When the source is turned on, the last hour is read first (`DMESG_BACKFILL`, `0` = only new
+  messages). The sequence number of the last message and the boot ID are saved in
+  `/data/dmesg-state.json`: after a restart of Logstream, reading resumes after the last
+  message (no duplicates); after a reboot of the host, the new buffer is read from
+  `DMESG_BACKFILL` ago. Messages overwritten in the kernel buffer before being read are counted
+  in Settings.
+
+## Linux audit (auditd)
+
+Logstream can follow the log of the Linux audit daemon (`/var/log/audit/audit.log`): logins,
+`sudo`, watched files, syscalls matched by your `auditctl` rules… The source is **off by
+default**: turn it on in **Settings > Log sources > Linux audit (auditd)**, where the file path can
+also be changed (saved in `/data/audit.json`; `AUDIT_LOG` sets the default path).
+
+- **Mount**: uncomment `/var/log/audit:/var/log/audit:ro` in `docker-compose.yml`, then
+  `docker compose up -d`. (`/var/log` is already mounted under `/host` for the host system
+  logs, so `/host/var/log/audit/audit.log` also works as the path, without a new volume.)
+- **Permissions** (host side): auditd writes its log as `root`, mode `0600`, in a `0700`
+  directory, and Logstream runs as UID `10001`. The reliable way is the group the container
+  already gets for the host logs (`HOST_LOGS_GID`, `adm` by default):
+  ```bash
+  sudo sed -i 's/^log_group.*/log_group = adm/' /etc/audit/auditd.conf
+  sudo chgrp adm /var/log/audit && sudo chmod 750 /var/log/audit
+  sudo service auditd restart        # systemctl refuses to restart auditd on some distributions
+  ```
+  auditd then creates every rotated file with mode `0640` and group `adm`. An ACL also works
+  (`sudo setfacl -m u:10001:rx /var/log/audit && sudo setfacl -m u:10001:r /var/log/audit/audit.log`)
+  but must be applied again to each new file after a rotation. Settings > Log sources says when the
+  file is missing or access is denied; it is tried again every second.
+- **Events**: the records of one event (`SYSCALL`, `EXECVE`, `CWD`, `PATH`, `PROCTITLE`…) share
+  the same `msg=audit(time:serial)` and become **one** log, once no record came for 2 seconds.
+  The message is a summary — who, what, result — such as
+  `cedric (as root): exec apt install nginx [exec] → success` or
+  `bob: PAM:authentication via /usr/sbin/sshd from 203.0.113.9 on ssh → failed`; the raw
+  records are kept in `audit_raw` and shown under the summary (folded in compact density until
+  the line is opened). Hexadecimal values (`proctitle`, `cmd`, arguments with spaces) are
+  decoded.
+- **Fields**: `audit_type` (type of the first record, also used as **app**), `audit_serial`,
+  `auid` (login user ID), `user` (its name, or the uid when it cannot be resolved), `exe`, `res`
+  (`success` / `failed`, from `res=` or `success=`) and `key` (rule key). Severity is `warning`
+  when the result is a failure, `info` otherwise; facility `authpriv`; source `audit`. In the
+  list, audit lines have a violet left edge, a light tint and an **AUDIT** badge before the app;
+  the **Source** filter shows them alone.
+- **Names**: `user` comes from the enriched format (`log_format = ENRICHED`, the default of
+  recent auditd versions), else from the host's `/etc/passwd` when it is mounted as
+  `/host/etc/passwd`. **host** is the `node=` of the records (`name_format = hostname` in
+  `auditd.conf`), else `AUDIT_HOST`, else `/host/etc/hostname`, else `localhost`.
+- When the source is turned on, the last hour is read first (`AUDIT_BACKFILL`, `0` = only new
+  events). The inode and an offset before which every event was sent are saved in
+  `/data/audit-state.json`: after a restart, reading resumes there; after a rotation by auditd,
+  the end of `audit.log.1` is read before the new file.
+
+## Log files (Traefik, nginx, applications…)
+
+Logstream can follow the log files of other programs: Traefik, nginx, Apache, or any application
+that writes to a file. Declare each file in **Settings > Log sources > Log files** (saved in
+`/data/files.json`, up to 50): a **name** (the **app** of its entries, by default the file name),
+the **path** of the file as mounted in the container, a **format** (*Automatic* by default), an
+optional **host** (default: `/host/etc/hostname`, else `localhost`) and a switch.
+
+- **Mount** the directory read-only, e.g. in `docker-compose.yml`:
+  `- /var/log/traefik:/logs/traefik:ro`, then `docker compose up -d`. Files under `/var/log`
+  are already reachable as `/host/var/log/…`. The files must be readable by UID `10001` or the
+  group of `HOST_LOGS_GID` (`adm` by default); Settings > Log sources says when no file matches
+  or access is denied.
+- **Path**: one file, e.g. `/logs/traefik/access.log` (no wildcards). As soon as it is typed, the
+  file is checked: found or not, readable or not, its detected format and its last line.
+- **Format detection** (*Automatic*): from the last lines of the file (50 at most), the
+  structured format that at least 80 % of them follow — Traefik access JSON, JSON lines, access
+  log CLF, logfmt — else plain text. It is done again when the format is changed; the format
+  found is shown in the status of the file. Choose another format to force it (a regular
+  expression is never detected).
+- **Formats**:
+  - *Plain text*: one entry per line; the level is detected as for Docker logs, a timestamp at
+    the start of the line (ISO 8601, `2006/01/02 15:04:05`, `[02/Jan/2006:15:04:05 +0200]`,
+    `Jan  2 15:04:05`) becomes the message time and is removed from the message. Indented
+    lines (stack traces) are added to the previous entry.
+  - *JSON lines*: `msg` / `message`, `time` / `ts` / `@timestamp` (text or Unix time) and
+    `level` (word or number, pino/bunyan included) are recognized; the other keys become
+    fields, nested objects with dotted names (`req.id`).
+  - *logfmt*: `key=value` pairs, as in Traefik's default logs or Go programs; same keys as JSON.
+  - *Access log (CLF)*: Common / Combined Log Format of nginx and Apache, and Traefik's access
+    log in this format (router, server URL and duration included). Fields `client_ip`,
+    `http_method`, `http_path`, `http_status`, `http_size`, `user_agent`, `http_referer`,
+    `router`, `backend`, `duration_ms`.
+  - *Traefik access (JSON)*: the JSON access log of Traefik (`accessLog.format: json`),
+    summarized as `GET app.example.com/ 200 · 12.5 ms · app@docker · 203.0.113.9`, every key in
+    the fields. Traefik's own logs in the same file are read as JSON lines.
+  - *Regular expression* (RE2 / Go syntax): the named groups `time`, `level`, `msg`, `host`
+    and `app` fill these fields, `status` gives the level of an access log, the other groups
+    become fields, e.g. `^(?P<time>\S+ \S+) \[(?P<level>\w+)\] (?P<module>\S+): (?P<msg>.*)$`.
+    A line that does not match is added to the previous entry.
+
+  For access logs, 5xx statuses are `err`, 4xx `warning`, the others `info`. A line that does
+  not fit the chosen format is kept as plain text. Fields named like LogStream's own ones
+  (`host`, `app`…) are renamed `field_host`, `field_app`…
+- **Display**: source `file`, the path in `log_file`. In the list these lines have a green left
+  edge, a light tint and a **FILE** badge before the app; the **Source** filter shows them alone.
+- When a file is turned on, the lines of the last hour already in it are read first
+  (`FILES_BACKFILL`, `0` = only new lines; lines without a timestamp are then left out); a file
+  created later is read from its start. The inode and an offset are saved per file in
+  `/data/files-state.json`: after a restart reading resumes there; after a rotation
+  (`access.log` renamed `access.log.1`), the end of the old file is read before the new one;
+  a file truncated in place (`copytruncate`) is read again from its start.
+
+## LogStream internal logs
+
+Logstream records its own events as logs, with **host** `logstream`, **app** the component
+(`store`, `syslog`, `docker`, `host`, `dmesg`, `audit`, `files`, `main`) and the `logstream` source:
+VictoriaLogs unreachable and back, lost batches, batches refused from the disk buffer, syslog
+listener errors, Docker API or container streams lost, sources that cannot read, start and
+stop. They are on by default from level **warning**; **Settings > Log sources > LogStream internal
+logs** turns them off or changes the minimum level (saved in `/data/selflog.json`). The same
+event (component and message) is recorded at most once every 30 seconds, with the number of
+repeats left out; events go through a bounded queue and never wait for the storage, so an
+unreachable VictoriaLogs cannot make them pile up or loop. Everything still goes to
+`docker compose logs logstream` as before. In the list, these lines have a turquoise left edge
+and an **LS** badge before the app; the **Source** filter shows them alone.
+
 ## CSV export
 
 The **Export** button (next to the log count) downloads every stored log matching the
@@ -173,26 +360,111 @@ remembered per browser.
     format of the reception time: `DD/MM/YYYY HH:MM:SS` (default, the usual French display),
     with milliseconds, `YYYY-MM-DD`, 12-hour clock, ISO 8601 or Unix epoch. The time zone
     applies to every date shown.
-- **Filters**: color tags. Each tag has a keyword, a background color (the text automatically
+- **Patterns**: color tags. Each tag has a keyword, a background color (the text automatically
   switches to black or white to stay readable) and options: whole word, match case,
   regular expression, active. Tags are stored on the server in `/data/tags.json`
   (`logstream-data` volume), so they are shared by every browser. Default tags (pastel):
-  `warning` (orange), `error` (red), `ok` (green). Default tags still using the colors of
+  `warning` (orange) and `error` (red); `ok` (green) is in the *Log levels* preset. Default
-  earlier versions are switched to the pastel ones automatically.
+  tags still using the colors of earlier versions are switched to the pastel ones
-- **Interface**
+  automatically.
+  The *+ Preset…* menu adds ready-made tags: HTTP/HTTPS access logs (status codes, methods,
+  probes, bots, TLS and proxy errors), system logs (SSH, sudo, kernel, systemd, firewall),
+  applications (Docker, databases) and general patterns (log levels, IPv4 addresses). Tags
+  already in the list are skipped, and the added tags can be edited like any other. The list
+  comes from a text file you can edit (`PRESETS_FILE`): see [docs/presets.md](docs/presets.md)
+  for each preset and the file format. In a regular expression, a group named `hl`
+  (`(?<hl>…)`) colors only that part of the match: the presets use it to color the status code
+  or the method, not the text around it.
+  Each tag gets a two-digit code (`01`, `02`…) assigned by the server: it stays with the tag
+  until the tag is deleted (codes are also given to tags created by earlier versions). The
+  *Pattern* column of the log list shows, as grey badges, the codes of the active tags found in
+  each message; it has room for 3, beyond that it shows 2 and `+N`, and the tooltip lists them
+  all.
+- **Appearance**
   - *Theme*: System (follows the computer/phone preference), Light or Dark. The sun/moon
     button in the header switches between light and dark.
-  - *Log display*: font size (tiny, small, medium, large) and font: the system monospace
+  - *Log display*: font size (tiny, small, medium, large), density (normal, or compact to
-    font, or one of 12 free fonts made for dense text (JetBrains Mono, Fira Code, Source
+    fit about 50% more lines on screen) and font: the system monospace font, one of 3 narrow
-    Code Pro, IBM Plex Mono, Cascadia Code, Roboto Mono, Ubuntu Mono, Inconsolata, Red Hat
+    built-in fonts served by LogStream itself, which work offline (Inconsolata Condensed, the
-    Mono, Noto Sans Mono, Victor Mono, DM Mono). They are loaded by the browser from
+    narrowest, Iosevka and Ubuntu Mono), or one of 11 free fonts made for dense text (JetBrains
+    Mono, Fira Code, Source Code Pro, IBM Plex Mono, Cascadia Code, Roboto Mono, Inconsolata,
+    Red Hat Mono, Noto Sans Mono, Victor Mono, DM Mono). These 11 are loaded by the browser from
     [Bunny Fonts](https://fonts.bunny.net), a privacy-friendly European font service; without
-    internet access, the system font is used. Ligatures are disabled so `->` or `!=` show as typed.
+    internet access, the system font is used. Ligatures are disabled so `->` or `!=` show as
-- **Data**: "Delete all logs" permanently erases every stored log (you must type
+    typed. The densest setting is Tiny + Compact + Inconsolata Condensed.
+- **Database**: database figures read from VictoriaLogs (refreshed at most every 30 s): stored
+  lines, size on disk (index included), raw size and compression ratio, period covered with
+  the retention, lines of the last 24 h and last hour, distinct hosts and apps, free disk
+  space. Sizes use KB/MB/GB (Ko/Mo/Go in French). "Delete all logs" permanently erases every stored log (you must type
   `PURGE` to confirm). Tags and settings are kept. VictoriaLogs needs `-delete.enable`
-  (already set in `docker-compose.yml`); set `ALLOW_PURGE=false` to disable the feature.
+  (already set in `docker-compose.yml`). The feature is off by default: set `ALLOW_PURGE=true`
-  Anyone who can open the UI can purge: set `AUTH_USER` / `AUTH_PASS` if the UI is reachable
+  to allow it. Any admin can then purge: turn on [authentication](#authentication) if the UI
-  by others.
+  is reachable by others.
+
+## Authentication
+
+`AUTH_MODE` picks how the UI and the API are protected (`/healthz` always stays open):
+
+- **`local`** (default): a login page with the account `AUTH_USER` / `AUTH_PASS`; leave them
+  empty to have no authentication (for instance behind a reverse proxy that already checks). The
+  UI then shows a warning banner, which can be closed.
+- **`oidc`**: login through an OpenID Connect provider (Keycloak, Authentik, Authelia, Zitadel…),
+  authorization code flow with PKCE.
+
+In `local` mode the login page follows the theme and language of the UI. The session lasts
+`SESSION_TTL` (12 h by default), survives restarts (its signing key is in `/data/session.key`) and
+ends when `AUTH_USER` or `AUTH_PASS` changes; the log out button (top right) ends it. Failed logins
+are written in the logs with the client address (`auth: failed login for "bob" from 192.0.2.7`).
+Scripts can still call the API with HTTP Basic credentials (`curl -u user:pass`).
+
+An optional **read-only account**, `AUTH_VIEWER_USER` / `AUTH_VIEWER_PASS`, can search, follow
+the live view and export, but cannot change tags, sources or purge: those settings are greyed
+out in its UI and the API answers `403`.
+
+To show your logo on the login page, mount a PNG in the container and point `LOGIN_LOGO` to it:
+
+```yaml
+# docker-compose.yml, logstream service
+    volumes:
+      - ./logo.png:/config/logo.png:ro
+```
+```bash
+# .env
+LOGIN_LOGO=/config/logo.png
+```
+
+To use OIDC:
+
+1. In the provider, create a **confidential** client (with a secret) for logstream and register
+   the redirect URL `https://logs.example.org/auth/callback` (your address).
+2. In `.env`:
+   ```bash
+   AUTH_MODE=oidc
+   OIDC_ISSUER=https://sso.example.org/realms/home      # exactly the "issuer" of the provider
+   OIDC_CLIENT_ID=logstream
+   OIDC_CLIENT_SECRET=...
+   OIDC_REDIRECT_URL=https://logs.example.org/auth/callback
+   ```
+3. `docker compose up -d`. The logs show `oidc authentication enabled`, or the reason the
+   provider could not be read (wrong issuer, unreachable…).
+
+Opening the UI sends you to the provider's login page, then back to logstream. The session
+lasts `SESSION_TTL` (12 h by default) and survives restarts (its signing key is in
+`/data/session.key`); when it ends, the page goes through the login again. The log out button
+(top right) ends the logstream session, then opens the provider's log out page if it has one.
+
+Every user the provider accepts for this client can log in: restrict access in the provider
+(Keycloak: client roles or a dedicated realm; Authentik: application bindings). Logins are written
+in the logstream logs (`oidc: alice logged in`). With an `https` redirect URL, the cookies are
+only sent over HTTPS: logstream must be reached through a TLS reverse proxy.
+
+To give read-only access to some users, set `OIDC_ADMIN_GROUP` (for instance
+`logstream-admins`): only its members are admins, the others are read-only. The groups are read
+from the `groups` claim of the ID token (`OIDC_GROUPS_CLAIM` to use another one); in Keycloak,
+add a "Group Membership" mapper to the client (a leading `/` is ignored).
+
+Whatever the mode, every answer carries security headers (Content-Security-Policy,
+X-Frame-Options…), and the API refuses changes sent from another site (cross-site requests).
 
 ## Host names (reverse DNS)
 
@@ -204,11 +476,6 @@ on display, and the host filter shows `name (IP)`.
 The container uses Docker's DNS, which forwards to the host's resolvers. If your local names
 are only known by your router or a local DNS (Pi-hole, AdGuard, Unbound…), set
 `DNS_SERVER=192.168.1.1` (its address). Set `RDNS=off` to disable lookups.
-- **Color tags**: each tag has a keyword, a background color (the text automatically
-  switches to black or white to stay readable) and options: whole word, match case,
-  regular expression, active. Tags are stored in `/data/tags.json` (`logstream-data` volume).
-  Default tags (pastel): `warning` (orange), `error` (red), `ok` (green). Default tags
-  still using the colors of earlier versions are switched to the pastel ones automatically.
 
 ## Configuration
 
@@ -217,21 +484,45 @@ are only known by your router or a local DNS (Pi-hole, AdGuard, Unbound…), set
 | `SYSLOG_PORT` | `514` | syslog port published on the host |
 | `HTTP_PORT` | `8080` | web UI port |
 | `RETENTION` | `30d` | how long VictoriaLogs keeps logs |
-| `AUTH_USER` / `AUTH_PASS` | empty | HTTP Basic authentication for the UI |
+| `AUTH_MODE` | `local` | `local` (login page) or `oidc`, see [Authentication](#authentication) |
+| `AUTH_USER` / `AUTH_PASS` | empty | account of the login page (`local` mode); empty = no authentication |
+| `AUTH_VIEWER_USER` / `AUTH_VIEWER_PASS` | empty | optional read-only account (`local` mode) |
+| `LOGIN_LOGO` | empty | PNG shown on the login page, path inside the container (`local` mode) |
+| `SESSION_TTL` | `12h` | session lifetime (both modes; `OIDC_SESSION_TTL` still works) |
+| `OIDC_ISSUER` | empty | issuer URL of the OpenID Connect provider (`oidc` mode) |
+| `OIDC_CLIENT_ID` / `OIDC_CLIENT_SECRET` | empty | client registered in the provider |
+| `OIDC_REDIRECT_URL` | empty | callback URL of logstream, e.g. `https://logs.example.org/auth/callback` |
+| `OIDC_SCOPES` | `openid profile email` | requested scopes |
+| `OIDC_ADMIN_GROUP` | empty | only members of this group are admins, the others read-only (empty = everyone is admin) |
+| `OIDC_GROUPS_CLAIM` | `groups` | ID token claim that lists the groups |
 | `RDNS` | `on` | resolve IP hosts to DNS names |
 | `DNS_SERVER` | empty | DNS server for reverse lookups (`ip` or `ip:port`) |
-| `ALLOW_PURGE` | `true` | allow "Delete all logs" in Settings |
+| `ALLOW_PURGE` | `false` | allow "Delete all logs" in Settings |
+| `SYSLOG_TCP_MAX_CONNS` | `512` | syslog TCP connections open at once; more are refused |
+| `SYSLOG_TCP_IDLE` | `30m` | a syslog TCP connection silent this long is closed (senders reconnect) |
 | `EXPORT_MAX` | `100000` | maximum number of rows in a CSV export |
+| `PRESETS_FILE` | `/data/presets.json` | color tag presets file; the built-in list when missing (see [docs/presets.md](docs/presets.md)) |
 | `DOCKER_LOGS` | `on` in compose | collect the logs of the local Docker containers |
 | `DOCKER_HOST` | `tcp://docker-proxy:2375` in compose | Docker API address (`unix:///var/run/docker.sock` outside compose) |
 | `DOCKER_BACKFILL` | `1h` | history read from a container seen for the first time |
+| `HOST_LOGS_GID` | `4` (adm) in compose | group given to the container to read the host logs |
+| `HOST_LOGS_BACKFILL` | `1h` | history read when the host system logs source is turned on |
+| `HOST_LOGS_ROOT` | `/host` | where the host directories are mounted |
+| `DMESG_BACKFILL` | `1h` | history read when the kernel messages source is turned on |
+| `DMESG_DEVICE` | `/dev/kmsg` | kernel log device read by the kernel messages source |
+| `AUDIT_LOG` | `/var/log/audit/audit.log` | default path of the Linux audit log (can be changed in Settings > Log sources) |
+| `AUDIT_BACKFILL` | `1h` | history read when the Linux audit source is turned on |
+| `AUDIT_HOST` | empty | host name of the audit events when the records carry no `node=` |
+| `FILES_BACKFILL` | `1h` | history read when a log file is turned on (Settings > Log sources > Log files) |
 | `TZ` | `Europe/Paris` | time zone for RFC 3164 timestamps (which carry none) |
 | `BATCH_SIZE`, `FLUSH_MS`, `QUEUE_SIZE` | `1000`, `1000`, `100000` | ingestion tuning |
+| `SPOOL_MAX_MB` | `1024` | disk buffer size for batches VictoriaLogs could not take (`0` = no buffer: retried for 15 s, then dropped) |
 
 ## Debugging
 
 - `docker compose logs -f logstream`: receive errors and errors sending to VictoriaLogs.
-- The bottom bar shows received / stored / dropped counters and the last storage error.
+- The bottom bar shows received / stored / dropped counters, the messages waiting in the disk
+  buffer, and the last storage error.
 - <http://localhost:9428/select/vmui>: VictoriaLogs' own UI to try LogsQL queries.
 - API:
   ```bash
@@ -261,14 +552,16 @@ To update one of them:
    VictoriaLogs keeps its storage format across minor versions; read the changelog before a
    major version change.
 2. Change the version in the file above, then run `docker compose up -d --build`.
-3. Check the bottom bar (received / stored / dropped) and **Settings > Sources**. To go back,
+3. Check the bottom bar (received / stored / dropped) and **Settings > Log sources**. To go back,
    restore the previous version and run the same command.
 
 ## Code layout
 
 | File | Contents |
 |---|---|
-| `main.go` | configuration, startup, authentication |
+| `main.go` | configuration, startup |
+| `auth.go` | authentication: OpenID Connect (discovery, PKCE, ID token checks) and the signed session cookie |
+| `auth_local.go` | `local` mode: login page (`web/login.html`), session cookie, `LOGIN_LOGO` |
 | `syslog.go` | UDP/TCP listeners and RFC 3164 / 5424 parsing |
 | `store.go` | batched inserts into VictoriaLogs and LogsQL queries |
 | `query.go` | turns UI filters into LogsQL; live-view filter |
@@ -277,10 +570,16 @@ To update one of them:
 | `rdns.go` | cached reverse DNS lookups |
 | `export.go` | streamed CSV export |
 | `docker.go` | Docker container logs (API, followers, positions, level detection) |
-| `syslogserver.go` | syslog listeners opened and closed from Settings > Sources |
+| `syslogserver.go` | syslog listeners opened and closed from Settings > Log sources |
+| `hostlogs.go`, `journal.go` | host system logs: systemd journal reader (no `journalctl`) and `/var/log` follower |
+| `dmesg.go` | kernel messages: `/dev/kmsg` reader, record parsing, position by boot ID and sequence number |
+| `audit.go` | Linux audit: `audit.log` follower (rotation, saved offset), record parsing, grouping by event, summary |
+| `filesource.go`, `fileformats.go` | log files: follower (rotation, saved offsets, continuation lines) and formats (detection, text, JSON, logfmt, CLF, Traefik, regular expression) |
+| `selflog.go` | LogStream internal logs: `logEvent`, minimum level, throttling, bounded queue |
 | `tags.go` | color tag storage |
+| `presets.go`, `presets.json` | color tag presets (`/api/presets`), built-in list |
 | `api.go` | `/api/*` HTTP routes |
-| `web/` | UI (HTML, CSS, plain JavaScript, no build step), embedded in the binary; translations live in `web/app.js` (`I18N`) |
+| `web/` | UI (HTML, CSS, plain JavaScript, no build step), embedded in the binary; translations live in `web/app.js` (`I18N`), and in `web/login.html` for the login page |
 
 ## Note
 

api.go

@@ -16,11 +16,18 @@ type API struct {
 	store      *Store
 	hub        *Hub
 	tags       *TagStore
+	presets    string // presets file, built-in presets when missing
 	rdns       *ReverseDNS
 	allowPurge bool
 	exportMax  int
 	docker     *DockerManager // nil when DOCKER_LOGS is off
 	syslog     *SyslogServer
+	host       *HostLogs
+	dmesg      *Dmesg
+	audit      *Audit
+	files      *Files
+	selfLog    *SelfLog
+	dbstats    dbStatsCache
 }
 
 func (a *API) Routes(mux *http.ServeMux) {
@@ -30,9 +37,11 @@ func (a *API) Routes(mux *http.ServeMux) {
 	mux.HandleFunc("GET /api/facets", a.facets)
 	mux.HandleFunc("GET /api/stream", a.stream)
 	mux.HandleFunc("GET /api/stats", a.stats)
+	mux.HandleFunc("GET /api/dbstats", a.dbStats)
 	mux.HandleFunc("GET /api/tags", a.listTags)
 	mux.HandleFunc("POST /api/tags", a.createTag)
 	mux.HandleFunc("POST /api/tags/reset", a.resetTags)
+	mux.HandleFunc("GET /api/presets", a.listPresets)
 	mux.HandleFunc("PUT /api/tags/{id}", a.updateTag)
 	mux.HandleFunc("DELETE /api/tags/{id}", a.deleteTag)
 	mux.HandleFunc("GET /api/purge", a.purgeStatus)
@@ -42,6 +51,132 @@ func (a *API) Routes(mux *http.ServeMux) {
 	mux.HandleFunc("PUT /api/syslog", a.syslogConfigure)
 	mux.HandleFunc("GET /api/docker", a.dockerStatus)
 	mux.HandleFunc("PUT /api/docker", a.dockerConfigure)
+	mux.HandleFunc("GET /api/hostlogs", a.hostLogsStatus)
+	mux.HandleFunc("PUT /api/hostlogs", a.hostLogsConfigure)
+	mux.HandleFunc("GET /api/dmesg", a.dmesgStatus)
+	mux.HandleFunc("PUT /api/dmesg", a.dmesgConfigure)
+	mux.HandleFunc("GET /api/audit", a.auditStatus)
+	mux.HandleFunc("PUT /api/audit", a.auditConfigure)
+	mux.HandleFunc("GET /api/files", a.filesStatus)
+	mux.HandleFunc("PUT /api/files", a.filesConfigure)
+	mux.HandleFunc("POST /api/files/detect", a.filesDetect)
+	mux.HandleFunc("GET /api/selflog", a.selfLogStatus)
+	mux.HandleFunc("PUT /api/selflog", a.selfLogConfigure)
+}
+
+// GET /api/audit: state of the Linux audit source (Settings > Sources).
+func (a *API) auditStatus(w http.ResponseWriter, r *http.Request) {
+	writeJSON(w, http.StatusOK, a.audit.Status())
+}
+
+// PUT /api/audit {"enabled": bool, "path": "/var/log/audit/audit.log"}
+func (a *API) auditConfigure(w http.ResponseWriter, r *http.Request) {
+	var cfg auditConfig
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&cfg); err != nil {
+		writeErr(w, http.StatusBadRequest, err)
+		return
+	}
+	if err := a.audit.Configure(cfg); err != nil {
+		writeErr(w, http.StatusInternalServerError, err)
+		return
+	}
+	log.Printf("audit source changed from %s: %+v", r.RemoteAddr, cfg)
+	writeJSON(w, http.StatusOK, a.audit.Status())
+}
+
+// GET /api/files: the log files of the "Files" source (Settings > Sources).
+func (a *API) filesStatus(w http.ResponseWriter, r *http.Request) {
+	writeJSON(w, http.StatusOK, a.files.Status())
+}
+
+// PUT /api/files {"sources": [{"id", "name", "path", "format" (auto or a format), "regex", "host", "enabled"}]}
+// (no id: a new file).
+func (a *API) filesConfigure(w http.ResponseWriter, r *http.Request) {
+	var cfg filesConfig
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 64<<10)).Decode(&cfg); err != nil {
+		writeErr(w, http.StatusBadRequest, err)
+		return
+	}
+	if err := a.files.Configure(cfg); err != nil {
+		writeErr(w, http.StatusBadRequest, err)
+		return
+	}
+	log.Printf("file sources changed from %s: %d files", r.RemoteAddr, len(cfg.Sources))
+	writeJSON(w, http.StatusOK, a.files.Status())
+}
+
+// POST /api/files/detect {"path": "/logs/traefik/access.log"}: whether the file
+// can be read, its detected format and its last line (Settings > Sources). A
+// POST, so that read-only accounts cannot read files through it.
+func (a *API) filesDetect(w http.ResponseWriter, r *http.Request) {
+	var req struct {
+		Path string `json:"path"`
+	}
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&req); err != nil {
+		writeErr(w, http.StatusBadRequest, err)
+		return
+	}
+	writeJSON(w, http.StatusOK, DetectFile(req.Path))
+}
+
+// GET /api/selflog: state of the LogStream internal logs (Settings > Sources).
+func (a *API) selfLogStatus(w http.ResponseWriter, r *http.Request) {
+	writeJSON(w, http.StatusOK, a.selfLog.Status())
+}
+
+// PUT /api/selflog {"enabled": bool, "min_level": "warning"}
+func (a *API) selfLogConfigure(w http.ResponseWriter, r *http.Request) {
+	var cfg selfLogConfig
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&cfg); err != nil {
+		writeErr(w, http.StatusBadRequest, err)
+		return
+	}
+	if err := a.selfLog.Configure(cfg); err != nil {
+		writeErr(w, http.StatusInternalServerError, err)
+		return
+	}
+	log.Printf("internal logs changed from %s: %+v", r.RemoteAddr, cfg)
+	writeJSON(w, http.StatusOK, a.selfLog.Status())
+}
+
+// GET /api/dmesg: state of the kernel messages source (Settings > Sources).
+func (a *API) dmesgStatus(w http.ResponseWriter, r *http.Request) {
+	writeJSON(w, http.StatusOK, a.dmesg.Status())
+}
+
+// PUT /api/dmesg {"enabled": bool}
+func (a *API) dmesgConfigure(w http.ResponseWriter, r *http.Request) {
+	var cfg dmesgConfig
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&cfg); err != nil {
+		writeErr(w, http.StatusBadRequest, err)
+		return
+	}
+	if err := a.dmesg.Configure(cfg); err != nil {
+		writeErr(w, http.StatusInternalServerError, err)
+		return
+	}
+	log.Printf("kernel messages source changed from %s: %+v", r.RemoteAddr, cfg)
+	writeJSON(w, http.StatusOK, a.dmesg.Status())
+}
+
+// GET /api/hostlogs: state of the host system logs source (Settings > Sources).
+func (a *API) hostLogsStatus(w http.ResponseWriter, r *http.Request) {
+	writeJSON(w, http.StatusOK, a.host.Status())
+}
+
+// PUT /api/hostlogs {"enabled": bool}
+func (a *API) hostLogsConfigure(w http.ResponseWriter, r *http.Request) {
+	var cfg hostLogsConfig
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&cfg); err != nil {
+		writeErr(w, http.StatusBadRequest, err)
+		return
+	}
+	if err := a.host.Configure(cfg); err != nil {
+		writeErr(w, http.StatusInternalServerError, err)
+		return
+	}
+	log.Printf("host system logs changed from %s: %+v", r.RemoteAddr, cfg)
+	writeJSON(w, http.StatusOK, a.host.Status())
 }
 
 // GET /api/syslog: syslog reception state (Settings > Sources).
@@ -289,10 +424,25 @@ func (a *API) stats(w http.ResponseWriter, r *http.Request) {
 	writeJSON(w, http.StatusOK, s)
 }
 
+// GET /api/dbstats[?refresh=1]: size and content of the VictoriaLogs base
+// (Settings > Data), cached for 30 s.
+func (a *API) dbStats(w http.ResponseWriter, r *http.Request) {
+	st, err := a.dbstats.get(r.Context(), a.store, r.URL.Query().Get("refresh") == "1")
+	if err != nil {
+		writeErr(w, http.StatusBadGateway, err)
+		return
+	}
+	writeJSON(w, http.StatusOK, st)
+}
+
 func (a *API) listTags(w http.ResponseWriter, r *http.Request) {
 	writeJSON(w, http.StatusOK, a.tags.List())
 }
 
+func (a *API) listPresets(w http.ResponseWriter, r *http.Request) {
+	writeJSON(w, http.StatusOK, loadPresets(a.presets))
+}
+
 func decodeTag(w http.ResponseWriter, r *http.Request) (Tag, error) {
 	var t Tag
 	err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 64*1024)).Decode(&t)

audit.go

@@ -0,0 +1,830 @@
+package main
+
+import (
+	"bufio"
+	"bytes"
+	"context"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"io"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+	"unicode/utf8"
+)
+
+// auditConfig is saved in /data/audit.json and edited in Settings > Sources.
+type auditConfig struct {
+	Enabled bool   `json:"enabled"`
+	Path    string `json:"path"` // empty: AUDIT_LOG (/var/log/audit/audit.log)
+}
+
+// auditPos is where reading resumes: the inode of the file and an offset
+// before which every event was sent.
+type auditPos struct {
+	Ino uint64 `json:"ino"`
+	Off int64  `json:"off"`
+}
+
+// Audit follows the log of the Linux audit daemon (auditd) and sends one
+// entry per audit event: the records of an event (SYSCALL, EXECVE, PATH…)
+// share the same msg=audit(time:serial) and are grouped together.
+type Audit struct {
+	defPath   string
+	hostRoot  string // to read the host's /etc/passwd and /etc/hostname when mounted
+	hostName  string // AUDIT_HOST: host name of the entries when records have no node=
+	sink      func(*Entry)
+	backfill  time.Duration
+	cfgPath   string
+	statePath string
+	wake      chan struct{}
+
+	// Owned by the Run goroutine.
+	pos       auditPos // checkpoint saved in /data
+	readOff   int64    // where the next read starts (after the lines of open events)
+	known     bool
+	dirty     bool
+	notBefore time.Time
+	asm       *auditAssembler
+	users     *userNames
+
+	mu        sync.Mutex
+	cfg       auditConfig
+	reset     bool
+	open      bool
+	read      uint64
+	errCode   string
+	errDetail string
+}
+
+func NewAudit(defPath, hostRoot, hostName, dataDir string, backfill time.Duration, sink func(*Entry)) *Audit {
+	a := &Audit{
+		defPath:   defPath,
+		hostRoot:  hostRoot,
+		hostName:  hostName,
+		sink:      sink,
+		backfill:  backfill,
+		cfgPath:   filepath.Join(dataDir, "audit.json"),
+		statePath: filepath.Join(dataDir, "audit-state.json"),
+		wake:      make(chan struct{}, 1),
+		asm:       newAuditAssembler(),
+		users:     &userNames{path: filepath.Join(hostRoot, "etc/passwd")},
+	}
+	if b, err := os.ReadFile(a.cfgPath); err == nil {
+		_ = json.Unmarshal(b, &a.cfg)
+	}
+	if b, err := os.ReadFile(a.statePath); err == nil && json.Unmarshal(b, &a.pos) == nil {
+		a.known, a.readOff = true, a.pos.Off
+	}
+	if a.hostName == "" {
+		a.hostName = readTrim(filepath.Join(hostRoot, "etc/hostname"))
+	}
+	return a
+}
+
+func (a *Audit) path() string {
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	if a.cfg.Path != "" {
+		return a.cfg.Path
+	}
+	return a.defPath
+}
+
+// Run polls the file every second while the source is enabled.
+func (a *Audit) Run(ctx context.Context) {
+	tick := time.NewTicker(time.Second)
+	defer tick.Stop()
+	n := 0
+	for {
+		select {
+		case <-ctx.Done():
+			a.emit(a.asm.flushAll())
+			a.checkpoint()
+			a.saveState()
+			return
+		case <-tick.C:
+		case <-a.wake:
+		}
+		a.mu.Lock()
+		enabled, reset := a.cfg.Enabled, a.reset
+		a.reset = false
+		a.mu.Unlock()
+		if reset {
+			// Disabled then enabled again (or another file): start over from
+			// AUDIT_BACKFILL ago rather than reading everything in between.
+			a.asm = newAuditAssembler()
+			a.pos, a.readOff, a.known, a.dirty = auditPos{}, 0, false, true
+		}
+		if enabled {
+			a.poll(time.Now())
+		}
+		if n++; n%5 == 0 || reset {
+			a.saveState()
+		}
+	}
+}
+
+func (a *Audit) saveState() {
+	if !a.dirty {
+		return
+	}
+	a.dirty = false
+	if err := writeJSONFile(a.statePath, a.pos); err != nil {
+		logEvent("audit", 4, "saving position: %v", err)
+	}
+}
+
+func (a *Audit) fail(err error) {
+	code := "read"
+	switch {
+	case errors.Is(err, fs.ErrNotExist):
+		code = "not_found"
+	case errors.Is(err, fs.ErrPermission):
+		code = "permission"
+	}
+	a.mu.Lock()
+	changed := code != a.errCode
+	a.open, a.errCode, a.errDetail = false, code, err.Error()
+	a.mu.Unlock()
+	if changed {
+		logEvent("audit", 4, "cannot read the audit log: %v", err)
+	}
+}
+
+const maxAuditRead = 4 << 20 // per poll
+
+// poll reads the lines added since the previous one, follows rotations and
+// sends the events that are complete.
+func (a *Audit) poll(now time.Time) {
+	path := a.path()
+	fi, err := os.Stat(path)
+	if err != nil {
+		a.fail(err)
+		return
+	}
+	ino := inodeOf(fi)
+	if !a.known {
+		// First read: the last AUDIT_BACKFILL (0: only new events).
+		a.pos, a.readOff, a.known, a.dirty = auditPos{Ino: ino}, 0, true, true
+		a.notBefore = time.Time{}
+		if a.backfill <= 0 {
+			a.readOff = fi.Size()
+		} else {
+			a.notBefore = now.Add(-a.backfill)
+		}
+	}
+	if a.pos.Ino != ino {
+		// Rotated by auditd (audit.log -> audit.log.1): finish the old file,
+		// then read the new one from its start.
+		if old, err := os.Stat(path + ".1"); err == nil && inodeOf(old) == a.pos.Ino {
+			for {
+				off, more, err := a.readFrom(path+".1", a.readOff, now)
+				if err != nil || !more {
+					break
+				}
+				a.readOff = off
+			}
+		}
+		a.emit(a.asm.flushAll())
+		a.pos, a.readOff, a.dirty = auditPos{Ino: ino}, 0, true
+	} else if fi.Size() < a.readOff {
+		a.emit(a.asm.flushAll()) // truncated
+		a.readOff = 0
+	}
+	if fi.Size() > a.readOff {
+		off, _, err := a.readFrom(path, a.readOff, now)
+		if err != nil {
+			a.fail(err)
+			return
+		}
+		a.readOff = off
+	}
+	a.mu.Lock()
+	if !a.open {
+		a.open, a.errCode, a.errDetail = true, "", ""
+	}
+	a.mu.Unlock()
+	a.emit(a.asm.flush(now, auditCloseDelay))
+	a.checkpoint()
+}
+
+// checkpoint saves an offset before which every event is sent: the start of
+// the oldest event still open, or the read offset.
+func (a *Audit) checkpoint() {
+	off := a.readOff
+	if o, ok := a.asm.minOff(); ok && o < off {
+		off = o
+	}
+	if off != a.pos.Off {
+		a.pos.Off, a.dirty = off, true
+	}
+}
+
+// readFrom feeds the complete lines written after off to the assembler and
+// returns the new offset; more tells whether the read window was full.
+func (a *Audit) readFrom(path string, off int64, now time.Time) (int64, bool, error) {
+	f, err := os.Open(path)
+	if err != nil {
+		return off, false, err
+	}
+	defer f.Close()
+	buf, err := io.ReadAll(io.NewSectionReader(f, off, maxAuditRead))
+	if err != nil {
+		return off, false, err
+	}
+	full := len(buf) == maxAuditRead
+	end := bytes.LastIndexByte(buf, '\n')
+	if end < 0 {
+		if !full {
+			return off, false, nil // partial line: wait for its end
+		}
+		end = len(buf) - 1
+	}
+	start := off
+	for _, line := range bytes.Split(buf[:end], []byte{'\n'}) {
+		a.emit(a.asm.add(string(line), start, now))
+		start += int64(len(line)) + 1
+	}
+	return off + int64(end) + 1, full, nil
+}
+
+func (a *Audit) emit(evs []*auditEvent) {
+	for _, ev := range evs {
+		if !a.notBefore.IsZero() && ev.Time.Before(a.notBefore) {
+			continue
+		}
+		a.sink(auditEntry(ev, a.users, a.hostName, time.Now()))
+		a.mu.Lock()
+		a.read++
+		a.mu.Unlock()
+	}
+}
+
+func inodeOf(fi os.FileInfo) uint64 {
+	if st, ok := fi.Sys().(*syscall.Stat_t); ok {
+		return uint64(st.Ino)
+	}
+	return 0
+}
+
+// Configure saves and applies the configuration.
+func (a *Audit) Configure(cfg auditConfig) error {
+	cfg.Path = strings.TrimSpace(cfg.Path)
+	a.mu.Lock()
+	if (a.cfg.Enabled && !cfg.Enabled) || cfg.Path != a.cfg.Path {
+		a.reset = true
+		a.open, a.errCode, a.errDetail = false, "", ""
+	}
+	a.cfg = cfg
+	a.mu.Unlock()
+	select {
+	case a.wake <- struct{}{}:
+	default:
+	}
+	return writeJSONFile(a.cfgPath, cfg)
+}
+
+// Status is the state shown in Settings > Sources.
+func (a *Audit) Status() map[string]any {
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	return map[string]any{
+		"enabled":     a.cfg.Enabled,
+		"path":        a.cfg.Path,
+		"defaultPath": a.defPath,
+		"open":        a.open && a.cfg.Enabled,
+		"read":        a.read,
+		"code":        a.errCode,
+		"error":       a.errDetail,
+	}
+}
+
+/* ---------- Records and events ---------- */
+
+// auditCloseDelay: an event is complete when no line of it came for this
+// long, or when the log has moved this far past its time (auditd does not
+// write the EOE record that ends multi-record events).
+const auditCloseDelay = 2 * time.Second
+
+type auditField struct {
+	k, v string
+	bare bool // written without quotes: may be hexadecimal
+}
+
+type auditRecord struct {
+	Type   string
+	Fields []auditField
+	Raw    string
+}
+
+type auditEvent struct {
+	ID      string // "1700000000.123:456"
+	Serial  string
+	Time    time.Time
+	Node    string
+	Records []auditRecord
+	off     int64     // offset of its first line in the file
+	seen    time.Time // when its last line was read
+	order   int
+}
+
+// parseAuditLine decodes "[node=N ]type=T msg=audit(SEC.MS:SERIAL): k=v …".
+func parseAuditLine(line string) (rec auditRecord, id, node string, ok bool) {
+	line = strings.TrimRight(line, "\r")
+	rec.Raw = line
+	head, rest, found := strings.Cut(line, "): ")
+	if !found {
+		head, found = strings.CutSuffix(line, ")")
+		if !found {
+			return rec, "", "", false
+		}
+	}
+	pre, ts, found := strings.Cut(head, "msg=audit(")
+	if !found || ts == "" {
+		return rec, "", "", false
+	}
+	for _, f := range parseAuditKV(pre) {
+		switch f.k {
+		case "type":
+			rec.Type = f.v
+		case "node":
+			node = f.v
+		}
+	}
+	if rec.Type == "" {
+		return rec, "", "", false
+	}
+	// Enriched format (log_format=ENRICHED): resolved values after a GS byte.
+	main, enriched, _ := strings.Cut(rest, "\x1d")
+	rec.Fields = parseAuditKV(main)
+	for _, f := range parseAuditKV(enriched) {
+		rec.Fields = append(rec.Fields, auditField{k: "@" + f.k, v: f.v})
+	}
+	return rec, ts, node, true
+}
+
+// hexFields are the fields auditd writes in hexadecimal when they contain
+// spaces, quotes or control characters (and then without quotes).
+var hexFields = map[string]bool{
+	"proctitle": true, "cmd": true, "exe": true, "comm": true, "key": true, "acct": true,
+	"name": true, "cwd": true, "data": true, "path": true, "old": true, "new": true,
+}
+
+// parseAuditKV splits "k=v k2="v 2" msg='k3=v3 …'"; the fields of a quoted
+// msg='…' (user space records) follow the outer ones.
+func parseAuditKV(s string) []auditField {
+	var out []auditField
+	var inner []auditField
+	for i := 0; i < len(s); {
+		for i < len(s) && s[i] == ' ' {
+			i++
+		}
+		eq := strings.IndexByte(s[i:], '=')
+		if eq <= 0 {
+			break
+		}
+		k := s[i : i+eq]
+		if sp := strings.IndexByte(k, ' '); sp >= 0 { // word without value
+			i += sp + 1
+			continue
+		}
+		i += eq + 1
+		var v string
+		quoted := i < len(s) && (s[i] == '"' || s[i] == '\'')
+		if quoted {
+			q := s[i]
+			end := strings.IndexByte(s[i+1:], q)
+			if end < 0 {
+				end = len(s) - i - 1
+			}
+			v = s[i+1 : i+1+end]
+			i += end + 2
+		} else {
+			end := strings.IndexByte(s[i:], ' ')
+			if end < 0 {
+				end = len(s) - i
+			}
+			v = s[i : i+end]
+			i += end
+		}
+		if k == "msg" && quoted && strings.Contains(v, "=") {
+			inner = append(inner, parseAuditKV(v)...)
+			continue
+		}
+		if !quoted && hexFields[k] {
+			v = decodeHex(k, v)
+		}
+		if !quoted && (v == "(null)" || v == "?") {
+			v = ""
+		}
+		out = append(out, auditField{k: k, v: v, bare: !quoted})
+	}
+	return append(out, inner...)
+}
+
+// decodeHex decodes a hexadecimal value, or returns it unchanged.
+func decodeHex(k, v string) string {
+	if len(v) < 2 || len(v)%2 != 0 {
+		return v
+	}
+	b, err := hex.DecodeString(v)
+	if err != nil {
+		return v
+	}
+	if k == "proctitle" {
+		b = bytes.ReplaceAll(bytes.TrimRight(b, "\x00"), []byte{0}, []byte{' '})
+	}
+	if !utf8.Valid(b) {
+		return v
+	}
+	return string(b)
+}
+
+func isDigits(s string) bool {
+	for i := 0; i < len(s); i++ {
+		if s[i] < '0' || s[i] > '9' {
+			return false
+		}
+	}
+	return s != ""
+}
+
+// parseAuditTime decodes "1700000000.123:456" into a time and a serial.
+func parseAuditTime(id string) (time.Time, string) {
+	ts, serial, _ := strings.Cut(id, ":")
+	sec, frac, _ := strings.Cut(ts, ".")
+	s, err := strconv.ParseInt(sec, 10, 64)
+	if err != nil {
+		return time.Time{}, serial
+	}
+	ms, _ := strconv.ParseInt((frac + "000")[:3], 10, 64)
+	return time.Unix(s, ms*int64(time.Millisecond)), serial
+}
+
+// auditAssembler groups the records of each event.
+type auditAssembler struct {
+	open   map[string]*auditEvent
+	newest time.Time // most recent event time read
+	n      int
+}
+
+func newAuditAssembler() *auditAssembler {
+	return &auditAssembler{open: map[string]*auditEvent{}}
+}
+
+const maxOpenAudit = 2000
+
+// add takes one line read at offset off and returns the events it completes
+// (EOE record, or too many events open).
+func (a *auditAssembler) add(line string, off int64, now time.Time) []*auditEvent {
+	if strings.TrimSpace(line) == "" {
+		return nil
+	}
+	rec, id, node, ok := parseAuditLine(line)
+	if !ok {
+		return nil
+	}
+	ev := a.open[id]
+	if rec.Type == "EOE" {
+		if ev == nil {
+			return nil
+		}
+		delete(a.open, id)
+		return []*auditEvent{ev}
+	}
+	if ev == nil {
+		t, serial := parseAuditTime(id)
+		a.n++
+		ev = &auditEvent{ID: id, Serial: serial, Time: t, Node: node, off: off, order: a.n}
+		a.open[id] = ev
+		if t.After(a.newest) {
+			a.newest = t
+		}
+	}
+	ev.Records = append(ev.Records, rec)
+	ev.seen = now
+	if len(a.open) > maxOpenAudit {
+		return a.closeOldest(len(a.open) - maxOpenAudit/2)
+	}
+	return nil
+}
+
+// flush returns the events with no new line for delay, or that the log has
+// left delay behind, in the order they started.
+func (a *auditAssembler) flush(now time.Time, delay time.Duration) []*auditEvent {
+	var out []*auditEvent
+	for id, ev := range a.open {
+		if now.Sub(ev.seen) >= delay || a.newest.Sub(ev.Time) >= delay {
+			out = append(out, ev)
+			delete(a.open, id)
+		}
+	}
+	sortEvents(out)
+	return out
+}
+
+func (a *auditAssembler) flushAll() []*auditEvent {
+	return a.closeOldest(len(a.open))
+}
+
+func (a *auditAssembler) closeOldest(n int) []*auditEvent {
+	all := make([]*auditEvent, 0, len(a.open))
+	for _, ev := range a.open {
+		all = append(all, ev)
+	}
+	sortEvents(all)
+	all = all[:n]
+	for _, ev := range all {
+		delete(a.open, ev.ID)
+	}
+	return all
+}
+
+func sortEvents(evs []*auditEvent) {
+	sort.Slice(evs, func(i, j int) bool { return evs[i].order < evs[j].order })
+}
+
+func (a *auditAssembler) minOff() (int64, bool) {
+	var m int64
+	ok := false
+	for _, ev := range a.open {
+		if !ok || ev.off < m {
+			m, ok = ev.off, true
+		}
+	}
+	return m, ok
+}
+
+/* ---------- Entries ---------- */
+
+// get returns the first non-empty value of a field across the records.
+func (ev *auditEvent) get(k string) string {
+	for _, r := range ev.Records {
+		for _, f := range r.Fields {
+			if f.k == k && f.v != "" {
+				return f.v
+			}
+		}
+	}
+	return ""
+}
+
+func (ev *auditEvent) record(typ string) *auditRecord {
+	for i := range ev.Records {
+		if ev.Records[i].Type == typ {
+			return &ev.Records[i]
+		}
+	}
+	return nil
+}
+
+// result is "success", "failed" or "" from res= or success=.
+func (ev *auditEvent) result() string {
+	v := strings.ToLower(firstNonEmpty(ev.get("res"), ev.get("success"), ev.get("result")))
+	switch v {
+	case "success", "yes", "1":
+		return "success"
+	case "failed", "fail", "failure", "no", "0":
+		return "failed"
+	}
+	return ""
+}
+
+const auditUnset = "4294967295"
+
+// userOf resolves a uid: enriched name, then the host's /etc/passwd, then the number.
+func (ev *auditEvent) userOf(field string, users *userNames) string {
+	uid := ev.get(field)
+	if uid == "" || uid == auditUnset || uid == "-1" {
+		return ""
+	}
+	if n := strings.Trim(ev.get("@"+strings.ToUpper(field)), `"`); n != "" && n != "unset" && n != uid {
+		return n
+	}
+	if n := users.name(uid); n != "" {
+		return n
+	}
+	return uid
+}
+
+// auditEntry turns an event into an Entry: app is the type of its first record.
+func auditEntry(ev *auditEvent, users *userNames, hostName string, now time.Time) *Entry {
+	typ := "UNKNOWN"
+	if len(ev.Records) > 0 {
+		typ = ev.Records[0].Type
+	}
+	res := ev.result()
+	auid := ev.get("auid")
+	user := ev.userOf("auid", users)
+	if user == "" {
+		user = ev.userOf("uid", users)
+	}
+	sev := 6
+	if res == "failed" {
+		sev = 4
+	}
+	raw := make([]string, len(ev.Records))
+	for i, r := range ev.Records {
+		raw[i] = r.Raw
+	}
+	host := firstNonEmpty(ev.Node, hostName, "localhost")
+	e := &Entry{
+		Time:       ev.Time,
+		Received:   now,
+		Host:       host,
+		App:        typ,
+		ProcID:     ev.get("pid"),
+		Facility:   "authpriv",
+		Severity:   severityNames[sev],
+		SevNum:     sev,
+		Message:    auditSummary(ev, typ, user, users, res),
+		Proto:      "file",
+		SourceType: "audit",
+		Extra: map[string]string{
+			"audit_type":   typ,
+			"audit_serial": ev.Serial,
+			"auid":         auid,
+			"user":         user,
+			"exe":          ev.get("exe"),
+			"res":          res,
+			"key":          ev.get("key"),
+			"audit_raw":    strings.Join(raw, "\n"),
+		},
+		Wait: true,
+	}
+	if e.Time.IsZero() {
+		e.Time = now
+	}
+	return e
+}
+
+// auditSummary writes "who: what [key] → result".
+func auditSummary(ev *auditEvent, typ, user string, users *userNames, res string) string {
+	who := firstNonEmpty(user, "?")
+	// Syscalls: the login user and, when different, the user the process ran
+	// as (sudo, su). In user space records, uid is the daemon's (sshd: root).
+	if uid := ev.userOf("uid", users); ev.get("syscall") != "" && uid != "" && user != "" && uid != user {
+		who += " (as " + uid + ")"
+	}
+	var what string
+	exe := ev.get("exe")
+	switch {
+	case typ == "USER_CMD":
+		what = "sudo " + firstNonEmpty(ev.get("cmd"), "?")
+		if cwd := ev.get("cwd"); cwd != "" {
+			what += " (in " + cwd + ")"
+		}
+	case strings.HasPrefix(typ, "USER_") || strings.HasPrefix(typ, "CRED_") || typ == "LOGIN" ||
+		typ == "ADD_USER" || typ == "DEL_USER" || typ == "ADD_GROUP" || typ == "DEL_GROUP":
+		if acct := ev.get("acct"); acct != "" {
+			who = acct
+		}
+		what = firstNonEmpty(ev.get("op"), strings.ToLower(typ))
+		if exe != "" {
+			what += " via " + exe
+		}
+		if addr := firstNonEmpty(ev.get("hostname"), ev.get("addr")); addr != "" {
+			what += " from " + addr
+		}
+		if term := ev.get("terminal"); term != "" && term != "?" {
+			what += " on " + term
+		}
+	case ev.record("EXECVE") != nil:
+		what = "exec " + execLine(ev.record("EXECVE"))
+	case ev.get("syscall") != "":
+		what = "syscall " + firstNonEmpty(ev.get("@SYSCALL"), ev.get("syscall"))
+		if name := pathName(ev); name != "" {
+			what += " " + name
+		}
+		if prog := firstNonEmpty(exe, ev.get("comm")); prog != "" {
+			what += " by " + prog
+		}
+	default:
+		what = strings.ToLower(typ)
+		if op := ev.get("op"); op != "" {
+			what += " " + op
+		}
+		if exe != "" {
+			what += " by " + exe
+		}
+	}
+	if key := ev.get("key"); key != "" {
+		what += " [" + key + "]"
+	}
+	if res != "" {
+		what += " → " + res
+	}
+	return who + ": " + what
+}
+
+// execLine joins the arguments of an EXECVE record (a0, a1…).
+func execLine(r *auditRecord) string {
+	args := map[int]string{}
+	argc := 0
+	for _, f := range r.Fields {
+		if f.k == "argc" {
+			argc, _ = strconv.Atoi(f.v)
+		} else if len(f.k) > 1 && f.k[0] == 'a' && isDigits(f.k[1:]) {
+			n, _ := strconv.Atoi(f.k[1:])
+			if f.bare {
+				args[n] = decodeHex(f.k, f.v) // arguments with spaces or quotes
+			} else {
+				args[n] = f.v
+			}
+		}
+	}
+	if argc == 0 {
+		argc = len(args)
+	}
+	parts := make([]string, 0, argc)
+	for i := 0; i < argc; i++ {
+		if v, ok := args[i]; ok {
+			parts = append(parts, v)
+		}
+	}
+	s := strings.Join(parts, " ")
+	if len(s) > 300 {
+		s = s[:300] + "…"
+	}
+	return s
+}
+
+// pathName is the file a syscall worked on: the first PATH record that is not
+// its parent directory.
+func pathName(ev *auditEvent) string {
+	var first string
+	for _, r := range ev.Records {
+		if r.Type != "PATH" {
+			continue
+		}
+		var name, nametype string
+		for _, f := range r.Fields {
+			switch f.k {
+			case "name":
+				name = f.v
+			case "nametype":
+				nametype = f.v
+			}
+		}
+		if name == "" {
+			continue
+		}
+		if nametype != "PARENT" {
+			return name
+		}
+		if first == "" {
+			first = name
+		}
+	}
+	return first
+}
+
+/* ---------- User names ---------- */
+
+// userNames reads uid -> name from the host's /etc/passwd when it is mounted,
+// at most once a minute.
+type userNames struct {
+	path    string
+	names   map[string]string
+	loaded  time.Time
+	modTime time.Time
+}
+
+func (u *userNames) name(uid string) string {
+	if u == nil {
+		return ""
+	}
+	if time.Since(u.loaded) > time.Minute {
+		u.loaded = time.Now()
+		if fi, err := os.Stat(u.path); err == nil && !fi.ModTime().Equal(u.modTime) {
+			u.modTime = fi.ModTime()
+			u.names = readPasswd(u.path)
+		}
+	}
+	return u.names[uid]
+}
+
+func readPasswd(path string) map[string]string {
+	f, err := os.Open(path)
+	if err != nil {
+		return nil
+	}
+	defer f.Close()
+	names := map[string]string{}
+	sc := bufio.NewScanner(f)
+	for sc.Scan() {
+		p := strings.Split(sc.Text(), ":")
+		if len(p) >= 3 && p[0] != "" && !strings.HasPrefix(p[0], "#") {
+			names[p[2]] = p[0]
+		}
+	}
+	return names
+}

audit_test.go

@@ -0,0 +1,356 @@
+package main
+
+import (
+	"context"
+	"encoding/hex"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+)
+
+func hexOf(s string) string { return strings.ToUpper(hex.EncodeToString([]byte(s))) }
+
+// Lines of real audit.log events (trimmed).
+var (
+	auditExec = []string{
+		`type=SYSCALL msg=audit(1759572000.123:4242): arch=c000003e syscall=59 success=yes exit=0 a0=4142 a1=55d2 a2=55d3 a3=0 items=2 ppid=1000 pid=1234 auid=1000 uid=0 gid=0 euid=0 tty=pts0 ses=3 comm="apt" exe="/usr/bin/apt" subj=unconfined key="exec"`,
+		`type=EXECVE msg=audit(1759572000.123:4242): argc=3 a0="apt" a1="install" a2=` + hexOf(`my "pkg"`),
+		`type=CWD msg=audit(1759572000.123:4242): cwd="/root"`,
+		`type=PATH msg=audit(1759572000.123:4242): item=0 name="/usr/bin/apt" inode=1 dev=fd:00 mode=0100755 ouid=0 ogid=0 nametype=NORMAL`,
+		`type=PROCTITLE msg=audit(1759572000.123:4242): proctitle=` + hexOf("apt\x00install\x00my \"pkg\""),
+	}
+	auditSudo = `type=USER_CMD msg=audit(1759572001.500:4250): pid=2000 uid=1000 auid=1000 ses=3 subj=unconfined msg='cwd="/home/cedric" cmd=` + hexOf("systemctl restart nginx") + ` exe="/usr/bin/sudo" terminal=pts/0 res=success'`
+	auditSSH  = `type=USER_AUTH msg=audit(1759572002.000:4260): pid=3000 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='op=PAM:authentication grantors=? acct="bob" exe="/usr/sbin/sshd" hostname=203.0.113.9 addr=203.0.113.9 terminal=ssh res=failed'`
+	auditOpen = []string{
+		"node=web1 type=SYSCALL msg=audit(1759572003.250:4270): arch=c000003e syscall=257 success=no exit=-13 items=1 pid=4000 auid=1001 uid=1001 comm=\"cat\" exe=\"/usr/bin/cat\" key=\"shadow\"\x1dARCH=x86_64 SYSCALL=openat AUID=\"alice\" UID=\"alice\"",
+		`node=web1 type=PATH msg=audit(1759572003.250:4270): item=0 name="/etc/shadow" inode=7 nametype=NORMAL`,
+	}
+)
+
+func TestParseAuditLine(t *testing.T) {
+	rec, id, node, ok := parseAuditLine(auditOpen[0])
+	if !ok || rec.Type != "SYSCALL" || id != "1759572003.250:4270" || node != "web1" {
+		t.Fatalf("got %+v %q %q %v", rec, id, node, ok)
+	}
+	ev := &auditEvent{Records: []auditRecord{rec}}
+	for k, want := range map[string]string{"syscall": "257", "exe": "/usr/bin/cat", "key": "shadow", "@SYSCALL": "openat", "@AUID": "alice"} {
+		if got := ev.get(k); got != want {
+			t.Errorf("%s = %q, want %q", k, got, want)
+		}
+	}
+	for _, bad := range []string{"", "garbage", "type=X no header", "msg=audit(1.2:3): no type"} {
+		if _, _, _, ok := parseAuditLine(bad); ok {
+			t.Errorf("%q parsed", bad)
+		}
+	}
+}
+
+func TestParseAuditKV(t *testing.T) {
+	tests := []struct {
+		in   string
+		want map[string]string
+	}{
+		{`a=1 b="two words" c=`, map[string]string{"a": "1", "b": "two words", "c": ""}},
+		{`pid=1 msg='op=login acct="bob" res=success'`, map[string]string{"pid": "1", "op": "login", "acct": "bob", "res": "success"}},
+		{`comm=` + hexOf("my prog") + ` exe="/bin/x"`, map[string]string{"comm": "my prog", "exe": "/bin/x"}},
+		{`key=(null) hostname=? terminal=?`, map[string]string{"key": "", "hostname": "", "terminal": ""}},
+		{`a0=4142 name=ABC`, map[string]string{"a0": "4142", "name": "ABC"}}, // not hexadecimal fields, or odd length
+		{`avc:  denied  { read } for  pid=12 comm="x"`, map[string]string{"pid": "12", "comm": "x"}},
+	}
+	for _, tt := range tests {
+		got := map[string]string{}
+		for _, f := range parseAuditKV(tt.in) {
+			got[f.k] = f.v
+		}
+		for k, v := range tt.want {
+			if got[k] != v {
+				t.Errorf("%q: %s = %q, want %q (all: %v)", tt.in, k, got[k], v, got)
+			}
+		}
+	}
+}
+
+func TestAuditAssembler(t *testing.T) {
+	now := time.Unix(1759572010, 0)
+	a := newAuditAssembler()
+	// Two interleaved events, then an EOE that closes the first one.
+	lines := []string{auditExec[0], auditSudo, auditExec[1], auditExec[2]}
+	off := int64(0)
+	for _, l := range lines {
+		if out := a.add(l, off, now); out != nil {
+			t.Fatalf("closed early: %v", out)
+		}
+		off += int64(len(l)) + 1
+	}
+	if len(a.open) != 2 {
+		t.Fatalf("%d open events", len(a.open))
+	}
+	if o, ok := a.minOff(); !ok || o != 0 {
+		t.Fatalf("minOff %d %v", o, ok)
+	}
+	out := a.add(`type=EOE msg=audit(1759572000.123:4242): `, off, now)
+	if len(out) != 1 || out[0].Serial != "4242" || len(out[0].Records) != 3 {
+		t.Fatalf("EOE: %+v", out)
+	}
+	if o, _ := a.minOff(); o != int64(len(lines[0])+1) {
+		t.Fatalf("minOff after EOE: %d", o)
+	}
+	// An EOE of an unknown event is ignored.
+	if out := a.add(`type=EOE msg=audit(1.0:1): `, 0, now); out != nil {
+		t.Fatal(out)
+	}
+	// The sudo event is closed by time.
+	if out := a.flush(now.Add(time.Second), 2*time.Second); len(out) != 0 {
+		t.Fatalf("closed too early: %v", out)
+	}
+	out = a.flush(now.Add(3*time.Second), 2*time.Second)
+	if len(out) != 1 || out[0].Records[0].Type != "USER_CMD" {
+		t.Fatalf("flush: %+v", out)
+	}
+	// Events the log has left behind are closed without waiting.
+	a.add(auditSSH, 0, now)
+	a.add(auditOpen[0], 0, now)
+	out = a.flush(now, 1*time.Second)
+	if len(out) != 1 || out[0].Serial != "4260" || len(a.open) != 1 {
+		t.Fatalf("newest: %+v, open %d", out, len(a.open))
+	}
+	if out = a.flushAll(); len(out) != 1 || out[0].Serial != "4270" || len(a.open) != 0 {
+		t.Fatalf("flushAll: %+v", out)
+	}
+}
+
+func auditEv(t *testing.T, lines ...string) *auditEvent {
+	t.Helper()
+	a := newAuditAssembler()
+	for _, l := range lines {
+		a.add(l, 0, time.Time{})
+	}
+	out := a.flushAll()
+	if len(out) != 1 {
+		t.Fatalf("%d events", len(out))
+	}
+	return out[0]
+}
+
+func TestAuditEntry(t *testing.T) {
+	dir := t.TempDir()
+	passwd := filepath.Join(dir, "passwd")
+	if err := os.WriteFile(passwd, []byte("root:x:0:0::/root:/bin/sh\ncedric:x:1000:1000::/home/cedric:/bin/bash\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	now := time.Unix(1759572100, 0)
+	tests := []struct {
+		name   string
+		lines  []string
+		host   string
+		app    string
+		sev    string
+		msg    string
+		fields map[string]string
+	}{
+		{"exec", auditExec, "srv", "SYSCALL", "info",
+			`cedric (as root): exec apt install my "pkg" [exec] → success`,
+			map[string]string{"audit_type": "SYSCALL", "audit_serial": "4242", "auid": "1000", "user": "cedric", "exe": "/usr/bin/apt", "res": "success", "key": "exec"}},
+		{"sudo", []string{auditSudo}, "srv", "USER_CMD", "info",
+			"cedric: sudo systemctl restart nginx (in /home/cedric) → success",
+			map[string]string{"user": "cedric", "exe": "/usr/bin/sudo", "res": "success", "key": ""}},
+		{"ssh failed", []string{auditSSH}, "srv", "USER_AUTH", "warning",
+			"bob: PAM:authentication via /usr/sbin/sshd from 203.0.113.9 on ssh → failed",
+			map[string]string{"auid": "4294967295", "user": "root", "res": "failed"}},
+		{"enriched, node", auditOpen, "web1", "SYSCALL", "warning",
+			"alice: syscall openat /etc/shadow by /usr/bin/cat [shadow] → failed",
+			map[string]string{"user": "alice", "auid": "1001", "res": "failed", "key": "shadow"}},
+		{"unknown uid", []string{`type=USER_LOGIN msg=audit(1759572004.000:4280): pid=1 uid=0 auid=4242 ses=9 msg='op=login id=4242 exe="/usr/sbin/sshd" hostname=? addr=10.0.0.5 terminal=/dev/pts/1 res=success'`},
+			"srv", "USER_LOGIN", "info", "4242: login via /usr/sbin/sshd from 10.0.0.5 on /dev/pts/1 → success",
+			map[string]string{"user": "4242"}},
+		{"no result", []string{`type=CONFIG_CHANGE msg=audit(1759572005.000:4290): auid=1000 ses=3 op=add_rule key="watch" list=4`},
+			"srv", "CONFIG_CHANGE", "info", "cedric: config_change add_rule [watch]",
+			map[string]string{"res": ""}},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			ev := auditEv(t, tt.lines...)
+			e := auditEntry(ev, &userNames{path: passwd}, "srv", now)
+			if e.Host != tt.host || e.App != tt.app || e.Severity != tt.sev || e.SourceType != "audit" || !e.Wait {
+				t.Errorf("entry %+v", e)
+			}
+			if e.Message != tt.msg {
+				t.Errorf("message\n got %q\nwant %q", e.Message, tt.msg)
+			}
+			for k, v := range tt.fields {
+				if e.Extra[k] != v {
+					t.Errorf("%s = %q, want %q", k, e.Extra[k], v)
+				}
+			}
+			if e.Extra["audit_raw"] != strings.Join(tt.lines, "\n") {
+				t.Errorf("raw %q", e.Extra["audit_raw"])
+			}
+			if !e.Time.Equal(ev.Time) || e.Time.IsZero() {
+				t.Errorf("time %v", e.Time)
+			}
+		})
+	}
+}
+
+type entrySink struct {
+	mu sync.Mutex
+	es []*Entry
+}
+
+func (s *entrySink) add(e *Entry) {
+	s.mu.Lock()
+	s.es = append(s.es, e)
+	s.mu.Unlock()
+}
+
+func (s *entrySink) serials() []string {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	var out []string
+	for _, e := range s.es {
+		out = append(out, e.Extra["audit_serial"])
+	}
+	return out
+}
+
+func appendLines(t *testing.T, path string, lines ...string) {
+	t.Helper()
+	f, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer f.Close()
+	for _, l := range lines {
+		if _, err := f.WriteString(l + "\n"); err != nil {
+			t.Fatal(err)
+		}
+	}
+}
+
+func TestAuditFollow(t *testing.T) {
+	dir := t.TempDir()
+	logPath := filepath.Join(dir, "audit.log")
+	data := filepath.Join(dir, "data")
+	if err := os.Mkdir(data, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	sink := &entrySink{}
+	newAuditT := func() *Audit {
+		a := NewAudit(logPath, dir, "srv", data, 100*365*24*time.Hour, sink.add)
+		a.cfg.Enabled = true
+		return a
+	}
+	a := newAuditT()
+	now := time.Unix(1759572100, 0)
+
+	// Missing file: reported, then read once it exists.
+	a.poll(now)
+	if st := a.Status(); st["code"] != "not_found" || st["open"] != false {
+		t.Fatalf("status %v", st)
+	}
+	appendLines(t, logPath, auditExec[:3]...)
+	appendLines(t, logPath, auditSudo)
+	a.poll(now)
+	if got := sink.serials(); len(got) != 0 { // events may still get records
+		t.Fatalf("after first poll: %v", got)
+	}
+	// A partial line is not read.
+	f, _ := os.OpenFile(logPath, os.O_APPEND|os.O_WRONLY, 0)
+	f.WriteString(auditSSH[:20])
+	f.Close()
+	a.poll(now.Add(5 * time.Second))
+	if got := sink.serials(); len(got) != 2 || got[0] != "4242" || got[1] != "4250" {
+		t.Fatalf("after second poll: %v", got)
+	}
+	if a.Status()["open"] != true || a.Status()["read"] != uint64(2) {
+		t.Fatalf("status %v", a.Status())
+	}
+	a.saveState()
+
+	// Restart: nothing is sent twice, the partial line is read once complete.
+	a = newAuditT()
+	f, _ = os.OpenFile(logPath, os.O_APPEND|os.O_WRONLY, 0)
+	f.WriteString(auditSSH[20:] + "\n")
+	f.Close()
+	a.poll(now.Add(10 * time.Second))
+	a.poll(now.Add(20 * time.Second))
+	if got := sink.serials(); len(got) != 3 || got[2] != "4260" {
+		t.Fatalf("after restart: %v", got)
+	}
+
+	// Rotation: lines added to the old file before the switch are not lost.
+	appendLines(t, logPath, auditOpen[0])
+	a.poll(now.Add(21 * time.Second)) // read, still open
+	appendLines(t, logPath, auditOpen[1])
+	if err := os.Rename(logPath, logPath+".1"); err != nil {
+		t.Fatal(err)
+	}
+	appendLines(t, logPath, strings.Replace(auditSudo, "4250", "4300", 1))
+	a.poll(now.Add(22 * time.Second))
+	a.poll(now.Add(30 * time.Second))
+	got := sink.serials()
+	if len(got) != 5 || got[3] != "4270" || got[4] != "4300" {
+		t.Fatalf("after rotation: %v", got)
+	}
+	sink.mu.Lock()
+	if n := len(sink.es[3].Extra["audit_raw"]); n == 0 || !strings.Contains(sink.es[3].Extra["audit_raw"], "/etc/shadow") {
+		t.Errorf("rotated event missing its PATH record: %q", sink.es[3].Extra["audit_raw"])
+	}
+	sink.mu.Unlock()
+}
+
+func TestAuditBackfillAndConfigure(t *testing.T) {
+	dir := t.TempDir()
+	logPath := filepath.Join(dir, "audit.log")
+	appendLines(t, logPath, auditExec...)
+	appendLines(t, logPath, auditSudo)
+	sink := &entrySink{}
+	// Only events newer than the backfill window are sent on the first read.
+	a := NewAudit(logPath, dir, "", dir, time.Hour, sink.add)
+	a.cfg.Enabled = true
+	a.poll(time.Unix(1759572001+7200, 0))
+	a.emit(a.asm.flushAll())
+	if got := sink.serials(); len(got) != 0 {
+		t.Fatalf("too old: %v", got)
+	}
+	a = NewAudit(logPath, dir, "", filepath.Join(dir, "other"), time.Hour, sink.add)
+	if err := os.Mkdir(filepath.Join(dir, "other"), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	a.cfg.Enabled = true
+	a.poll(time.Unix(1759572001+60, 0))
+	a.emit(a.asm.flushAll())
+	if got := sink.serials(); len(got) != 2 {
+		t.Fatalf("backfill: %v", got)
+	}
+	if e := sink.es[0]; e.Host != "localhost" {
+		t.Errorf("host %q", e.Host)
+	}
+
+	// Run picks the configuration up; turning it off resets the position.
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan struct{})
+	go func() { a.Run(ctx); close(done) }()
+	if err := a.Configure(auditConfig{Enabled: false}); err != nil {
+		t.Fatal(err)
+	}
+	deadline := time.Now().Add(2 * time.Second)
+	for time.Now().Before(deadline) {
+		if b, err := os.ReadFile(filepath.Join(dir, "other", "audit-state.json")); err == nil && strings.Contains(string(b), `"off": 0`) {
+			break
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+	cancel()
+	<-done
+	if st := a.Status(); st["enabled"] != false || st["defaultPath"] != logPath {
+		t.Fatalf("status %v", st)
+	}
+	b, _ := os.ReadFile(filepath.Join(dir, "other", "audit-state.json"))
+	if !strings.Contains(string(b), `"off": 0`) {
+		t.Fatalf("state %s", b)
+	}
+}

auth.go

@@ -0,0 +1,658 @@
+package main
+
+import (
+	"bytes"
+	"crypto"
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/hmac"
+	"crypto/rand"
+	"crypto/rsa"
+	"crypto/sha256"
+	_ "crypto/sha512" // SHA-384/512 for RS384, ES384, RS512…
+	"crypto/subtle"
+	"encoding/base64"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"log"
+	"math/big"
+	"net/http"
+	"net/url"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"time"
+)
+
+// Web UI authentication. AUTH_MODE=local (default) shows a login page when AUTH_USER /
+// AUTH_PASS are set (auth_local.go); AUTH_MODE=oidc delegates the login to an
+// OpenID Connect provider (Keycloak, Authentik, Authelia…) with the authorization code
+// flow and PKCE. Only the standard library is used.
+
+const (
+	sessionCookie = "logstream_session"
+	loginCookie   = "logstream_login_" // + state: one cookie per login in progress
+	loginTTL      = 10 * time.Minute
+	clockSkew     = time.Minute
+)
+
+type authConfig struct {
+	mode         string
+	user, pass   string // local mode
+	viewerUser   string // local mode: optional read-only account
+	viewerPass   string
+	adminGroup   string // oidc: only members of this group are admins (empty: everyone)
+	groupsClaim  string // oidc: ID token claim listing the groups
+	issuer       string
+	clientID     string
+	clientSecret string
+	redirectURL  string
+	scopes       string
+	sessionTTL   time.Duration
+	dataDir      string
+	loginLogo    string // local mode: PNG shown on the login page
+
+}
+
+// newAuth returns the middleware that protects the UI and the API (except /healthz).
+func newAuth(c authConfig, next http.Handler) (http.Handler, error) {
+	if c.sessionTTL <= 0 {
+		c.sessionTTL = 12 * time.Hour
+	}
+	switch strings.ToLower(c.mode) {
+	case "", "local":
+		if c.user == "" {
+			return next, nil
+		}
+		l := newLocal(c)
+		l.next = next
+		return l, nil
+	case "oidc":
+		o, err := newOIDC(c)
+		if err != nil {
+			return nil, err
+		}
+		o.next = next
+		log.Printf("oidc authentication enabled (issuer %s)", c.issuer)
+		return o, nil
+	}
+	return nil, fmt.Errorf("AUTH_MODE=%q: expected local or oidc", c.mode)
+}
+
+type oidcMeta struct {
+	Issuer           string   `json:"issuer"`
+	AuthEndpoint     string   `json:"authorization_endpoint"`
+	TokenEndpoint    string   `json:"token_endpoint"`
+	JWKSURI          string   `json:"jwks_uri"`
+	EndSession       string   `json:"end_session_endpoint"`
+	TokenAuthMethods []string `json:"token_endpoint_auth_methods_supported"`
+}
+
+type OIDC struct {
+	cfg      authConfig
+	callback string // path of OIDC_REDIRECT_URL
+	secure   bool   // cookies only sent over HTTPS
+	key      []byte // signs the session and login cookies
+	client   *http.Client
+	next     http.Handler
+
+	mu     sync.Mutex
+	meta   *oidcMeta
+	keys   map[string]crypto.PublicKey
+	keysAt time.Time
+}
+
+func newOIDC(c authConfig) (*OIDC, error) {
+	var missing []string
+	for _, v := range [][2]string{
+		{"OIDC_ISSUER", c.issuer}, {"OIDC_CLIENT_ID", c.clientID},
+		{"OIDC_CLIENT_SECRET", c.clientSecret}, {"OIDC_REDIRECT_URL", c.redirectURL},
+	} {
+		if v[1] == "" {
+			missing = append(missing, v[0])
+		}
+	}
+	if len(missing) > 0 {
+		return nil, fmt.Errorf("AUTH_MODE=oidc: missing %s", strings.Join(missing, ", "))
+	}
+	ru, err := url.Parse(c.redirectURL)
+	if err != nil || ru.Host == "" || ru.Path == "" || ru.Path == "/" {
+		return nil, fmt.Errorf("OIDC_REDIRECT_URL=%q: expected a full URL such as https://logs.example.org/auth/callback", c.redirectURL)
+	}
+	if c.scopes == "" {
+		c.scopes = "openid profile email"
+	}
+	if !strings.Contains(" "+c.scopes+" ", " openid ") {
+		c.scopes = "openid " + c.scopes
+	}
+	if c.groupsClaim == "" {
+		c.groupsClaim = "groups"
+	}
+	return &OIDC{
+		cfg:      c,
+		callback: ru.Path,
+		secure:   ru.Scheme == "https",
+		key:      sessionKey(c.dataDir),
+		client:   &http.Client{Timeout: 10 * time.Second},
+	}, nil
+}
+
+// checkProvider reads the provider configuration at startup so a mistake shows in the logs.
+func (o *OIDC) checkProvider() {
+	if _, err := o.discover(); err != nil {
+		log.Printf("oidc: %v", err)
+	}
+}
+
+// sessionKey is kept in DATA_DIR so sessions survive a restart.
+func sessionKey(dir string) []byte {
+	path := filepath.Join(dir, "session.key")
+	if k, err := os.ReadFile(path); err == nil && len(k) >= 32 {
+		return k
+	}
+	k := make([]byte, 32)
+	if _, err := rand.Read(k); err != nil {
+		log.Fatalf("session key: %v", err)
+	}
+	if err := os.WriteFile(path, k, 0o600); err != nil {
+		log.Printf("auth: cannot save %s (%v): sessions end when logstream restarts", path, err)
+	}
+	return k
+}
+
+type session struct {
+	User   string `json:"u"`
+	Exp    int64  `json:"e"`
+	Viewer bool   `json:"v,omitempty"` // read-only user
+}
+
+// writeAuthRequired answers API calls without a session; the UI turns it into a reload
+// (and so into a new login).
+func writeAuthRequired(w http.ResponseWriter) {
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(http.StatusUnauthorized)
+	_, _ = w.Write([]byte(`{"error":"authentication required","code":"auth"}` + "\n"))
+}
+
+type loginState struct {
+	Nonce    string `json:"n"`
+	Verifier string `json:"v"`
+	Return   string `json:"r"`
+	Exp      int64  `json:"e"`
+}
+
+func (o *OIDC) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	switch r.URL.Path {
+	case "/healthz":
+		o.next.ServeHTTP(w, r)
+		return
+	case o.callback:
+		o.handleCallback(w, r)
+		return
+	case "/auth/logout":
+		o.handleLogout(w, r)
+		return
+	}
+	var s session
+	if c, err := r.Cookie(sessionCookie); err == nil && verifyCookie(o.key, c.Value, &s) && time.Now().Unix() < s.Exp {
+		if r.URL.Path == "/auth/me" {
+			writeJSON(w, http.StatusOK, map[string]string{"mode": "oidc", "user": s.User, "role": roleName(s.Viewer)})
+			return
+		}
+		if s.Viewer {
+			r = asViewer(r)
+		}
+		o.next.ServeHTTP(w, r)
+		return
+	}
+	// Not logged in: pages go to the provider, API calls get a 401 that the UI turns
+	// into a reload (and so into a new login).
+	if r.Method == http.MethodGet && !strings.HasPrefix(r.URL.Path, "/api/") && r.URL.Path != "/auth/me" {
+		o.startLogin(w, r)
+		return
+	}
+	writeAuthRequired(w)
+}
+
+func (o *OIDC) startLogin(w http.ResponseWriter, r *http.Request) {
+	meta, err := o.discover()
+	if err != nil {
+		log.Printf("oidc: %v", err)
+		http.Error(w, "identity provider unreachable, try again later", http.StatusBadGateway)
+		return
+	}
+	state, nonce, verifier := randomString(), randomString(), randomString()+randomString()
+	ret := r.URL.RequestURI()
+	if !strings.HasPrefix(ret, "/") || strings.HasPrefix(ret, "//") {
+		ret = "/"
+	}
+	http.SetCookie(w, &http.Cookie{
+		Name:     loginCookie + state,
+		Value:    signCookie(o.key, loginState{Nonce: nonce, Verifier: verifier, Return: ret, Exp: time.Now().Add(loginTTL).Unix()}),
+		Path:     "/",
+		MaxAge:   int(loginTTL.Seconds()),
+		HttpOnly: true,
+		Secure:   o.secure,
+		SameSite: http.SameSiteLaxMode, // sent back on the redirect from the provider
+	})
+	challenge := sha256.Sum256([]byte(verifier))
+	q := url.Values{
+		"response_type":         {"code"},
+		"client_id":             {o.cfg.clientID},
+		"redirect_uri":          {o.cfg.redirectURL},
+		"scope":                 {o.cfg.scopes},
+		"state":                 {state},
+		"nonce":                 {nonce},
+		"code_challenge":        {base64.RawURLEncoding.EncodeToString(challenge[:])},
+		"code_challenge_method": {"S256"},
+	}
+	http.Redirect(w, r, addQuery(meta.AuthEndpoint, q), http.StatusFound)
+}
+
+func (o *OIDC) handleCallback(w http.ResponseWriter, r *http.Request) {
+	q := r.URL.Query()
+	if e := q.Get("error"); e != "" {
+		log.Printf("oidc: login refused by the provider: %s %s", e, q.Get("error_description"))
+		http.Error(w, "login refused by the identity provider: "+e, http.StatusForbidden)
+		return
+	}
+	state := q.Get("state")
+	var ls loginState
+	c, err := r.Cookie(loginCookie + state)
+	if state == "" || err != nil || !verifyCookie(o.key, c.Value, &ls) || time.Now().Unix() > ls.Exp {
+		http.Error(w, "login expired or started in another browser: open LogStream again", http.StatusBadRequest)
+		return
+	}
+	http.SetCookie(w, &http.Cookie{Name: loginCookie + state, Path: "/", MaxAge: -1, HttpOnly: true, Secure: o.secure})
+
+	user, viewer, err := o.exchange(r, q.Get("code"), ls)
+	if err != nil {
+		log.Printf("oidc: login failed: %v", err)
+		http.Error(w, "login failed, see the LogStream logs", http.StatusForbidden)
+		return
+	}
+	log.Printf("oidc: %s logged in (%s)", user, roleName(viewer))
+	http.SetCookie(w, &http.Cookie{
+		Name:     sessionCookie,
+		Value:    signCookie(o.key, session{User: user, Exp: time.Now().Add(o.cfg.sessionTTL).Unix(), Viewer: viewer}),
+		Path:     "/",
+		MaxAge:   int(o.cfg.sessionTTL.Seconds()),
+		HttpOnly: true,
+		Secure:   o.secure,
+		SameSite: http.SameSiteLaxMode,
+	})
+	http.Redirect(w, r, ls.Return, http.StatusFound)
+}
+
+// The session ends here; the provider's own session ends on its logout page if it has one.
+func (o *OIDC) handleLogout(w http.ResponseWriter, r *http.Request) {
+	http.SetCookie(w, &http.Cookie{Name: sessionCookie, Path: "/", MaxAge: -1, HttpOnly: true, Secure: o.secure})
+	if meta, err := o.discover(); err == nil && meta.EndSession != "" {
+		http.Redirect(w, r, addQuery(meta.EndSession, url.Values{"client_id": {o.cfg.clientID}}), http.StatusFound)
+		return
+	}
+	http.Redirect(w, r, "/", http.StatusFound)
+}
+
+// exchange trades the code for tokens and returns the user name from the verified ID
+// token, and whether the user is read-only (not in OIDC_ADMIN_GROUP).
+func (o *OIDC) exchange(r *http.Request, code string, ls loginState) (string, bool, error) {
+	if code == "" {
+		return "", false, errors.New("no code in the callback")
+	}
+	meta, err := o.discover()
+	if err != nil {
+		return "", false, err
+	}
+	form := url.Values{
+		"grant_type":    {"authorization_code"},
+		"code":          {code},
+		"redirect_uri":  {o.cfg.redirectURL},
+		"code_verifier": {ls.Verifier},
+	}
+	// client_secret_basic is the default; some providers only accept client_secret_post.
+	post := len(meta.TokenAuthMethods) > 0 && !contains(meta.TokenAuthMethods, "client_secret_basic") && contains(meta.TokenAuthMethods, "client_secret_post")
+	if post {
+		form.Set("client_id", o.cfg.clientID)
+		form.Set("client_secret", o.cfg.clientSecret)
+	}
+	req, err := http.NewRequestWithContext(r.Context(), http.MethodPost, meta.TokenEndpoint, strings.NewReader(form.Encode()))
+	if err != nil {
+		return "", false, err
+	}
+	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	req.Header.Set("Accept", "application/json")
+	if !post {
+		req.SetBasicAuth(url.QueryEscape(o.cfg.clientID), url.QueryEscape(o.cfg.clientSecret))
+	}
+	res, err := o.client.Do(req)
+	if err != nil {
+		return "", false, fmt.Errorf("token endpoint: %w", err)
+	}
+	defer res.Body.Close()
+	body, _ := io.ReadAll(io.LimitReader(res.Body, 1<<20))
+	if res.StatusCode != http.StatusOK {
+		return "", false, fmt.Errorf("token endpoint: %s: %s", res.Status, bytes.TrimSpace(body))
+	}
+	var tok struct {
+		IDToken string `json:"id_token"`
+	}
+	if err := json.Unmarshal(body, &tok); err != nil || tok.IDToken == "" {
+		return "", false, errors.New("token endpoint: no id_token in the response")
+	}
+	claims, err := o.verifyIDToken(tok.IDToken, ls.Nonce)
+	if err != nil {
+		return "", false, err
+	}
+	viewer := o.cfg.adminGroup != "" && !hasGroup(claims[o.cfg.groupsClaim], o.cfg.adminGroup)
+	for _, k := range []string{"preferred_username", "email", "name", "sub"} {
+		if v, _ := claims[k].(string); v != "" {
+			return v, viewer, nil
+		}
+	}
+	return "", false, errors.New("id_token: no sub")
+}
+
+// hasGroup tells whether the groups claim (a list, or a single string) holds
+// group; a leading "/" (Keycloak group paths) is ignored.
+func hasGroup(claim any, group string) bool {
+	group = strings.TrimPrefix(group, "/")
+	var groups []string
+	switch v := claim.(type) {
+	case string:
+		groups = strings.Fields(strings.ReplaceAll(v, ",", " "))
+	case []any:
+		for _, g := range v {
+			if s, ok := g.(string); ok {
+				groups = append(groups, s)
+			}
+		}
+	}
+	for _, g := range groups {
+		if strings.TrimPrefix(g, "/") == group {
+			return true
+		}
+	}
+	return false
+}
+
+// verifyIDToken checks the signature (keys from jwks_uri) and the claims of an ID token.
+func (o *OIDC) verifyIDToken(raw, nonce string) (map[string]any, error) {
+	parts := strings.Split(raw, ".")
+	if len(parts) != 3 {
+		return nil, errors.New("id_token: not a JWT")
+	}
+	var hdr struct {
+		Alg string `json:"alg"`
+		Kid string `json:"kid"`
+	}
+	if err := decodeSegment(parts[0], &hdr); err != nil {
+		return nil, fmt.Errorf("id_token header: %w", err)
+	}
+	sig, err := base64.RawURLEncoding.DecodeString(parts[2])
+	if err != nil {
+		return nil, errors.New("id_token: bad signature encoding")
+	}
+	key, err := o.keyFor(hdr.Kid)
+	if err != nil {
+		return nil, err
+	}
+	if err := verifySignature(hdr.Alg, key, []byte(parts[0]+"."+parts[1]), sig); err != nil {
+		return nil, fmt.Errorf("id_token: %w", err)
+	}
+	var claims map[string]any
+	if err := decodeSegment(parts[1], &claims); err != nil {
+		return nil, fmt.Errorf("id_token claims: %w", err)
+	}
+	if iss, _ := claims["iss"].(string); iss != o.cfg.issuer {
+		return nil, fmt.Errorf("id_token: issuer %q, expected %q", iss, o.cfg.issuer)
+	}
+	var aud []string
+	switch v := claims["aud"].(type) {
+	case string:
+		aud = []string{v}
+	case []any:
+		for _, a := range v {
+			if s, ok := a.(string); ok {
+				aud = append(aud, s)
+			}
+		}
+	}
+	if !contains(aud, o.cfg.clientID) {
+		return nil, fmt.Errorf("id_token: audience %v does not include %q", aud, o.cfg.clientID)
+	}
+	if azp, ok := claims["azp"].(string); ok && len(aud) > 1 && azp != o.cfg.clientID {
+		return nil, fmt.Errorf("id_token: azp %q", azp)
+	}
+	now := time.Now()
+	exp, _ := claims["exp"].(float64)
+	if exp == 0 || now.After(time.Unix(int64(exp), 0).Add(clockSkew)) {
+		return nil, errors.New("id_token: expired (check the clocks)")
+	}
+	if iat, ok := claims["iat"].(float64); ok && time.Unix(int64(iat), 0).After(now.Add(clockSkew)) {
+		return nil, errors.New("id_token: issued in the future (check the clocks)")
+	}
+	if n, _ := claims["nonce"].(string); subtle.ConstantTimeCompare([]byte(n), []byte(nonce)) != 1 {
+		return nil, errors.New("id_token: wrong nonce")
+	}
+	return claims, nil
+}
+
+func verifySignature(alg string, key crypto.PublicKey, signed, sig []byte) error {
+	if len(alg) != 5 {
+		return fmt.Errorf("unsupported algorithm %q", alg)
+	}
+	var h crypto.Hash
+	switch alg[2:] {
+	case "256":
+		h = crypto.SHA256
+	case "384":
+		h = crypto.SHA384
+	case "512":
+		h = crypto.SHA512
+	}
+	if h == 0 {
+		return fmt.Errorf("unsupported algorithm %q", alg)
+	}
+	hh := h.New()
+	hh.Write(signed)
+	digest := hh.Sum(nil)
+	switch k := key.(type) {
+	case *rsa.PublicKey:
+		switch alg[:2] {
+		case "RS":
+			return rsa.VerifyPKCS1v15(k, h, digest, sig)
+		case "PS":
+			return rsa.VerifyPSS(k, h, digest, sig, &rsa.PSSOptions{SaltLength: rsa.PSSSaltLengthEqualsHash})
+		}
+	case *ecdsa.PublicKey:
+		size := (k.Curve.Params().BitSize + 7) / 8
+		if alg[:2] != "ES" || len(sig) != 2*size {
+			break
+		}
+		r, s := new(big.Int).SetBytes(sig[:size]), new(big.Int).SetBytes(sig[size:])
+		if ecdsa.Verify(k, digest, r, s) {
+			return nil
+		}
+		return errors.New("bad signature")
+	}
+	return fmt.Errorf("algorithm %q does not match the key", alg)
+}
+
+// discover reads the provider configuration once (and again after a failure).
+func (o *OIDC) discover() (*oidcMeta, error) {
+	o.mu.Lock()
+	defer o.mu.Unlock()
+	if o.meta != nil {
+		return o.meta, nil
+	}
+	u := strings.TrimSuffix(o.cfg.issuer, "/") + "/.well-known/openid-configuration"
+	var m oidcMeta
+	if err := o.getJSON(u, &m); err != nil {
+		return nil, fmt.Errorf("discovery: %w", err)
+	}
+	if m.Issuer != o.cfg.issuer {
+		return nil, fmt.Errorf("discovery: the provider says its issuer is %q, set OIDC_ISSUER to that exact value", m.Issuer)
+	}
+	if m.AuthEndpoint == "" || m.TokenEndpoint == "" || m.JWKSURI == "" {
+		return nil, errors.New("discovery: incomplete provider configuration")
+	}
+	o.meta = &m
+	return o.meta, nil
+}
+
+// keyFor returns the signing key kid; the key set is reloaded when the provider rotates its keys.
+func (o *OIDC) keyFor(kid string) (crypto.PublicKey, error) {
+	meta, err := o.discover()
+	if err != nil {
+		return nil, err
+	}
+	o.mu.Lock()
+	defer o.mu.Unlock()
+	pick := func() crypto.PublicKey {
+		if k, ok := o.keys[kid]; ok {
+			return k
+		}
+		if kid == "" && len(o.keys) == 1 {
+			for _, k := range o.keys {
+				return k
+			}
+		}
+		return nil
+	}
+	if k := pick(); k != nil {
+		return k, nil
+	}
+	if time.Since(o.keysAt) < 10*time.Second {
+		return nil, fmt.Errorf("id_token: unknown key %q", kid)
+	}
+	var set struct {
+		Keys []struct {
+			Kty string `json:"kty"`
+			Kid string `json:"kid"`
+			Use string `json:"use"`
+			N   string `json:"n"`
+			E   string `json:"e"`
+			Crv string `json:"crv"`
+			X   string `json:"x"`
+			Y   string `json:"y"`
+		} `json:"keys"`
+	}
+	if err := o.getJSON(meta.JWKSURI, &set); err != nil {
+		return nil, fmt.Errorf("jwks: %w", err)
+	}
+	keys := map[string]crypto.PublicKey{}
+	for _, k := range set.Keys {
+		if k.Use != "" && k.Use != "sig" {
+			continue
+		}
+		switch k.Kty {
+		case "RSA":
+			n, e := decodeBig(k.N), decodeBig(k.E)
+			if n != nil && e != nil && e.IsInt64() {
+				keys[k.Kid] = &rsa.PublicKey{N: n, E: int(e.Int64())}
+			}
+		case "EC":
+			var c elliptic.Curve
+			switch k.Crv {
+			case "P-256":
+				c = elliptic.P256()
+			case "P-384":
+				c = elliptic.P384()
+			case "P-521":
+				c = elliptic.P521()
+			}
+			x, y := decodeBig(k.X), decodeBig(k.Y)
+			if c != nil && x != nil && y != nil && c.IsOnCurve(x, y) {
+				keys[k.Kid] = &ecdsa.PublicKey{Curve: c, X: x, Y: y}
+			}
+		}
+	}
+	o.keys, o.keysAt = keys, time.Now()
+	if k := pick(); k != nil {
+		return k, nil
+	}
+	return nil, fmt.Errorf("id_token: unknown key %q", kid)
+}
+
+func (o *OIDC) getJSON(u string, v any) error {
+	res, err := o.client.Get(u)
+	if err != nil {
+		return err
+	}
+	defer res.Body.Close()
+	if res.StatusCode != http.StatusOK {
+		return fmt.Errorf("%s: %s", u, res.Status)
+	}
+	return json.NewDecoder(io.LimitReader(res.Body, 1<<20)).Decode(v)
+}
+
+// Cookies are base64url(JSON) + "." + base64url(HMAC-SHA256).
+func signCookie(key []byte, v any) string {
+	b, _ := json.Marshal(v)
+	p := base64.RawURLEncoding.EncodeToString(b)
+	m := hmac.New(sha256.New, key)
+	m.Write([]byte(p))
+	return p + "." + base64.RawURLEncoding.EncodeToString(m.Sum(nil))
+}
+
+func verifyCookie(key []byte, s string, v any) bool {
+	p, sig, ok := strings.Cut(s, ".")
+	if !ok {
+		return false
+	}
+	got, err := base64.RawURLEncoding.DecodeString(sig)
+	if err != nil {
+		return false
+	}
+	m := hmac.New(sha256.New, key)
+	m.Write([]byte(p))
+	if !hmac.Equal(got, m.Sum(nil)) {
+		return false
+	}
+	return decodeSegment(p, v) == nil
+}
+
+func decodeSegment(s string, v any) error {
+	b, err := base64.RawURLEncoding.DecodeString(s)
+	if err != nil {
+		return err
+	}
+	return json.Unmarshal(b, v)
+}
+
+func decodeBig(s string) *big.Int {
+	b, err := base64.RawURLEncoding.DecodeString(s)
+	if err != nil || len(b) == 0 {
+		return nil
+	}
+	return new(big.Int).SetBytes(b)
+}
+
+func randomString() string {
+	b := make([]byte, 16)
+	if _, err := rand.Read(b); err != nil {
+		panic(err)
+	}
+	return base64.RawURLEncoding.EncodeToString(b)
+}
+
+func addQuery(endpoint string, q url.Values) string {
+	sep := "?"
+	if strings.Contains(endpoint, "?") {
+		sep = "&"
+	}
+	return endpoint + sep + q.Encode()
+}
+
+func contains(list []string, s string) bool {
+	for _, v := range list {
+		if v == s {
+			return true
+		}
+	}
+	return false
+}

auth_local.go

@@ -0,0 +1,189 @@
+package main
+
+import (
+	"crypto/hmac"
+	"crypto/sha256"
+	"crypto/subtle"
+	"log"
+	"net"
+	"net/http"
+	"net/url"
+	"os"
+	"strings"
+	"time"
+)
+
+// Local mode with AUTH_USER set: a login page (web/login.html) in the colors of the UI and a
+// session kept in a signed cookie, like the OIDC mode. HTTP Basic credentials are still
+// accepted so scripts calling the API keep working, but the browser popup is gone.
+
+const loginPage = "/login.html"
+
+var loginFailDelay = time.Second // slows down password guessing
+
+type Local struct {
+	user, pass string
+	viewerUser string // optional read-only account
+	viewerPass string
+	ttl        time.Duration
+	logo       string // LOGIN_LOGO, served at /auth/logo
+	key        []byte
+	next       http.Handler
+}
+
+func newLocal(c authConfig) *Local {
+	// The key depends on the credentials: changing AUTH_USER or AUTH_PASS ends every session.
+	m := hmac.New(sha256.New, sessionKey(c.dataDir))
+	m.Write([]byte("local\x00" + c.user + "\x00" + c.pass + "\x00" + c.viewerUser + "\x00" + c.viewerPass))
+	if c.loginLogo != "" {
+		if _, err := os.Stat(c.loginLogo); err != nil {
+			log.Printf("auth: LOGIN_LOGO: %v", err)
+		}
+	}
+	log.Printf("local authentication enabled (user %s)", c.user)
+	if c.viewerUser != "" {
+		log.Printf("local read-only account enabled (user %s)", c.viewerUser)
+	}
+	return &Local{user: c.user, pass: c.pass, viewerUser: c.viewerUser, viewerPass: c.viewerPass, ttl: c.sessionTTL, logo: c.loginLogo, key: m.Sum(nil)}
+}
+
+func (l *Local) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	switch r.URL.Path {
+	case "/healthz", "/style.css":
+		l.next.ServeHTTP(w, r)
+		return
+	case "/auth/logo":
+		l.serveLogo(w, r)
+		return
+	case "/auth/login":
+		l.handleLogin(w, r)
+		return
+	case "/auth/logout":
+		http.SetCookie(w, &http.Cookie{Name: sessionCookie, Path: "/", MaxAge: -1, HttpOnly: true, Secure: isHTTPS(r)})
+		http.Redirect(w, r, loginPage, http.StatusFound)
+		return
+	}
+	s, ok := l.sessionUser(r)
+	if !ok {
+		if u, p, basic := r.BasicAuth(); basic {
+			if viewer, valid := l.check(u, p); valid {
+				s, ok = session{User: u, Viewer: viewer}, true
+			}
+		}
+	}
+	if ok && s.Viewer {
+		r = asViewer(r)
+	}
+	switch {
+	case r.URL.Path == loginPage:
+		if ok {
+			http.Redirect(w, r, safeReturn(r.URL.Query().Get("r")), http.StatusFound)
+			return
+		}
+		w.Header().Set("Cache-Control", "no-store")
+		l.next.ServeHTTP(w, r)
+	case ok && r.URL.Path == "/auth/me":
+		writeJSON(w, http.StatusOK, map[string]string{"mode": "local", "user": s.User, "role": roleName(s.Viewer)})
+	case ok:
+		l.next.ServeHTTP(w, r)
+	case r.Method == http.MethodGet && !strings.HasPrefix(r.URL.Path, "/api/") && r.URL.Path != "/auth/me":
+		target := loginPage
+		if ret := r.URL.RequestURI(); ret != "/" {
+			target += "?" + url.Values{"r": {ret}}.Encode()
+		}
+		http.Redirect(w, r, target, http.StatusFound)
+	default:
+		writeAuthRequired(w)
+	}
+}
+
+func (l *Local) handleLogin(w http.ResponseWriter, r *http.Request) {
+	if r.Method != http.MethodPost {
+		http.Redirect(w, r, loginPage, http.StatusFound)
+		return
+	}
+	user, pass := r.PostFormValue("user"), r.PostFormValue("pass")
+	ret := safeReturn(r.PostFormValue("r"))
+	viewer, valid := l.check(user, pass)
+	if !valid {
+		log.Printf("auth: failed login for %q from %s", user, clientIP(r))
+		time.Sleep(loginFailDelay)
+		q := url.Values{"e": {"1"}}
+		if ret != "/" {
+			q.Set("r", ret)
+		}
+		http.Redirect(w, r, loginPage+"?"+q.Encode(), http.StatusSeeOther)
+		return
+	}
+	log.Printf("auth: %s logged in from %s (%s)", user, clientIP(r), roleName(viewer))
+	http.SetCookie(w, &http.Cookie{
+		Name:     sessionCookie,
+		Value:    signCookie(l.key, session{User: user, Exp: time.Now().Add(l.ttl).Unix(), Viewer: viewer}),
+		Path:     "/",
+		MaxAge:   int(l.ttl.Seconds()),
+		HttpOnly: true,
+		Secure:   isHTTPS(r),
+		SameSite: http.SameSiteLaxMode,
+	})
+	http.Redirect(w, r, ret, http.StatusSeeOther)
+}
+
+func (l *Local) sessionUser(r *http.Request) (session, bool) {
+	var s session
+	c, err := r.Cookie(sessionCookie)
+	if err != nil || !verifyCookie(l.key, c.Value, &s) || time.Now().Unix() >= s.Exp {
+		return session{}, false
+	}
+	return s, true
+}
+
+// check validates a user and password: the admin account, or the read-only one
+// (viewer=true) when AUTH_VIEWER_USER is set.
+func (l *Local) check(user, pass string) (viewer, ok bool) {
+	if same(user, l.user) && same(pass, l.pass) {
+		return false, true
+	}
+	if l.viewerUser != "" && same(user, l.viewerUser) && same(pass, l.viewerPass) {
+		return true, true
+	}
+	return false, false
+}
+
+// same compares in constant time, so the answer time says nothing of the secret.
+func same(a, b string) bool {
+	return subtle.ConstantTimeCompare([]byte(a), []byte(b)) == 1
+}
+
+// serveLogo sends LOGIN_LOGO; without it the login page hides the image.
+func (l *Local) serveLogo(w http.ResponseWriter, r *http.Request) {
+	if l.logo == "" {
+		http.NotFound(w, r)
+		return
+	}
+	w.Header().Set("Cache-Control", "no-cache")
+	http.ServeFile(w, r, l.logo)
+}
+
+// safeReturn keeps the page to open after login inside logstream.
+func safeReturn(ret string) string {
+	if !strings.HasPrefix(ret, "/") || strings.HasPrefix(ret, "//") || strings.HasPrefix(ret, "/\\") || ret == loginPage {
+		return "/"
+	}
+	return ret
+}
+
+// isHTTPS is true when the browser talks HTTPS, directly or through a reverse proxy.
+func isHTTPS(r *http.Request) bool {
+	return r.TLS != nil || strings.EqualFold(r.Header.Get("X-Forwarded-Proto"), "https")
+}
+
+func clientIP(r *http.Request) string {
+	if f := r.Header.Get("X-Forwarded-For"); f != "" {
+		return strings.TrimSpace(strings.Split(f, ",")[0])
+	}
+	host, _, err := net.SplitHostPort(r.RemoteAddr)
+	if err != nil {
+		return r.RemoteAddr
+	}
+	return host
+}

auth_local_test.go

@@ -0,0 +1,142 @@
+package main
+
+import (
+	"net/http"
+	"net/http/cookiejar"
+	"net/url"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// newLocalApp puts the local login in front of a handler that echoes "app" and returns a
+// browser (client with cookies) that does not follow redirects.
+func newLocalApp(t *testing.T, c authConfig) (string, *http.Client) {
+	t.Helper()
+	loginFailDelay = 0
+	c.mode, c.dataDir = "local", t.TempDir()
+	h, err := newAuth(c, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("app " + r.URL.Path)) }))
+	if err != nil {
+		t.Fatal(err)
+	}
+	jar, _ := cookiejar.New(nil)
+	return "http://app.test", &http.Client{
+		Jar:           jar,
+		Transport:     hosts{"app.test": h},
+		CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
+	}
+}
+
+func login(t *testing.T, c *http.Client, app, user, pass, ret string) *http.Response {
+	t.Helper()
+	res, err := c.PostForm(app+"/auth/login", url.Values{"user": {user}, "pass": {pass}, "r": {ret}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	res.Body.Close()
+	return res
+}
+
+func TestLocalLoginFlow(t *testing.T) {
+	app, c := newLocalApp(t, authConfig{user: "admin", pass: "pw"})
+
+	res, _ := c.Get(app + "/api/logs?q=x")
+	if res.StatusCode != http.StatusUnauthorized || res.Header.Get("WWW-Authenticate") != "" {
+		t.Fatalf("API without session: %d %q", res.StatusCode, res.Header.Get("WWW-Authenticate"))
+	}
+	res, _ = c.Get(app + "/?q=disk")
+	if loc := res.Header.Get("Location"); res.StatusCode != http.StatusFound || loc != "/login.html?r=%2F%3Fq%3Ddisk" {
+		t.Fatalf("page without session: %d %q", res.StatusCode, loc)
+	}
+	for _, p := range []string{"/login.html", "/style.css", "/healthz"} {
+		if code, body := get(t, c, app+p); code != http.StatusOK || body != "app "+p {
+			t.Errorf("%s without session: %d %q", p, code, body)
+		}
+	}
+
+	res = login(t, c, app, "admin", "wrong", "/?q=disk")
+	if loc := res.Header.Get("Location"); res.StatusCode != http.StatusSeeOther || !strings.Contains(loc, "e=1") {
+		t.Fatalf("wrong password: %d %q", res.StatusCode, loc)
+	}
+	if code, _ := get(t, c, app+"/api/logs"); code != http.StatusUnauthorized {
+		t.Fatal("session created by a wrong password")
+	}
+
+	res = login(t, c, app, "admin", "pw", "//evil.example/")
+	if loc := res.Header.Get("Location"); loc != "/" {
+		t.Fatalf("open redirect: %q", loc)
+	}
+	res = login(t, c, app, "admin", "pw", "/?q=disk")
+	if loc := res.Header.Get("Location"); res.StatusCode != http.StatusSeeOther || loc != "/?q=disk" {
+		t.Fatalf("login: %d %q", res.StatusCode, loc)
+	}
+	if code, body := get(t, c, app+"/api/logs"); code != http.StatusOK || body != "app /api/logs" {
+		t.Fatalf("API with session: %d %q", code, body)
+	}
+	if code, body := get(t, c, app+"/auth/me"); code != http.StatusOK || !strings.Contains(body, `"user":"admin"`) || !strings.Contains(body, `"mode":"local"`) {
+		t.Fatalf("/auth/me: %d %s", code, body)
+	}
+	if res, _ := c.Get(app + "/login.html"); res.StatusCode != http.StatusFound {
+		t.Errorf("login page while logged in: %d", res.StatusCode)
+	}
+
+	if res, _ := c.Get(app + "/auth/logout"); res.Header.Get("Location") != "/login.html" {
+		t.Fatalf("logout: %q", res.Header.Get("Location"))
+	}
+	if code, _ := get(t, c, app+"/api/logs"); code != http.StatusUnauthorized {
+		t.Fatal("session still valid after logout")
+	}
+}
+
+func TestLocalBasicAuthForScripts(t *testing.T) {
+	app, c := newLocalApp(t, authConfig{user: "admin", pass: "pw"})
+	req, _ := http.NewRequest(http.MethodGet, app+"/api/logs", nil)
+	req.SetBasicAuth("admin", "pw")
+	if res, _ := c.Do(req); res.StatusCode != http.StatusOK {
+		t.Fatalf("basic auth: %d", res.StatusCode)
+	}
+	req.SetBasicAuth("admin", "nope")
+	if res, _ := c.Do(req); res.StatusCode != http.StatusUnauthorized {
+		t.Fatalf("wrong basic auth: %d", res.StatusCode)
+	}
+}
+
+func TestLocalPasswordChangeEndsSessions(t *testing.T) {
+	dir := t.TempDir()
+	loginFailDelay = 0
+	echo := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {})
+	h1, _ := newAuth(authConfig{mode: "local", user: "admin", pass: "old", dataDir: dir}, echo)
+	h2, _ := newAuth(authConfig{mode: "local", user: "admin", pass: "new", dataDir: dir}, echo)
+	cookie := signCookie(h1.(*Local).key, session{User: "admin", Exp: 9999999999})
+	r, _ := http.NewRequest(http.MethodGet, "/", nil)
+	r.AddCookie(&http.Cookie{Name: sessionCookie, Value: cookie})
+	if _, ok := h1.(*Local).sessionUser(r); !ok {
+		t.Fatal("session refused with the same password")
+	}
+	if _, ok := h2.(*Local).sessionUser(r); ok {
+		t.Fatal("session kept after a password change")
+	}
+}
+
+func TestLocalLogo(t *testing.T) {
+	app, c := newLocalApp(t, authConfig{user: "admin", pass: "pw"})
+	if code, _ := get(t, c, app+"/auth/logo"); code != http.StatusNotFound {
+		t.Errorf("no LOGIN_LOGO: %d", code)
+	}
+	png := filepath.Join(t.TempDir(), "logo.png")
+	_ = os.WriteFile(png, []byte("\x89PNG\r\n\x1a\nfake"), 0o644)
+	app, c = newLocalApp(t, authConfig{user: "admin", pass: "pw", loginLogo: png})
+	res, _ := c.Get(app + "/auth/logo")
+	if res.StatusCode != http.StatusOK || res.Header.Get("Content-Type") != "image/png" {
+		t.Errorf("LOGIN_LOGO: %d %q", res.StatusCode, res.Header.Get("Content-Type"))
+	}
+}
+
+func TestLocalWithoutUserIsOpen(t *testing.T) {
+	if h, _ := newAuth(authConfig{mode: "local"}, http.NotFoundHandler()); h == nil || isLocal(h) {
+		t.Error("local mode without AUTH_USER should not protect anything")
+	}
+}
+
+func isLocal(h http.Handler) bool { _, ok := h.(*Local); return ok }

auth_test.go

@@ -0,0 +1,230 @@
+package main
+
+import (
+	"crypto"
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/rsa"
+	"crypto/sha256"
+	"encoding/base64"
+	"encoding/json"
+	"math/big"
+	"net/http"
+	"net/http/cookiejar"
+	"net/http/httptest"
+	"net/url"
+	"strings"
+	"testing"
+	"time"
+)
+
+// hosts routes requests to in-memory handlers (no listening socket needed).
+type hosts map[string]http.Handler
+
+func (h hosts) RoundTrip(r *http.Request) (*http.Response, error) {
+	rec := httptest.NewRecorder()
+	h[r.URL.Host].ServeHTTP(rec, r)
+	res := rec.Result()
+	res.Request = r
+	return res, nil
+}
+
+// fakeIdP is a minimal OpenID provider: it logs in "alice" without asking.
+type fakeIdP struct {
+	mux      *http.ServeMux
+	rsaKey   *rsa.PrivateKey
+	ecKey    *ecdsa.PrivateKey
+	useEC    bool
+	codes    map[string]url.Values // code -> authorize request
+	claims   func(map[string]any)  // last-minute changes to the ID token
+	tokenErr bool
+}
+
+func newFakeIdP(t *testing.T) *fakeIdP {
+	rk, _ := rsa.GenerateKey(rand.Reader, 2048)
+	ek, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	p := &fakeIdP{rsaKey: rk, ecKey: ek, codes: map[string]url.Values{}}
+	mux := http.NewServeMux()
+	p.mux = mux
+	iss := "http://idp.test/realm"
+	mux.HandleFunc("/realm/.well-known/openid-configuration", func(w http.ResponseWriter, r *http.Request) {
+		_ = json.NewEncoder(w).Encode(map[string]any{
+			"issuer": iss, "authorization_endpoint": iss + "/auth", "token_endpoint": iss + "/token",
+			"jwks_uri": iss + "/jwks", "end_session_endpoint": iss + "/logout",
+		})
+	})
+	mux.HandleFunc("/realm/jwks", func(w http.ResponseWriter, r *http.Request) {
+		b := func(i *big.Int) string { return base64.RawURLEncoding.EncodeToString(i.Bytes()) }
+		_ = json.NewEncoder(w).Encode(map[string]any{"keys": []any{
+			map[string]string{"kty": "RSA", "kid": "r1", "use": "sig", "n": b(rk.N), "e": "AQAB"},
+			map[string]string{"kty": "EC", "kid": "e1", "crv": "P-256", "x": b(ek.X), "y": b(ek.Y)},
+		}})
+	})
+	mux.HandleFunc("/realm/auth", func(w http.ResponseWriter, r *http.Request) {
+		q := r.URL.Query()
+		code := randomString()
+		p.codes[code] = q
+		http.Redirect(w, r, q.Get("redirect_uri")+"?code="+code+"&state="+q.Get("state"), http.StatusFound)
+	})
+	mux.HandleFunc("/realm/token", func(w http.ResponseWriter, r *http.Request) {
+		_ = r.ParseForm()
+		id, secret, _ := r.BasicAuth()
+		authz, ok := p.codes[r.Form.Get("code")]
+		sum := sha256.Sum256([]byte(r.Form.Get("code_verifier")))
+		if p.tokenErr || !ok || id != "logstream" || secret != "s3cret" ||
+			base64.RawURLEncoding.EncodeToString(sum[:]) != authz.Get("code_challenge") ||
+			r.Form.Get("redirect_uri") != authz.Get("redirect_uri") {
+			http.Error(w, `{"error":"invalid_grant"}`, http.StatusBadRequest)
+			return
+		}
+		delete(p.codes, r.Form.Get("code"))
+		c := map[string]any{
+			"iss": iss, "aud": "logstream", "sub": "123", "preferred_username": "alice",
+			"exp": time.Now().Add(5 * time.Minute).Unix(), "iat": time.Now().Unix(), "nonce": authz.Get("nonce"),
+		}
+		if p.claims != nil {
+			p.claims(c)
+		}
+		_ = json.NewEncoder(w).Encode(map[string]string{"access_token": "x", "id_token": p.sign(c)})
+	})
+	return p
+}
+
+func (p *fakeIdP) sign(claims map[string]any) string {
+	alg, kid := "RS256", "r1"
+	if p.useEC {
+		alg, kid = "ES256", "e1"
+	}
+	h, _ := json.Marshal(map[string]string{"alg": alg, "kid": kid, "typ": "JWT"})
+	c, _ := json.Marshal(claims)
+	in := base64.RawURLEncoding.EncodeToString(h) + "." + base64.RawURLEncoding.EncodeToString(c)
+	d := sha256.Sum256([]byte(in))
+	var sig []byte
+	if p.useEC {
+		r, s, _ := ecdsa.Sign(rand.Reader, p.ecKey, d[:])
+		sig = make([]byte, 64)
+		r.FillBytes(sig[:32])
+		s.FillBytes(sig[32:])
+	} else {
+		sig, _ = rsa.SignPKCS1v15(rand.Reader, p.rsaKey, crypto.SHA256, d[:])
+	}
+	return in + "." + base64.RawURLEncoding.EncodeToString(sig)
+}
+
+// newOIDCApp puts logstream's auth in front of a handler that echoes "app" and returns
+// a browser (client with cookies) that reaches both the app and the provider.
+func newOIDCApp(t *testing.T, idp *fakeIdP) (string, *http.Client) {
+	h, err := newAuth(authConfig{
+		mode: "oidc", issuer: "http://idp.test/realm", clientID: "logstream", clientSecret: "s3cret",
+		redirectURL: "http://app.test/auth/callback", dataDir: t.TempDir(),
+	}, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("app " + r.URL.Path)) }))
+	if err != nil {
+		t.Fatal(err)
+	}
+	net := hosts{"app.test": h, "idp.test": idp.mux}
+	h.(*OIDC).client.Transport = net
+	jar, _ := cookiejar.New(nil)
+	return "http://app.test", &http.Client{Jar: jar, Transport: net}
+}
+
+func get(t *testing.T, c *http.Client, u string) (int, string) {
+	t.Helper()
+	res, err := c.Get(u)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer res.Body.Close()
+	var b strings.Builder
+	buf := make([]byte, 4096)
+	for {
+		n, err := res.Body.Read(buf)
+		b.Write(buf[:n])
+		if err != nil {
+			break
+		}
+	}
+	return res.StatusCode, b.String()
+}
+
+func TestOIDCLoginFlow(t *testing.T) {
+	for _, ec := range []bool{false, true} {
+		idp := newFakeIdP(t)
+		idp.useEC = ec
+		app, c := newOIDCApp(t, idp)
+
+		if code, _ := get(t, c, app+"/api/logs"); code != http.StatusUnauthorized {
+			t.Fatalf("api without session: %d", code)
+		}
+		if code, body := get(t, c, app+"/healthz"); code != 200 || body != "app /healthz" {
+			t.Fatalf("healthz: %d %q", code, body)
+		}
+		// A page goes through the provider and comes back to the page asked for.
+		if code, body := get(t, c, app+"/index.html?x=1"); code != 200 || body != "app /index.html" {
+			t.Fatalf("login (ec=%v): %d %q", ec, code, body)
+		}
+		if code, body := get(t, c, app+"/api/logs"); code != 200 || body != "app /api/logs" {
+			t.Fatalf("api with session: %d %q", code, body)
+		}
+		if code, body := get(t, c, app+"/auth/me"); code != 200 || !strings.Contains(body, `"user":"alice"`) {
+			t.Fatalf("me: %d %q", code, body)
+		}
+		// Logout drops the session (the fake provider has no logout page: 404).
+		get(t, c, app+"/auth/logout")
+		if code, _ := get(t, c, app+"/api/logs"); code != http.StatusUnauthorized {
+			t.Fatalf("api after logout: %d", code)
+		}
+	}
+}
+
+func TestOIDCRejectsBadTokens(t *testing.T) {
+	cases := map[string]func(map[string]any){
+		"wrong nonce":    func(c map[string]any) { c["nonce"] = "x" },
+		"wrong audience": func(c map[string]any) { c["aud"] = "other" },
+		"wrong issuer":   func(c map[string]any) { c["iss"] = "https://evil" },
+		"expired":        func(c map[string]any) { c["exp"] = time.Now().Add(-time.Hour).Unix() },
+	}
+	for name, change := range cases {
+		idp := newFakeIdP(t)
+		idp.claims = change
+		app, c := newOIDCApp(t, idp)
+		if code, _ := get(t, c, app+"/"); code != http.StatusForbidden {
+			t.Errorf("%s: login gave %d, expected 403", name, code)
+		}
+		if code, _ := get(t, c, app+"/api/logs"); code != http.StatusUnauthorized {
+			t.Errorf("%s: session created", name)
+		}
+	}
+}
+
+func TestOIDCCallbackNeedsLoginCookie(t *testing.T) {
+	idp := newFakeIdP(t)
+	app, c := newOIDCApp(t, idp)
+	if code, _ := get(t, c, app+"/auth/callback?code=abc&state=forged"); code != http.StatusBadRequest {
+		t.Fatalf("forged callback: %d", code)
+	}
+}
+
+func TestOIDCForgedSessionCookie(t *testing.T) {
+	idp := newFakeIdP(t)
+	app, c := newOIDCApp(t, idp)
+	u, _ := url.Parse(app)
+	payload := base64.RawURLEncoding.EncodeToString([]byte(`{"u":"mallory","e":9999999999}`))
+	c.Jar.SetCookies(u, []*http.Cookie{{Name: sessionCookie, Value: payload + ".AAAA"}})
+	if code, _ := get(t, c, app+"/api/logs"); code != http.StatusUnauthorized {
+		t.Fatalf("forged session accepted: %d", code)
+	}
+}
+
+func TestAuthModeConfig(t *testing.T) {
+	next := http.NotFoundHandler()
+	if _, err := newAuth(authConfig{mode: "oidc"}, next); err == nil || !strings.Contains(err.Error(), "OIDC_CLIENT_ID") {
+		t.Errorf("missing variables not reported: %v", err)
+	}
+	if _, err := newAuth(authConfig{mode: "ldap"}, next); err == nil {
+		t.Error("unknown mode accepted")
+	}
+	if h, err := newAuth(authConfig{mode: "local"}, next); err != nil || h == nil {
+		t.Errorf("local mode: %v", err)
+	}
+}

dbstats.go

@@ -0,0 +1,167 @@
+package main
+
+import (
+	"bufio"
+	"context"
+	"fmt"
+	"io"
+	"net/http"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+// DBStats describes what VictoriaLogs holds (Settings > Data). Sizes are in
+// bytes; a field VictoriaLogs did not give stays at its zero value.
+type DBStats struct {
+	Rows       int64  `json:"rows"`       // stored log lines
+	DiskBytes  int64  `json:"diskBytes"`  // compressed data + index, on disk
+	IndexBytes int64  `json:"indexBytes"` // index part of DiskBytes
+	RawBytes   int64  `json:"rawBytes"`   // data before compression
+	FreeBytes  int64  `json:"freeBytes"`  // free space on the VictoriaLogs volume
+	Days       int64  `json:"days"`       // per-day partitions
+	Retention  string `json:"retention"`  // -retentionPeriod, e.g. "30d"
+	Oldest     string `json:"oldest,omitempty"`
+	Newest     string `json:"newest,omitempty"`
+	Last1h     int64  `json:"last1h"`
+	Last24h    int64  `json:"last24h"`
+	Hosts      int64  `json:"hosts"`
+	Apps       int64  `json:"apps"`
+	QueryError string `json:"queryError,omitempty"` // the LogsQL part failed, the metrics are still valid
+	Updated    string `json:"updated"`
+}
+
+// dbStatsCache keeps the last answer for a while: the LogsQL part reads the
+// whole base, which is too heavy to repeat each time the tab is opened.
+type dbStatsCache struct {
+	mu   sync.Mutex
+	at   time.Time
+	last *DBStats
+}
+
+const dbStatsTTL = 30 * time.Second
+
+func (c *dbStatsCache) get(ctx context.Context, s *Store, force bool) (*DBStats, error) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	if c.last != nil && !force && time.Since(c.at) < dbStatsTTL {
+		return c.last, nil
+	}
+	st, err := s.DBStats(ctx)
+	if err != nil {
+		return nil, err
+	}
+	c.last, c.at = st, time.Now()
+	return st, nil
+}
+
+// DBStats reads the storage metrics of VictoriaLogs (/metrics), then counts
+// with LogsQL what the metrics do not tell (period, recent volume, sources).
+func (s *Store) DBStats(ctx context.Context) (*DBStats, error) {
+	st, err := s.storageMetrics(ctx)
+	if err != nil {
+		return nil, err
+	}
+	st.Updated = time.Now().UTC().Format(time.RFC3339)
+	if st.Rows == 0 {
+		return st, nil
+	}
+
+	qctx, cancel := context.WithTimeout(ctx, 20*time.Second)
+	defer cancel()
+	rows, err := s.Query(qctx, "* | stats min(_time) oldest, max(_time) newest, count_uniq(host) hosts, count_uniq(app) apps")
+	if err == nil && len(rows) > 0 {
+		r := rows[0]
+		st.Oldest, _ = r["oldest"].(string)
+		st.Newest, _ = r["newest"].(string)
+		st.Hosts, st.Apps = toInt(r["hosts"]), toInt(r["apps"])
+		rows, err = s.Query(qctx, "_time:24h | stats count() last24h, count() if (_time:1h) last1h")
+		if err == nil && len(rows) > 0 {
+			st.Last24h, st.Last1h = toInt(rows[0]["last24h"]), toInt(rows[0]["last1h"])
+		}
+	}
+	if err != nil {
+		st.QueryError = err.Error()
+	}
+	return st, nil
+}
+
+func (s *Store) storageMetrics(ctx context.Context) (*DBStats, error) {
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, s.base+"/metrics", nil)
+	if err != nil {
+		return nil, err
+	}
+	resp, err := s.client.Do(req)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		msg, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
+		return nil, fmt.Errorf("VictoriaLogs /metrics: HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(msg)))
+	}
+	return parseStorageMetrics(resp.Body)
+}
+
+// parseStorageMetrics picks the storage figures out of the Prometheus text
+// format of VictoriaLogs' /metrics (app/vlstorage).
+func parseStorageMetrics(r io.Reader) (*DBStats, error) {
+	st := &DBStats{}
+	sc := bufio.NewScanner(r)
+	sc.Buffer(make([]byte, 64*1024), 1<<20)
+	for sc.Scan() {
+		line := sc.Text()
+		if line == "" || line[0] == '#' {
+			continue
+		}
+		name, labels, value := splitMetric(line)
+		switch name {
+		case "vl_storage_rows":
+			st.Rows += int64(value)
+		case "vl_data_size_bytes":
+			st.DiskBytes += int64(value)
+			if labels["type"] == "indexdb" {
+				st.IndexBytes += int64(value)
+			}
+		case "vl_uncompressed_data_size_bytes":
+			st.RawBytes += int64(value)
+		case "vl_free_disk_space_bytes":
+			st.FreeBytes = int64(value)
+		case "vl_partitions":
+			st.Days = int64(value)
+		case "flag":
+			if labels["name"] == "retentionPeriod" {
+				st.Retention = labels["value"]
+			}
+		}
+	}
+	return st, sc.Err()
+}
+
+// splitMetric splits `name{a="x",b="y"} 12` into its parts. Label values with
+// escaped quotes are not expected in the metrics read here.
+func splitMetric(line string) (string, map[string]string, float64) {
+	sp := strings.LastIndexByte(line, ' ')
+	if sp < 0 {
+		return "", nil, 0
+	}
+	value, _ := strconv.ParseFloat(line[sp+1:], 64)
+	head := line[:sp]
+	name, rest, ok := strings.Cut(head, "{")
+	if !ok {
+		return head, nil, value
+	}
+	labels := map[string]string{}
+	rest = strings.TrimSuffix(rest, "}")
+	for rest != "" {
+		k, v, ok := strings.Cut(rest, `="`)
+		if !ok {
+			break
+		}
+		val, after, _ := strings.Cut(v, `"`)
+		labels[strings.TrimSpace(k)] = val
+		rest = strings.TrimPrefix(after, ",")
+	}
+	return name, labels, value
+}

dbstats_test.go

@@ -0,0 +1,89 @@
+package main
+
+import (
+	"encoding/json"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+)
+
+// Excerpt of VictoriaLogs v1.52 /metrics.
+const vlMetrics = `# a comment
+vl_free_disk_space_bytes{path="/vlogs"} 52428800000
+vl_storage_rows{type="storage/inmemory"} 120
+vl_storage_rows{type="storage/small"} 3000
+vl_storage_rows{type="storage/big"} 1000000
+vl_partitions 12
+vl_data_size_bytes{type="indexdb"} 2048
+vl_data_size_bytes{type="storage"} 1048576
+vl_compressed_data_size_bytes{type="storage/big"} 1000000
+vl_uncompressed_data_size_bytes{type="storage/inmemory"} 4096
+vl_uncompressed_data_size_bytes{type="storage/small"} 100000
+vl_uncompressed_data_size_bytes{type="storage/big"} 20000000
+flag{name="retentionPeriod", value="30d", is_set="true"} 1
+flag{name="httpListenAddr", value=":9428", is_set="true"} 1
+`
+
+func TestParseStorageMetrics(t *testing.T) {
+	st, err := parseStorageMetrics(strings.NewReader(vlMetrics))
+	if err != nil {
+		t.Fatal(err)
+	}
+	want := DBStats{Rows: 1003120, DiskBytes: 1050624, IndexBytes: 2048, RawBytes: 20104096, FreeBytes: 52428800000, Days: 12, Retention: "30d"}
+	if *st != want {
+		t.Fatalf("got %+v\nwant %+v", *st, want)
+	}
+}
+
+// statsVL answers /metrics and the LogsQL queries of DBStats.
+type statsVL struct{ queries int }
+
+func (v *statsVL) RoundTrip(r *http.Request) (*http.Response, error) {
+	body := vlMetrics
+	if r.URL.Path == "/select/logsql/query" {
+		v.queries++
+		_ = r.ParseForm()
+		if strings.HasPrefix(r.PostForm.Get("query"), "_time:24h") {
+			body = `{"last24h":"5000","last1h":"300"}` + "\n"
+		} else {
+			body = `{"oldest":"2026-09-21T08:00:00Z","newest":"2026-10-03T15:00:00Z","hosts":"4","apps":"17"}` + "\n"
+		}
+	}
+	return &http.Response{StatusCode: http.StatusOK, Body: io.NopCloser(strings.NewReader(body)), Header: http.Header{}}, nil
+}
+
+func TestDBStatsHandler(t *testing.T) {
+	vl := &statsVL{}
+	store := NewStore("http://vl", 10, 10, time.Second, nil)
+	store.client.Transport = vl
+	store.streamClient.Transport = vl
+	a := &API{store: store}
+
+	get := func(url string) DBStats {
+		rec := httptest.NewRecorder()
+		a.dbStats(rec, httptest.NewRequest("GET", url, nil))
+		if rec.Code != http.StatusOK {
+			t.Fatalf("status %d: %s", rec.Code, rec.Body)
+		}
+		var st DBStats
+		if err := json.Unmarshal(rec.Body.Bytes(), &st); err != nil {
+			t.Fatal(err)
+		}
+		return st
+	}
+	st := get("/api/dbstats")
+	if st.Rows != 1003120 || st.Oldest != "2026-09-21T08:00:00Z" || st.Hosts != 4 || st.Apps != 17 || st.Last24h != 5000 || st.Last1h != 300 || st.QueryError != "" {
+		t.Fatalf("unexpected stats: %+v", st)
+	}
+	get("/api/dbstats")
+	if vl.queries != 2 {
+		t.Fatalf("second call within the TTL should be cached, got %d queries", vl.queries)
+	}
+	get("/api/dbstats?refresh=1")
+	if vl.queries != 4 {
+		t.Fatalf("refresh=1 should query again, got %d queries", vl.queries)
+	}
+}

dmesg.go

@@ -0,0 +1,481 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"io"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+	"unicode/utf8"
+)
+
+// dmesgConfig is saved in /data/dmesg.json and edited in Settings > Sources.
+type dmesgConfig struct {
+	Enabled bool `json:"enabled"`
+}
+
+// dmesgPos is where reading resumes: the last record sent, valid for one boot
+// only (the kernel numbers its records from 0 at each boot).
+type dmesgPos struct {
+	BootID string `json:"boot_id"`
+	Seq    uint64 `json:"seq"`
+}
+
+// Dmesg collects the kernel ring buffer of the machine hosting the stack by
+// reading /dev/kmsg, the device dmesg itself reads.
+type Dmesg struct {
+	device     string // /dev/kmsg
+	bootIDPath string // identifies the current boot
+	uptimePath string // gives the boot time, to date the records
+	sink       func(*Entry)
+	backfill   time.Duration
+	cfgPath    string
+	statePath  string
+	retry      time.Duration // delay before opening the device again after a failure
+	wake       chan struct{}
+
+	mu        sync.Mutex
+	cfg       dmesgConfig
+	reset     bool
+	pos       dmesgPos
+	dirty     bool
+	open      bool
+	read      uint64
+	lost      uint64
+	errCode   string
+	errDetail string
+}
+
+func NewDmesg(device, dataDir string, backfill time.Duration, sink func(*Entry)) *Dmesg {
+	d := &Dmesg{
+		device:     device,
+		bootIDPath: "/proc/sys/kernel/random/boot_id",
+		uptimePath: "/proc/uptime",
+		sink:       sink,
+		backfill:   backfill,
+		cfgPath:    filepath.Join(dataDir, "dmesg.json"),
+		statePath:  filepath.Join(dataDir, "dmesg-state.json"),
+		retry:      10 * time.Second,
+		wake:       make(chan struct{}, 1),
+	}
+	if b, err := os.ReadFile(d.cfgPath); err == nil {
+		_ = json.Unmarshal(b, &d.cfg)
+	}
+	if b, err := os.ReadFile(d.statePath); err == nil {
+		_ = json.Unmarshal(b, &d.pos)
+	}
+	return d
+}
+
+// kmsgSession is one opening of the device, read by its own goroutine.
+type kmsgSession struct {
+	f       *os.File
+	done    chan struct{}
+	err     error
+	stopped bool // closed on purpose: err is not a failure
+}
+
+func (s *kmsgSession) stop() {
+	s.stopped = true
+	s.f.Close() // ends a pending read (/dev/kmsg goes through the poller)
+	select {
+	case <-s.done:
+	case <-time.After(2 * time.Second):
+	}
+}
+
+// Run keeps the device open while the source is enabled.
+func (d *Dmesg) Run(ctx context.Context) {
+	save := time.NewTicker(5 * time.Second)
+	defer save.Stop()
+	var cur *kmsgSession
+	var retryAt time.Time
+	for {
+		d.mu.Lock()
+		enabled, reset := d.cfg.Enabled, d.reset
+		d.reset = false
+		d.mu.Unlock()
+		if reset {
+			// Disabled then enabled again: start over from DMESG_BACKFILL ago
+			// rather than reading everything logged in between.
+			if cur != nil {
+				cur.stop()
+				cur = nil
+			}
+			d.mu.Lock()
+			d.pos, d.dirty = dmesgPos{}, true
+			d.mu.Unlock()
+			d.saveState()
+			retryAt = time.Time{}
+		}
+		if !enabled && cur != nil {
+			cur.stop()
+			cur = nil
+			d.mu.Lock()
+			d.open = false
+			d.mu.Unlock()
+		}
+		if enabled && cur == nil && !time.Now().Before(retryAt) {
+			var err error
+			if cur, err = d.start(); err != nil {
+				d.fail(err)
+				retryAt = time.Now().Add(d.retry)
+			}
+		}
+		var done chan struct{}
+		if cur != nil {
+			done = cur.done
+		}
+		wait := d.retry
+		if cur == nil && enabled {
+			wait = max(time.Until(retryAt), 10*time.Millisecond)
+		}
+		timer := time.NewTimer(wait)
+		select {
+		case <-ctx.Done():
+			timer.Stop()
+			if cur != nil {
+				cur.stop()
+			}
+			d.saveState()
+			return
+		case <-d.wake:
+		case <-save.C:
+			d.saveState()
+		case <-timer.C:
+		case <-done:
+			if !cur.stopped {
+				d.fail(cur.err)
+			}
+			cur = nil
+			retryAt = time.Now().Add(d.retry)
+		}
+		timer.Stop()
+	}
+}
+
+func (d *Dmesg) saveState() {
+	d.mu.Lock()
+	if !d.dirty {
+		d.mu.Unlock()
+		return
+	}
+	d.dirty = false
+	pos := d.pos
+	d.mu.Unlock()
+	if err := writeJSONFile(d.statePath, pos); err != nil {
+		logEvent("dmesg", 4, "saving position: %v", err)
+	}
+}
+
+// fail records why the device is not being read (nil error: end of a
+// regular file, only seen in tests).
+func (d *Dmesg) fail(err error) {
+	code, detail := "", ""
+	if err != nil && !errors.Is(err, io.EOF) {
+		detail = err.Error()
+		switch {
+		case errors.Is(err, fs.ErrNotExist):
+			code = "not_found"
+		case errors.Is(err, fs.ErrPermission):
+			code = "permission"
+		default:
+			code = "read"
+		}
+	}
+	d.mu.Lock()
+	changed := code != d.errCode
+	d.open, d.errCode, d.errDetail = false, code, detail
+	d.mu.Unlock()
+	if changed && code != "" {
+		logEvent("dmesg", 4, "cannot read the kernel messages: %v", err)
+	}
+}
+
+// start opens the device and starts reading it in the background.
+func (d *Dmesg) start() (*kmsgSession, error) {
+	f, err := os.Open(d.device)
+	if err != nil {
+		return nil, err
+	}
+	bootID := readTrim(d.bootIDPath)
+	boot := d.bootTime(time.Now())
+	d.mu.Lock()
+	pos := d.pos
+	d.open, d.errCode, d.errDetail = true, "", ""
+	d.mu.Unlock()
+
+	// Same boot as the saved position: skip what was already sent. Otherwise
+	// read from DMESG_BACKFILL ago (0: only new records).
+	resume := pos.BootID != "" && pos.BootID == bootID
+	var notBefore time.Time
+	if !resume {
+		if d.backfill <= 0 {
+			if _, err := f.Seek(0, io.SeekEnd); err != nil {
+				f.Close()
+				return nil, err
+			}
+		} else if !boot.IsZero() {
+			notBefore = time.Now().Add(-d.backfill)
+		}
+		d.mu.Lock()
+		d.pos, d.dirty = dmesgPos{BootID: bootID}, true
+		d.mu.Unlock()
+	}
+	s := &kmsgSession{f: f, done: make(chan struct{})}
+	go func() {
+		defer close(s.done)
+		s.err = d.consume(f, bootID, boot, resume, pos.Seq, notBefore)
+	}()
+	return s, nil
+}
+
+func readTrim(path string) string {
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return ""
+	}
+	return strings.TrimSpace(string(b))
+}
+
+// bootTime is now minus the uptime; zero when unknown (records are then dated
+// when they are read).
+func (d *Dmesg) bootTime(now time.Time) time.Time {
+	f := strings.Fields(readTrim(d.uptimePath))
+	if len(f) == 0 {
+		return time.Time{}
+	}
+	up, err := strconv.ParseFloat(f[0], 64)
+	if err != nil || up <= 0 {
+		return time.Time{}
+	}
+	return now.Add(-time.Duration(up * float64(time.Second)))
+}
+
+// consume reads records until an error. Each read of /dev/kmsg returns one
+// record; several records in one read (a regular file, in tests) are split.
+func (d *Dmesg) consume(r io.Reader, bootID string, boot time.Time, resume bool, after uint64, notBefore time.Time) error {
+	buf := make([]byte, 64<<10) // a record is at most 8 KiB
+	last, haveLast := after, resume
+	for {
+		n, err := r.Read(buf)
+		if n > 0 {
+			for _, raw := range splitKmsg(buf[:n]) {
+				rec, ok := parseKmsg(raw)
+				if !ok {
+					continue
+				}
+				if haveLast && rec.Seq <= last {
+					continue // already sent before a restart
+				}
+				var lost uint64
+				if haveLast && rec.Seq > last+1 {
+					lost = rec.Seq - last - 1 // overwritten in the ring buffer before being read
+				}
+				last, haveLast = rec.Seq, true
+				now := time.Now()
+				t := now
+				if !boot.IsZero() {
+					t = boot.Add(time.Duration(rec.Usec) * time.Microsecond)
+				}
+				if !notBefore.IsZero() && t.Before(notBefore) {
+					d.advance(bootID, rec.Seq, 0, 0)
+					continue
+				}
+				d.sink(kmsgEntry(rec, t, now))
+				d.advance(bootID, rec.Seq, 1, lost)
+			}
+		}
+		if err != nil {
+			if errors.Is(err, os.ErrClosed) {
+				return nil
+			}
+			if errors.Is(err, syscall.EPIPE) {
+				continue // records overwritten while reading: the next read goes on
+			}
+			return err
+		}
+	}
+}
+
+func (d *Dmesg) advance(bootID string, seq, read, lost uint64) {
+	d.mu.Lock()
+	d.pos, d.dirty = dmesgPos{BootID: bootID, Seq: seq}, true
+	d.read += read
+	d.lost += lost
+	d.mu.Unlock()
+}
+
+// kmsgRecord is one record of /dev/kmsg:
+//
+//	prio,seq,usec,flags[,...];message
+//	 KEY=value          (optional dictionary lines)
+type kmsgRecord struct {
+	Prio int
+	Seq  uint64
+	Usec uint64
+	Msg  string
+	Dict map[string]string
+}
+
+// splitKmsg cuts a buffer into records: a new record starts after a newline
+// that is not followed by a space (dictionary lines start with one).
+func splitKmsg(b []byte) [][]byte {
+	var out [][]byte
+	start := 0
+	for i := 0; i < len(b); i++ {
+		if b[i] == '\n' && (i+1 == len(b) || b[i+1] != ' ') {
+			out = append(out, b[start:i])
+			start = i + 1
+		}
+	}
+	if start < len(b) {
+		out = append(out, b[start:])
+	}
+	return out
+}
+
+func parseKmsg(raw []byte) (kmsgRecord, bool) {
+	var rec kmsgRecord
+	lines := bytes.Split(raw, []byte{'\n'})
+	head, msg, ok := bytes.Cut(lines[0], []byte{';'})
+	if !ok {
+		return rec, false
+	}
+	f := strings.Split(string(head), ",")
+	if len(f) < 3 {
+		return rec, false
+	}
+	prio, err1 := strconv.Atoi(f[0])
+	seq, err2 := strconv.ParseUint(f[1], 10, 64)
+	usec, err3 := strconv.ParseUint(f[2], 10, 64)
+	if err1 != nil || err2 != nil || err3 != nil || prio < 0 {
+		return rec, false
+	}
+	rec.Prio, rec.Seq, rec.Usec = prio, seq, usec
+	rec.Msg = unescapeKmsg(string(msg))
+	for _, l := range lines[1:] {
+		if k, v, ok := strings.Cut(strings.TrimPrefix(string(l), " "), "="); ok && k != "" {
+			if rec.Dict == nil {
+				rec.Dict = map[string]string{}
+			}
+			rec.Dict[k] = unescapeKmsg(v)
+		}
+	}
+	return rec, true
+}
+
+// unescapeKmsg decodes the \xNN escapes the kernel uses for non-printable bytes.
+func unescapeKmsg(s string) string {
+	if !strings.Contains(s, `\x`) {
+		return s
+	}
+	var b strings.Builder
+	for i := 0; i < len(s); i++ {
+		if s[i] == '\\' && i+3 < len(s) && s[i+1] == 'x' {
+			if v, err := strconv.ParseUint(s[i+2:i+4], 16, 8); err == nil {
+				b.WriteByte(byte(v))
+				i += 3
+				continue
+			}
+		}
+		b.WriteByte(s[i])
+	}
+	return b.String()
+}
+
+// kmsgEntry turns a record into an Entry. Kernel records have the kern
+// facility; programs writing to /dev/kmsg (systemd at boot…) use another
+// one and prefix their name: "systemd[1]: …".
+func kmsgEntry(rec kmsgRecord, t, now time.Time) *Entry {
+	sev := rec.Prio & 7
+	fac := rec.Prio >> 3
+	if fac >= len(facilityNames) {
+		fac = 1 // user
+	}
+	msg := strings.TrimRight(rec.Msg, "\n")
+	if !utf8.ValidString(msg) {
+		msg = strings.ToValidUTF8(msg, "�")
+	}
+	app, pid := "kernel", ""
+	if fac != 0 {
+		if a, p, rest, ok := splitTag(msg); ok {
+			app, pid, msg = a, p, rest
+		}
+	}
+	e := &Entry{
+		Time:       t,
+		Received:   now,
+		Host:       "localhost",
+		App:        app,
+		ProcID:     pid,
+		Facility:   facilityNames[fac],
+		Severity:   severityNames[sev],
+		SevNum:     sev,
+		Message:    msg,
+		Proto:      "kmsg",
+		SourceType: "kernel",
+		Wait:       true,
+	}
+	if s, dev := rec.Dict["SUBSYSTEM"], rec.Dict["DEVICE"]; s != "" || dev != "" {
+		e.Extra = map[string]string{}
+		if s != "" {
+			e.Extra["subsystem"] = s
+		}
+		if dev != "" {
+			e.Extra["device"] = dev
+		}
+	}
+	return e
+}
+
+// splitTag cuts "name[pid]: message" or "name: message".
+func splitTag(msg string) (app, pid, rest string, ok bool) {
+	tag, rest, found := strings.Cut(msg, ": ")
+	if !found || tag == "" || len(tag) > 48 || strings.ContainsAny(tag, " \t") {
+		return "", "", "", false
+	}
+	if i := strings.IndexByte(tag, '['); i > 0 && strings.HasSuffix(tag, "]") {
+		return tag[:i], tag[i+1 : len(tag)-1], rest, true
+	}
+	return tag, "", rest, true
+}
+
+// Configure saves and applies the configuration.
+func (d *Dmesg) Configure(cfg dmesgConfig) error {
+	d.mu.Lock()
+	if d.cfg.Enabled && !cfg.Enabled {
+		d.reset = true
+		d.errCode, d.errDetail = "", ""
+	}
+	d.cfg = cfg
+	d.mu.Unlock()
+	select {
+	case d.wake <- struct{}{}:
+	default:
+	}
+	return writeJSONFile(d.cfgPath, cfg)
+}
+
+// Status is the state shown in Settings > Sources.
+func (d *Dmesg) Status() map[string]any {
+	d.mu.Lock()
+	defer d.mu.Unlock()
+	return map[string]any{
+		"enabled": d.cfg.Enabled,
+		"open":    d.open && d.cfg.Enabled,
+		"device":  d.device,
+		"read":    d.read,
+		"lost":    d.lost,
+		"code":    d.errCode,
+		"error":   d.errDetail,
+	}
+}

dmesg_test.go

@@ -0,0 +1,265 @@
+package main
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+)
+
+func TestParseKmsg(t *testing.T) {
+	rec, ok := parseKmsg([]byte("6,339,5140900,-;NET: Registered protocol family 10\n SUBSYSTEM=net\n DEVICE=n2"))
+	if !ok || rec.Prio != 6 || rec.Seq != 339 || rec.Usec != 5140900 || rec.Msg != "NET: Registered protocol family 10" {
+		t.Fatalf("record %+v", rec)
+	}
+	if rec.Dict["SUBSYSTEM"] != "net" || rec.Dict["DEVICE"] != "n2" {
+		t.Fatalf("dict %+v", rec.Dict)
+	}
+	// Extra header fields (newer kernels: caller id) and a ';' in the message.
+	rec, ok = parseKmsg([]byte("4,12,99,c,caller=T1;a;b \\x1b[31mred\\x5c"))
+	if !ok || rec.Seq != 12 || rec.Msg != "a;b \x1b[31mred\\" {
+		t.Fatalf("record %+v", rec)
+	}
+	for _, bad := range []string{"", "no header", "6,1;short", "x,1,2,-;msg", "6,-1,2,-;msg", "-1,1,2,-;msg"} {
+		if _, ok := parseKmsg([]byte(bad)); ok {
+			t.Errorf("%q parsed", bad)
+		}
+	}
+}
+
+func TestUnescapeKmsg(t *testing.T) {
+	for in, want := range map[string]string{
+		`plain`:       "plain",
+		`a\x0ab`:      "a\nb",
+		`end\x4`:      `end\x4`,
+		`bad\xzz`:     `bad\xzz`,
+		`\x41\x42\x4`: `AB\x4`,
+	} {
+		if got := unescapeKmsg(in); got != want {
+			t.Errorf("%q: got %q, want %q", in, got, want)
+		}
+	}
+}
+
+func TestSplitKmsg(t *testing.T) {
+	got := splitKmsg([]byte("6,1,0,-;one\n SUBSYSTEM=pci\n6,2,0,-;two\n6,3,0,-;three"))
+	if len(got) != 3 || string(got[0]) != "6,1,0,-;one\n SUBSYSTEM=pci" || string(got[2]) != "6,3,0,-;three" {
+		t.Fatalf("got %q", got)
+	}
+}
+
+func TestKmsgEntry(t *testing.T) {
+	boot := time.Date(2026, 10, 4, 6, 0, 0, 0, time.UTC)
+	now := boot.Add(time.Hour)
+	rec, _ := parseKmsg([]byte("3,7,2500000,-;usb 1-1: device descriptor read/64, error -71\n SUBSYSTEM=usb\n DEVICE=c189:1"))
+	e := kmsgEntry(rec, boot.Add(2500*time.Millisecond), now)
+	if e.App != "kernel" || e.Facility != "kern" || e.Severity != "err" || e.SevNum != 3 || e.SourceType != "kernel" || e.Proto != "kmsg" || !e.Wait {
+		t.Fatalf("entry %+v", e)
+	}
+	if e.Message != "usb 1-1: device descriptor read/64, error -71" || e.Extra["subsystem"] != "usb" || e.Extra["device"] != "c189:1" {
+		t.Fatalf("entry %+v", e)
+	}
+	if !e.Time.Equal(boot.Add(2500*time.Millisecond)) || !e.Received.Equal(now) {
+		t.Fatalf("times %v %v", e.Time, e.Received)
+	}
+	// Written by a program (facility daemon = 3<<3 | info).
+	rec, _ = parseKmsg([]byte("30,8,3000000,-;systemd[1]: Started Journal Service."))
+	e = kmsgEntry(rec, now, now)
+	if e.App != "systemd" || e.ProcID != "1" || e.Facility != "daemon" || e.Severity != "info" || e.Message != "Started Journal Service." || e.Extra != nil {
+		t.Fatalf("entry %+v", e)
+	}
+	// Facility out of range.
+	rec, _ = parseKmsg([]byte("255,9,0,-;odd"))
+	if e = kmsgEntry(rec, now, now); e.Facility != "user" || e.Severity != "debug" {
+		t.Fatalf("entry %+v", e)
+	}
+}
+
+// fakeHost writes a kmsg file, a boot id and an uptime in a temp dir.
+func fakeHost(t *testing.T, uptime float64, records ...string) (*Dmesg, *[]*Entry, *sync.Mutex) {
+	t.Helper()
+	dir := t.TempDir()
+	write := func(name, s string) string {
+		p := filepath.Join(dir, name)
+		if err := os.WriteFile(p, []byte(s), 0o644); err != nil {
+			t.Fatal(err)
+		}
+		return p
+	}
+	var mu sync.Mutex
+	var got []*Entry
+	d := NewDmesg(write("kmsg", strings.Join(records, "\n")+"\n"), filepath.Join(dir, "data"), time.Hour, func(e *Entry) {
+		mu.Lock()
+		got = append(got, e)
+		mu.Unlock()
+	})
+	d.bootIDPath = write("boot_id", "b1\n")
+	d.uptimePath = write("uptime", strconv.FormatFloat(uptime, 'f', 2, 64)+" 1234.00\n")
+	d.retry = 20 * time.Millisecond
+	return d, &got, &mu
+}
+
+func TestDmesgBackfillAndResume(t *testing.T) {
+	// Booted 3 h ago: the record at 1 s is outside the 1 h backfill window,
+	// the ones at 2.5 h and 2.9 h are inside.
+	d, got, mu := fakeHost(t, 3*3600,
+		"6,1,1000000,-;Linux version 6.12",
+		"4,2,9000000000,-;eth0: link down",
+		"6,3,10440000000,-;eth0: link up",
+	)
+	s, err := d.start()
+	if err != nil {
+		t.Fatal(err)
+	}
+	<-s.done
+	mu.Lock()
+	if len(*got) != 2 || (*got)[0].Message != "eth0: link down" || (*got)[1].Message != "eth0: link up" {
+		t.Fatalf("got %+v", *got)
+	}
+	if age := time.Since((*got)[1].Time); age < 5*time.Minute || age > 7*time.Minute {
+		t.Fatalf("record dated %v ago", age)
+	}
+	mu.Unlock()
+	d.saveState()
+
+	// Restart with the saved position: same boot, nothing is sent again,
+	// and a new record is.
+	f, _ := os.OpenFile(d.device, os.O_APPEND|os.O_WRONLY, 0)
+	_, _ = f.WriteString("6,6,10500000000,-;eth0: renamed\n")
+	f.Close()
+	d2 := NewDmesg(d.device, filepath.Dir(d.statePath), time.Hour, d.sink)
+	d2.bootIDPath, d2.uptimePath = d.bootIDPath, d.uptimePath
+	if d2.pos != (dmesgPos{BootID: "b1", Seq: 3}) {
+		t.Fatalf("saved position %+v", d2.pos)
+	}
+	s, err = d2.start()
+	if err != nil {
+		t.Fatal(err)
+	}
+	<-s.done
+	mu.Lock()
+	defer mu.Unlock()
+	if len(*got) != 3 || (*got)[2].Message != "eth0: renamed" || (*got)[2].App != "kernel" {
+		t.Fatalf("after restart %+v", (*got)[2:])
+	}
+	if st := d2.Status(); st["lost"] != uint64(2) || st["read"] != uint64(1) {
+		t.Fatalf("status %+v", st) // records 4 and 5 were overwritten
+	}
+}
+
+func TestDmesgNewBoot(t *testing.T) {
+	d, got, mu := fakeHost(t, 60, "6,1,1000000,-;booted")
+	d.pos = dmesgPos{BootID: "old", Seq: 500}
+	s, err := d.start()
+	if err != nil {
+		t.Fatal(err)
+	}
+	<-s.done
+	mu.Lock()
+	defer mu.Unlock()
+	if len(*got) != 1 || d.pos != (dmesgPos{BootID: "b1", Seq: 1}) {
+		t.Fatalf("got %d entries, position %+v", len(*got), d.pos)
+	}
+}
+
+func TestDmesgNoBackfill(t *testing.T) {
+	d, got, mu := fakeHost(t, 60, "6,1,1000000,-;old")
+	d.backfill = 0
+	s, err := d.start()
+	if err != nil {
+		t.Fatal(err)
+	}
+	<-s.done
+	mu.Lock()
+	defer mu.Unlock()
+	if len(*got) != 0 {
+		t.Fatalf("got %+v", *got)
+	}
+}
+
+func TestDmesgRun(t *testing.T) {
+	d, got, mu := fakeHost(t, 60, "6,1,1000000,-;first", "6,2,2000000,-;second")
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan struct{})
+	go func() {
+		d.Run(ctx)
+		close(done)
+	}()
+	if err := d.Configure(dmesgConfig{Enabled: true}); err != nil {
+		t.Fatal(err)
+	}
+	deadline := time.Now().Add(5 * time.Second)
+	for {
+		mu.Lock()
+		n := len(*got)
+		mu.Unlock()
+		if n >= 2 {
+			break
+		}
+		if time.Now().After(deadline) {
+			t.Fatalf("got %d entries", n)
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+	// The file is re-opened after each end of file: still no duplicate.
+	time.Sleep(100 * time.Millisecond)
+	cancel()
+	<-done
+	mu.Lock()
+	defer mu.Unlock()
+	if len(*got) != 2 {
+		t.Fatalf("got %d entries", len(*got))
+	}
+	b, err := os.ReadFile(d.statePath)
+	if err != nil || !strings.Contains(string(b), `"seq": 2`) {
+		t.Fatalf("state %s %v", b, err)
+	}
+	b, _ = os.ReadFile(d.cfgPath)
+	if !strings.Contains(string(b), `"enabled": true`) {
+		t.Fatalf("config %s", b)
+	}
+}
+
+func TestDmesgMissingDevice(t *testing.T) {
+	d := NewDmesg(filepath.Join(t.TempDir(), "kmsg"), t.TempDir(), time.Hour, func(*Entry) {})
+	d.retry = 20 * time.Millisecond
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan struct{})
+	go func() {
+		d.Run(ctx)
+		close(done)
+	}()
+	_ = d.Configure(dmesgConfig{Enabled: true})
+	deadline := time.Now().Add(5 * time.Second)
+	for d.Status()["code"] != "not_found" {
+		if time.Now().After(deadline) {
+			t.Fatalf("status %+v", d.Status())
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+	// Turning it off clears the error.
+	_ = d.Configure(dmesgConfig{Enabled: false})
+	if st := d.Status(); st["code"] != "" || st["enabled"] != false {
+		t.Fatalf("status %+v", st)
+	}
+	cancel()
+	<-done
+}
+
+func TestDmesgPermission(t *testing.T) {
+	if os.Geteuid() == 0 {
+		t.Skip("root reads everything")
+	}
+	p := filepath.Join(t.TempDir(), "kmsg")
+	_ = os.WriteFile(p, []byte("6,1,0,-;x\n"), 0o000)
+	d := NewDmesg(p, t.TempDir(), time.Hour, func(*Entry) {})
+	_, err := d.start()
+	d.fail(err)
+	if st := d.Status(); st["code"] != "permission" {
+		t.Fatalf("status %+v", st)
+	}
+}

docker-compose.yml

@@ -7,29 +7,73 @@ services:
       - victorialogs
       - docker-proxy
     ports:
-      - "${SYSLOG_PORT:-514}:5514/udp"
+      - "${SYSLOG_PORT:-5514}:5514/udp"
-      - "${SYSLOG_PORT:-514}:5514/tcp"
+      - "${SYSLOG_PORT:-5514}:5514/tcp"
-      - "${HTTP_PORT:-8080}:8080"
+      - "${HTTP_PORT:-8989}:8080"
     environment:
       VLOGS_URL: http://victorialogs:9428
-      SYSLOG_PUBLIC_PORT: ${SYSLOG_PORT:-514}     # shown in Settings > Sources (the mapping is set above)
+      SYSLOG_PUBLIC_PORT: ${SYSLOG_PORT:-514}     # le port d'ecoute syslog par defaut (attention aux ports <1024)
       TZ: ${TZ:-Europe/Paris}
-      AUTH_USER: ${AUTH_USER:-}   # leave empty to disable authentication
+      AUTH_MODE: ${AUTH_MODE:-local}   # local (page de connexion, compte ci-dessous) ou oidc
+      AUTH_USER: ${AUTH_USER:-}   # vide = pas d'authentification, on delegue ca au reverse proxy traefik
       AUTH_PASS: ${AUTH_PASS:-}
-      RDNS: ${RDNS:-on}               # replace IP hosts with their DNS name (PTR)
+      AUTH_VIEWER_USER: ${AUTH_VIEWER_USER:-}   # compte en lecture seule (optionnel)
-      DNS_SERVER: ${DNS_SERVER:-}     # e.g. 192.168.1.1 to query your LAN DNS; empty = system resolver
+      AUTH_VIEWER_PASS: ${AUTH_VIEWER_PASS:-}
-      ALLOW_PURGE: ${ALLOW_PURGE:-true}
+      LOGIN_LOGO: ${LOGIN_LOGO:-}     # PNG affiche sur la page de connexion (chemin dans le conteneur, voir volumes)
+      PRESETS_FILE: ${PRESETS_FILE:-} # prereglages de tags (defaut /data/presets.json, voir docs/presets.fr.md)
+      OIDC_ISSUER: ${OIDC_ISSUER:-}
+      OIDC_CLIENT_ID: ${OIDC_CLIENT_ID:-}
+      OIDC_CLIENT_SECRET: ${OIDC_CLIENT_SECRET:-}
+      OIDC_REDIRECT_URL: ${OIDC_REDIRECT_URL:-}
+      OIDC_SCOPES: ${OIDC_SCOPES:-openid profile email}
+      OIDC_ADMIN_GROUP: ${OIDC_ADMIN_GROUP:-}     # vide = tout le monde est admin, sinon les autres sont en lecture seule
+      OIDC_GROUPS_CLAIM: ${OIDC_GROUPS_CLAIM:-groups}
+      SESSION_TTL: ${SESSION_TTL:-${OIDC_SESSION_TTL:-12h}}   # duree de la session (local et oidc)
+      RDNS: ${RDNS:-on}               # resol dns
+      DNS_SERVER: ${DNS_SERVER:-}     # si resolv directe
+      ALLOW_PURGE: ${ALLOW_PURGE:-false}   # true pour autoriser « Supprimer tous les logs »
       EXPORT_MAX: ${EXPORT_MAX:-100000}
-      DOCKER_LOGS: ${DOCKER_LOGS:-on}             # collect the logs of this machine's containers
+      DOCKER_LOGS: ${DOCKER_LOGS:-on}             # collecte des logs des conteneurs Docker
-      DOCKER_HOST: tcp://docker-proxy:2375        # read-only Docker API gateway (below)
+      DOCKER_HOST: tcp://docker-proxy:2375        # lecture seul de l'API Docker
-      DOCKER_BACKFILL: ${DOCKER_BACKFILL:-1h}     # history read from a container seen for the first time
+      DOCKER_BACKFILL: ${DOCKER_BACKFILL:-1h}
+      HOST_LOGS_BACKFILL: ${HOST_LOGS_BACKFILL:-1h}   # historique lu a l'activation des logs systeme de l'hote
+      DMESG_BACKFILL: ${DMESG_BACKFILL:-1h}   # historique lu a l'activation des messages du noyau (dmesg)
+      AUDIT_BACKFILL: ${AUDIT_BACKFILL:-1h}   # historique lu a l'activation de l'audit Linux (auditd)
+      FILES_BACKFILL: ${FILES_BACKFILL:-1h}   # historique lu a l'activation d'un fichier de logs (Reglages > Sources)
+      AUDIT_HOST: ${AUDIT_HOST:-}             # nom d'hote des evenements d'audit (vide : node= des enregistrements, sinon /host/etc/hostname)
+    group_add:
+      - "${HOST_LOGS_GID:-4}"        # groupe autorise a lire les logs de l'hote (4 = adm sur Debian/Ubuntu)
+    # messages du noyau de l'hote (dmesg), source a activer dans Reglages > Sources :
+    # decommenter devices pour donner /dev/kmsg (lecture seule) au conteneur
+    # devices:
+    #   - /dev/kmsg:/dev/kmsg:r
+    # si l'hote a kernel.dmesg_restrict=1 (defaut Debian/Ubuntu), soit sur l'hote :
+    #   sysctl -w kernel.dmesg_restrict=0   (et kernel.dmesg_restrict = 0 dans /etc/sysctl.d/ pour le garder)
+    # soit lancer le conteneur en root avec la capacite SYSLOG (decommenter les deux) :
+    # user: "0:0"
+    # cap_add:
+    #   - SYSLOG
     volumes:
-      - logstream-data:/data         # tags.json, docker.json (container choices)
+      - logstream-data:/data         # tags.json, docker.json (les choix des conteneurs)
+      # logs systeme de l'hote, en lecture seule (source a activer dans Reglages > Sources)
+      - /var/log:/host/var/log:ro                     # journal systemd persistant et fichiers texte
+      - /run/log/journal:/host/run/log/journal:ro     # journal systemd volatile
+      # audit Linux (auditd), source a activer dans Reglages > Sources ; le fichier doit etre
+      # lisible par l'UID 10001 : log_group = adm dans /etc/audit/auditd.conf (voir le README)
+      # - /var/log/audit:/var/log/audit:ro
+      # noms des utilisateurs et de l'hote pour les evenements d'audit (optionnel)
+      # - /etc/passwd:/host/etc/passwd:ro
+      # - /etc/hostname:/host/etc/hostname:ro
+      # fichiers de logs d'autres programmes (Traefik, nginx...), a declarer dans Reglages > Sources > Fichiers de logs
+      # - /var/log/traefik:/logs/traefik:ro
+      # logo de la page de connexion, avec LOGIN_LOGO=/config/logo.png dans .env
+      # - ./logo.png:/config/logo.png:ro
+      # prereglages de tags personnalises, avec PRESETS_FILE=/config/presets.json dans .env
+      # - ./presets.json:/config/presets.json:ro
     labels:
-      logstream.exclude: "true"      # never collect Logstream's own logs
+      logstream.exclude: "true"      # pas de collect des logs logstream
 
   victorialogs:
-    # Pinned version: see "Updating" in the README before changing it
     image: victoriametrics/victoria-logs:v1.52.0
     container_name: logstream-victorialogs
     restart: unless-stopped
@@ -37,7 +81,7 @@ services:
       - -storageDataPath=/vlogs
       - -retentionPeriod=${RETENTION:-30d}
       - -httpListenAddr=:9428
-      - -delete.enable                # required by "Delete all logs" in Settings
+      - -delete.enable                # obliger pour autoriser la purge de la base via l'interface
     volumes:
       - vlogs-data:/vlogs
     ports:
@@ -45,9 +89,7 @@ services:
       - "127.0.0.1:9428:9428"
 
   docker-proxy:
-    # Read-only gateway to the Docker API: Logstream can only list containers,
+    # Docker proxy pour eviter de solliciter directement l'API docker
-    # read their logs, receive events and engine info. Anything else (start,
-    # stop, exec, images, volumes…) is refused.
     image: tecnativa/docker-socket-proxy:v0.5.0
     container_name: logstream-docker-proxy
     restart: unless-stopped
@@ -59,7 +101,7 @@ services:
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock:ro
     labels:
-      logstream.exclude: "true"      # its access log would only echo Logstream's own requests
+      logstream.exclude: "true"      # pour exclure les logs propres de logstream
 
 volumes:
   logstream-data:

docker.go

@@ -62,7 +62,7 @@ type DockerManager struct {
 	hostName   string
 	containers []DockerContainer
 	followers  map[string]*follower // container ID -> running follower
-	checkpoint map[string]time.Time // container ID -> timestamp of the last line read
+	checkpoint map[string]time.Time // container ID -> timestamp of the last line stored
 	dirty      bool
 	connected  bool
 	lastErr    string
@@ -242,7 +242,7 @@ func (m *DockerManager) setError(err error) {
 	m.mu.Lock()
 	defer m.mu.Unlock()
 	if m.lastErr != err.Error() {
-		log.Printf("docker: %v", err)
+		logEvent("docker", 3, "Docker API unreachable: %v", err)
 	}
 	m.connected = false
 	m.lastErr = err.Error()
@@ -290,7 +290,7 @@ func (m *DockerManager) sync(ctx context.Context) {
 	m.mu.Lock()
 	defer m.mu.Unlock()
 	if !m.connected {
-		log.Printf("docker: connected, %d containers", len(list))
+		logEvent("docker", 6, "connected to the Docker API, %d containers", len(list))
 	}
 	m.connected, m.lastErr = true, ""
 	seen := map[string]bool{}
@@ -371,7 +371,7 @@ func (m *DockerManager) follow(ctx context.Context, c DockerContainer) {
 			if ctx.Err() != nil {
 				return
 			}
-			log.Printf("docker %s: %v", c.Name, err)
+			logEvent("docker", 4, "container %s: cannot inspect it: %v", c.Name, err)
 			sleepCtx(ctx, 5*time.Second)
 			continue
 		}
@@ -383,22 +383,25 @@ func (m *DockerManager) follow(ctx context.Context, c DockerContainer) {
 			return
 		}
 		if err != nil && !errors.Is(err, io.EOF) && !errors.Is(err, io.ErrUnexpectedEOF) {
-			log.Printf("docker %s: %v", c.Name, err)
+			logEvent("docker", 4, "container %s: log stream lost, reconnecting: %v", c.Name, err)
 		}
 		sleepCtx(ctx, time.Second)
 	}
 }
 
-func sleepCtx(ctx context.Context, d time.Duration) {
+// sleepCtx waits for d, or returns false if the context ends first.
+func sleepCtx(ctx context.Context, d time.Duration) bool {
 	t := time.NewTimer(d)
 	defer t.Stop()
 	select {
 	case <-ctx.Done():
+		return false
 	case <-t.C:
+		return true
 	}
 }
 
-// since returns where to resume reading: just after the last line read, or
+// since returns where to resume reading: just after the last line stored, or
 // DOCKER_BACKFILL ago for a container seen for the first time.
 func (m *DockerManager) since(id string) time.Time {
 	m.mu.Lock()
@@ -530,8 +533,6 @@ func (m *DockerManager) emit(c DockerContainer, stream string, line []byte) {
 	} else if t, err := time.Parse(time.RFC3339Nano, s); err == nil {
 		ts, s = t, ""
 	}
-	defer m.setCheckpoint(c.ID, ts)
-
 	s = ansiRe.ReplaceAllString(s, "")
 	if !utf8.ValidString(s) {
 		s = strings.ToValidUTF8(s, string(utf8.RuneError))
@@ -575,6 +576,10 @@ func (m *DockerManager) emit(c DockerContainer, stream string, line []byte) {
 			"compose_service": c.Service,
 			"stream":          stream,
 		},
+		// Docker keeps the logs: wait for room in the queue rather than drop the
+		// line, and resume after it only once it is stored.
+		Wait: true,
+		Done: func() { m.setCheckpoint(c.ID, ts) },
 	})
 }
 

docs/presets.fr.md

@@ -0,0 +1,114 @@
+[English](presets.md) · **Français**
+
+# Préréglages de tags de couleur
+
+Dans **Paramètres › Motifs de recherche**, le menu **+ Préréglage…** ajoute d'un clic un groupe de tags de
+couleur tout faits. Les tags ajoutés sont des tags ordinaires : vous pouvez changer leur couleur,
+leur motif ou leurs options, ou les supprimer. Un tag dont le motif est déjà dans la liste n'est
+pas ajouté en double.
+
+La liste vient d'un fichier texte, [`presets.json`](../presets.json), intégré à LogStream. Vous
+pouvez le remplacer par votre propre fichier (voir [Utiliser votre propre fichier](#utiliser-votre-propre-fichier)).
+
+## Préréglages intégrés
+
+### HTTP/HTTPS
+
+Ces préréglages lisent les logs d'accès de nginx et Apache (formats common et combined), Traefik
+(CLF et JSON), Caddy (JSON) et HAProxy (`option httplog`).
+
+| Préréglage | Tags | Ce qui est coloré |
+| --- | --- | --- |
+| Codes HTTP | `HTTP 2xx` vert, `HTTP 3xx` bleu, `HTTP 4xx` orange, `HTTP 5xx` rouge | seulement le code de statut, par exemple `404` dans `"GET /x HTTP/1.1" 404 153`, `"status":404` ou `"DownstreamStatus":404`. Les autres nombres de la ligne (taille, chemin) ne sont pas touchés. |
+| Méthodes HTTP | `GET/HEAD/OPTIONS` gris, `POST/PUT/PATCH` violet, `DELETE` rose | seulement la méthode dans `"GET /chemin` ou `"method":"GET"` (en majuscules uniquement) |
+| Sondes et attaques | `sondes / attaques` | `wp-login.php`, `xmlrpc.php`, `wp-admin`, `phpmyadmin`, `/.env`, `/.git`, `/.aws`, `/cgi-bin/`, `../`, `%2e%2e`, `/etc/passwd`, `<script`, `union select` |
+| Robots et scripts | `robots / scripts` | les mots finissant par `bot` (`Googlebot`, `bingbot`…), `crawler`, `spider`, `curl`, `wget`, `python-requests`, `Go-http-client`, `zgrab`, `masscan`, `nmap`, `sqlmap`, `nikto` |
+| Erreurs TLS/HTTPS et proxy | `erreurs TLS`, `erreurs proxy` | échecs de handshake TLS, certificats expirés ou refusés, `x509:` ; `upstream timed out`, `no live upstreams`, `connect() failed`, `connection refused`, `bad gateway`, `gateway timeout`, `service unavailable` |
+
+### Système
+
+| Préréglage | Tags | Ce qui est coloré |
+| --- | --- | --- |
+| SSH et connexions | `échecs de connexion` rouge, `connexions` vert | sshd/PAM : `Failed password`, `Invalid user`, `authentication failure`, `incorrect password attempts`, `NOT in sudoers`… ; `Accepted publickey`, `session opened for user`, `New session … of user` |
+| Commandes sudo | `commandes sudo` | la commande lancée, par exemple `COMMAND=/usr/bin/apt` |
+| Noyau : OOM, plantages, disques | `mémoire épuisée`, `erreurs noyau` | `Out of memory`, `oom-killer`, `Killed process 4242` ; `Kernel panic`, `BUG:`, `Oops`, `Call Trace`, `segfault at`, `I/O error`, `EXT4-fs error`, `blocked for more than 120 seconds`, `soft lockup` |
+| Services systemd | `services en échec` rouge, `démarrage/arrêt de service` vert | `Failed to start`, `Failed with result`, `Main process exited, code=killed`, `Start request repeated too quickly` ; `Started`, `Stopping`, `Reloaded`, `Reached target` |
+| Pare-feu et fail2ban | `pare-feu` | `[UFW BLOCK]`, `[UFW ALLOW]`, `DROP`, `REJECT`, `Ban 203.0.113.9`, `Unban …`, `Found …` |
+
+### Applications
+
+| Préréglage | Tags | Ce qui est coloré |
+| --- | --- | --- |
+| Docker et conteneurs | `problèmes de conteneur` | `exited with code 137` (codes non nuls seulement), `OOMKilled`, `unhealthy`, `Back-off restarting`, `CrashLoopBackOff`, `container die/kill/oom` |
+| Bases de données | `erreurs base de données` | PostgreSQL et MySQL/MariaDB : `deadlock detected`, `duplicate key`, `too many connections`, `lock wait timeout`, `slow query`, `server has gone away`, `Access denied for user`, `password authentication failed`… |
+
+### Général
+
+| Préréglage | Tags | Ce qui est coloré |
+| --- | --- | --- |
+| Niveaux de log | `fatal / critique` rouge, `info / notice` bleu, `debug / trace` gris, `ok` vert | ces mots en mots entiers, quelle que soit la casse (les tags par défaut `warning` et `error` couvrent le reste) |
+| Adresses IPv4 | `adresses IPv4` | `192.168.1.20`, `203.0.113.9`… Les numéros de version à quatre parties comme `1.2.3.4` sont aussi colorés. |
+
+Quand des tags se chevauchent, celui placé le plus haut dans la liste l'emporte : les
+préréglages ajoutés après les tags par défaut ne les masquent donc jamais.
+
+## Utiliser votre propre fichier
+
+LogStream lit le fichier indiqué par `PRESETS_FILE`, `/data/presets.json` par défaut (dans le
+volume `logstream-data`). S'il n'existe pas, la liste intégrée est utilisée. Le fichier est relu
+à chaque ouverture des Paramètres : pas besoin de redémarrer après une modification.
+
+Avec docker-compose, le plus simple est de garder le fichier à côté de `docker-compose.yml` :
+
+1. Copiez [`presets.json`](../presets.json) depuis ce dépôt et modifiez-le.
+2. Dans `docker-compose.yml`, décommentez la ligne `- ./presets.json:/config/presets.json:ro`.
+3. Dans `.env`, mettez `PRESETS_FILE=/config/presets.json`, puis lancez `docker compose up -d`.
+
+Si le fichier est invalide (erreur JSON, expression régulière ou couleur incorrecte, id en
+double), les Paramètres affichent l'erreur et la liste intégrée est utilisée jusqu'à correction.
+
+## Format du fichier
+
+Le fichier est une liste JSON de groupes. Chaque groupe a un nom et une liste de préréglages ;
+chaque préréglage a un `id`, un nom et ses tags.
+
+```json
+[
+  {
+    "group": { "en": "My apps", "fr": "Mes applis" },
+    "presets": [
+      {
+        "id": "monappli",
+        "name": "Mon appli",
+        "tags": [
+          { "label": "paiement refusé", "color": "#fca5a5", "pattern": "payment (?:failed|refused)" },
+          { "label": "commande", "color": "#86efac", "pattern": "order #\\d+" },
+          { "label": "lent", "color": "#fde68a", "pattern": "SLOW", "regex": false, "caseSensitive": true }
+        ]
+      }
+    ]
+  }
+]
+```
+
+| Champ | Obligatoire | Signification |
+| --- | --- | --- |
+| `group` | oui | nom du groupe dans le menu |
+| `id` | oui | identifiant unique du préréglage |
+| `name` | oui | nom du préréglage dans le menu |
+| `tags[].pattern` | oui | ce qu'il faut colorer : une expression régulière, ou du texte simple avec `"regex": false` |
+| `tags[].color` | oui | couleur de fond, `#rrggbb` |
+| `tags[].label` | non | nom affiché dans la liste des tags à la place du motif |
+| `tags[].regex` | non | `true` par défaut |
+| `tags[].wholeWord` | non | mots entiers seulement, `false` par défaut |
+| `tags[].caseSensitive` | non | respecter la casse, `false` par défaut |
+
+Les noms et libellés sont soit un seul texte pour toutes les langues (`"Mon appli"`), soit un
+texte par langue (`{ "en": "My app", "fr": "Mon appli" }`) ; une langue absente se rabat sur
+l'anglais.
+
+Les expressions régulières doivent fonctionner à la fois dans le navigateur (JavaScript) et en
+Go, qui les vérifie : évitez les assertions arrière `(?<=…)`, avant `(?=…)` et les références
+arrière `\1`. En JSON, chaque barre oblique inverse s'écrit deux fois : `\d` devient `"\\d"`. Un
+groupe nommé `hl`, `(?<hl>…)`, ne colore que cette partie de la correspondance, comme le font
+les préréglages HTTP avec `(?<hl>5\\d\\d)`.

docs/presets.md

@@ -0,0 +1,111 @@
+**English** · [Français](presets.fr.md)
+
+# Color tag presets
+
+In **Settings › Patterns**, the **+ Preset…** menu adds a group of ready-made color tags in one
+click. Added tags are ordinary tags: you can change their color, pattern or options, or delete
+them. A tag whose pattern is already in the list is not added twice.
+
+The list comes from a text file, [`presets.json`](../presets.json), built into LogStream. You
+can replace it with your own file (see [Using your own file](#using-your-own-file)).
+
+## Built-in presets
+
+### HTTP/HTTPS
+
+These presets read access logs from nginx and Apache (common and combined formats), Traefik
+(CLF and JSON), Caddy (JSON) and HAProxy (`option httplog`).
+
+| Preset | Tags | What gets colored |
+| --- | --- | --- |
+| HTTP status codes | `HTTP 2xx` green, `HTTP 3xx` blue, `HTTP 4xx` orange, `HTTP 5xx` red | only the status code, e.g. `404` in `"GET /x HTTP/1.1" 404 153`, `"status":404` or `"DownstreamStatus":404`. Other numbers on the line (size, path) are left alone. |
+| HTTP methods | `GET/HEAD/OPTIONS` grey, `POST/PUT/PATCH` purple, `DELETE` pink | only the method in `"GET /path` or `"method":"GET"` (upper case only) |
+| Probes and attacks | `probes / attacks` | `wp-login.php`, `xmlrpc.php`, `wp-admin`, `phpmyadmin`, `/.env`, `/.git`, `/.aws`, `/cgi-bin/`, `../`, `%2e%2e`, `/etc/passwd`, `<script`, `union select` |
+| Bots and scripts | `bots / scripts` | words ending in `bot` (`Googlebot`, `bingbot`…), `crawler`, `spider`, `curl`, `wget`, `python-requests`, `Go-http-client`, `zgrab`, `masscan`, `nmap`, `sqlmap`, `nikto` |
+| TLS/HTTPS and proxy errors | `TLS errors`, `proxy errors` | TLS handshake failures, expired or rejected certificates, `x509:`; `upstream timed out`, `no live upstreams`, `connect() failed`, `connection refused`, `bad gateway`, `gateway timeout`, `service unavailable` |
+
+### System
+
+| Preset | Tags | What gets colored |
+| --- | --- | --- |
+| SSH and logins | `login failures` red, `logins` green | sshd/PAM: `Failed password`, `Invalid user`, `authentication failure`, `incorrect password attempts`, `NOT in sudoers`…; `Accepted publickey`, `session opened for user`, `New session … of user` |
+| sudo commands | `sudo commands` | the command run, e.g. `COMMAND=/usr/bin/apt` |
+| Kernel: OOM, crashes, disks | `out of memory`, `kernel errors` | `Out of memory`, `oom-killer`, `Killed process 4242`; `Kernel panic`, `BUG:`, `Oops`, `Call Trace`, `segfault at`, `I/O error`, `EXT4-fs error`, `blocked for more than 120 seconds`, `soft lockup` |
+| systemd services | `failed services` red, `service start/stop` green | `Failed to start`, `Failed with result`, `Main process exited, code=killed`, `Start request repeated too quickly`; `Started`, `Stopping`, `Reloaded`, `Reached target` |
+| Firewall and fail2ban | `firewall` | `[UFW BLOCK]`, `[UFW ALLOW]`, `DROP`, `REJECT`, `Ban 203.0.113.9`, `Unban …`, `Found …` |
+
+### Applications
+
+| Preset | Tags | What gets colored |
+| --- | --- | --- |
+| Docker and containers | `container problems` | `exited with code 137` (non-zero codes only), `OOMKilled`, `unhealthy`, `Back-off restarting`, `CrashLoopBackOff`, `container die/kill/oom` |
+| Databases | `database errors` | PostgreSQL and MySQL/MariaDB: `deadlock detected`, `duplicate key`, `too many connections`, `lock wait timeout`, `slow query`, `server has gone away`, `Access denied for user`, `password authentication failed`… |
+
+### General
+
+| Preset | Tags | What gets colored |
+| --- | --- | --- |
+| Log levels | `fatal / critical` red, `info / notice` blue, `debug / trace` grey, `ok` green | these words as whole words, any case (the default `warning` and `error` tags cover the rest) |
+| IPv4 addresses | `IPv4 addresses` | `192.168.1.20`, `203.0.113.9`… Four-part version numbers such as `1.2.3.4` are colored too. |
+
+When tags overlap, the one highest in the tag list wins, so presets added after the default
+tags never hide them.
+
+## Using your own file
+
+LogStream reads the file named by `PRESETS_FILE`, `/data/presets.json` by default (in the
+`logstream-data` volume). When the file does not exist, the built-in list is used. The file is
+read again each time Settings is opened: no restart is needed after an edit.
+
+With docker-compose, the simplest is to keep the file next to `docker-compose.yml`:
+
+1. Copy [`presets.json`](../presets.json) from this repository and edit it.
+2. In `docker-compose.yml`, uncomment the line `- ./presets.json:/config/presets.json:ro`.
+3. In `.env`, set `PRESETS_FILE=/config/presets.json`, then run `docker compose up -d`.
+
+If the file is invalid (JSON error, bad regular expression or color, duplicate id), Settings
+shows the error and the built-in list is used until the file is fixed.
+
+## File format
+
+The file is a JSON list of groups. Each group has a name and a list of presets; each preset has
+an `id`, a name and its tags.
+
+```json
+[
+  {
+    "group": { "en": "My apps", "fr": "Mes applis" },
+    "presets": [
+      {
+        "id": "myapp",
+        "name": "My app",
+        "tags": [
+          { "label": "payment failed", "color": "#fca5a5", "pattern": "payment (?:failed|refused)" },
+          { "label": "order", "color": "#86efac", "pattern": "order #\\d+" },
+          { "label": "slow", "color": "#fde68a", "pattern": "SLOW", "regex": false, "caseSensitive": true }
+        ]
+      }
+    ]
+  }
+]
+```
+
+| Field | Required | Meaning |
+| --- | --- | --- |
+| `group` | yes | name of the group in the menu |
+| `id` | yes | unique identifier of the preset |
+| `name` | yes | name of the preset in the menu |
+| `tags[].pattern` | yes | what to color: a regular expression, or plain text with `"regex": false` |
+| `tags[].color` | yes | background color, `#rrggbb` |
+| `tags[].label` | no | name shown in the tag list instead of the pattern |
+| `tags[].regex` | no | `true` by default |
+| `tags[].wholeWord` | no | only match whole words, `false` by default |
+| `tags[].caseSensitive` | no | match case, `false` by default |
+
+Names and labels are either one text for every language (`"My app"`) or one text per language
+(`{ "en": "My app", "fr": "Mon appli" }`); a missing language falls back to English.
+
+Regular expressions must work both in the browser (JavaScript) and in Go, which checks them:
+avoid look-behind `(?<=…)`, look-ahead `(?=…)` and back-references `\1`. In JSON, every
+backslash is written twice: `\d` becomes `"\\d"`. A group named `hl`, `(?<hl>…)`, colors only
+that part of the match, as the HTTP presets do with `(?<hl>5\\d\\d)`.

fileformats.go

@@ -0,0 +1,689 @@
+package main
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io"
+	"os"
+	"regexp"
+	"strconv"
+	"strings"
+	"time"
+	"unicode/utf8"
+)
+
+// Formats of the "Files" source (Settings > Sources), one per declared file.
+const (
+	fmtAuto    = "auto"    // detected from the last lines of the file
+	fmtText    = "text"    // one line per entry, level and leading timestamp detected
+	fmtJSON    = "json"    // JSON lines: msg/time/level recognized, other keys as fields
+	fmtLogfmt  = "logfmt"  // key=value pairs (Traefik "common" logs, Go programs…)
+	fmtRegex   = "regex"   // regular expression with named groups
+	fmtCLF     = "clf"     // access logs in Common/Combined Log Format (nginx, Apache, Traefik)
+	fmtTraefik = "traefik" // Traefik access log in JSON
+)
+
+var fileFormats = map[string]bool{fmtText: true, fmtJSON: true, fmtLogfmt: true, fmtRegex: true, fmtCLF: true, fmtTraefik: true}
+
+// fileRec is one parsed line (or several, for continuation lines).
+type fileRec struct {
+	t      time.Time
+	dated  bool // t comes from the line
+	sev    int
+	msg    string
+	host   string
+	app    string
+	fields map[string]string
+}
+
+const (
+	maxFileFields   = 64
+	maxFieldValue   = 4096
+	maxFileMsg      = 64 << 10 // message with its continuation lines
+	maxContinuation = 500      // continuation lines added to one entry
+)
+
+// fileParser turns a line of a file into an entry, for one format.
+type fileParser struct {
+	format string
+	re     *regexp.Regexp // fmtRegex
+}
+
+func newFileParser(format, expr string) (*fileParser, error) {
+	if !fileFormats[format] {
+		return nil, fmt.Errorf("unknown format %q", format)
+	}
+	p := &fileParser{format: format}
+	if format == fmtRegex {
+		if strings.TrimSpace(expr) == "" {
+			return nil, fmt.Errorf("the regex format needs a regular expression")
+		}
+		re, err := regexp.Compile(expr)
+		if err != nil {
+			return nil, fmt.Errorf("invalid regular expression: %v", err)
+		}
+		p.re = re
+	}
+	return p, nil
+}
+
+// multiline tells whether lines that do not start an entry are added to the
+// previous one (stack traces…). Structured formats are one entry per line.
+func (p *fileParser) multiline() bool {
+	return p.format == fmtText || p.format == fmtRegex
+}
+
+// continuation reports a line that belongs to the previous entry: indented,
+// or not matching the regular expression.
+func (p *fileParser) continuation(line string) bool {
+	if line != "" && (line[0] == ' ' || line[0] == '\t') {
+		return true
+	}
+	return p.format == fmtRegex && !p.re.MatchString(line)
+}
+
+// parse decodes one line; it never fails (a line in another format is kept as text).
+func (p *fileParser) parse(line string, now time.Time) *fileRec {
+	line = cleanLine(line)
+	switch p.format {
+	case fmtJSON, fmtTraefik:
+		if obj, ok := decodeJSONLine(line); ok {
+			if p.format == fmtTraefik {
+				if r := traefikRec(obj, line, now); r != nil {
+					return r
+				}
+			}
+			return jsonRec(obj, line, now)
+		}
+	case fmtLogfmt:
+		if kv := parseLogfmt(line); len(kv) > 0 {
+			return logfmtRec(kv, line, now)
+		}
+	case fmtRegex:
+		if m := p.re.FindStringSubmatch(line); m != nil {
+			return p.regexRec(m, line, now)
+		}
+	case fmtCLF:
+		if r := clfRec(line, now); r != nil {
+			return r
+		}
+	}
+	return textRec(line, now)
+}
+
+func cleanLine(s string) string {
+	s = strings.TrimRight(s, "\r")
+	s = ansiRe.ReplaceAllString(s, "")
+	if !utf8.ValidString(s) {
+		s = strings.ToValidUTF8(s, "�")
+	}
+	return s
+}
+
+// textRec: the line itself, without its leading timestamp when there is one.
+func textRec(line string, now time.Time) *fileRec {
+	r := &fileRec{t: now, sev: 6, msg: line}
+	if m := leadTimeRe.FindStringSubmatchIndex(line); m != nil {
+		if t, ok := parseLogTime(line[m[2]:m[3]], now); ok {
+			r.t, r.dated = t, true
+			if rest := line[m[1]:]; strings.TrimSpace(rest) != "" {
+				r.msg = rest
+			}
+		}
+	}
+	if n, ok := detectLevel(r.msg); ok {
+		r.sev = n
+	}
+	return r
+}
+
+/* ---------- Timestamps ---------- */
+
+// leadTimeRe: a timestamp at the start of a line, possibly in brackets
+// (ISO 8601 / RFC 3339, 2006/01/02 15:04:05, CLF, syslog "Jan _2 15:04:05").
+var leadTimeRe = regexp.MustCompile(`^\[?(\d{4}[-/]\d{2}[-/]\d{2}[T ]\d{2}:\d{2}:\d{2}(?:[.,]\d+)?(?: ?(?:Z|[+-]\d{2}:?\d{2}))?|\d{2}/[A-Z][a-z]{2}/\d{4}:\d{2}:\d{2}:\d{2} [+-]\d{4}|[A-Z][a-z]{2} +\d{1,2} \d{2}:\d{2}:\d{2})\]?[ \t]*(?:[-|:][ \t]+)?`)
+
+var timeLayouts = []string{
+	time.RFC3339Nano,
+	"2006-01-02T15:04:05Z0700",
+	"2006-01-02T15:04:05",
+	"02/Jan/2006:15:04:05 -0700",
+	"Jan _2 15:04:05",
+	time.RFC1123Z,
+	time.RFC1123,
+	time.UnixDate,
+	time.ANSIC,
+}
+
+// parseLogTime reads the usual log timestamps; times without a zone are local.
+func parseLogTime(s string, now time.Time) (time.Time, bool) {
+	s = strings.TrimSpace(s)
+	if s == "" {
+		return time.Time{}, false
+	}
+	if s[0] >= '0' && s[0] <= '9' && !strings.ContainsAny(s, ":/-") {
+		if whole, _, _ := strings.Cut(s, "."); len(whole) < 9 {
+			return time.Time{}, false // not a Unix time of this century
+		}
+		return epochTime(s)
+	}
+	// 2006/01/02 15:04:05,123 +02:00 -> 2006-01-02T15:04:05.123+02:00
+	if len(s) >= 19 && s[4] == '/' && s[7] == '/' {
+		s = s[:4] + "-" + s[5:7] + "-" + s[8:]
+	}
+	if len(s) >= 19 && s[4] == '-' && s[10] == ' ' {
+		s = s[:10] + "T" + s[11:]
+	}
+	if len(s) > 19 && s[4] == '-' {
+		s = s[:19] + strings.Replace(strings.Replace(s[19:], ",", ".", 1), " ", "", 1)
+	}
+	s = strings.Join(strings.Fields(s), " ") // "Jan  5" -> "Jan 5"
+	for _, layout := range timeLayouts {
+		if layout == "Jan _2 15:04:05" {
+			layout = "Jan 2 15:04:05"
+		}
+		t, err := time.ParseInLocation(layout, s, time.Local)
+		if err != nil {
+			continue
+		}
+		if t.Year() == 0 { // syslog timestamps have no year
+			t = t.AddDate(now.Year(), 0, 0)
+			if t.After(now.Add(24 * time.Hour)) {
+				t = t.AddDate(-1, 0, 0)
+			}
+		}
+		return t, true
+	}
+	return time.Time{}, false
+}
+
+// epochTime reads a Unix time in seconds, milliseconds, microseconds or
+// nanoseconds (guessed from its size), with an optional fraction.
+func epochTime(s string) (time.Time, bool) {
+	f, err := strconv.ParseFloat(s, 64)
+	if err != nil || f <= 0 {
+		return time.Time{}, false
+	}
+	switch {
+	case f < 1e11:
+		return time.Unix(0, int64(f*1e9)), true
+	case f < 1e14:
+		return time.UnixMilli(int64(f)), true
+	case f < 1e17:
+		return time.UnixMicro(int64(f)), true
+	}
+	return time.Unix(0, int64(f)), true
+}
+
+/* ---------- Fields ---------- */
+
+// reservedFields are stored by LogStream itself: fields of the logs with the
+// same name are renamed field_<name>.
+var reservedFields = map[string]bool{
+	"_time": true, "_msg": true, "_stream": true, "_stream_id": true, "received": true, "msg_time": true,
+	"host": true, "host_name": true, "host_ip": true, "app": true, "facility": true, "severity": true,
+	"sevnum": true, "source": true, "proto": true, "procid": true, "sd": true, "source_type": true, "log_file": true,
+}
+
+func fieldName(k string) string {
+	k = strings.Map(func(r rune) rune {
+		if r == '_' || r == '.' || r == '-' || r >= '0' && r <= '9' || r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' {
+			return r
+		}
+		return '_'
+	}, k)
+	if k == "" || reservedFields[k] || k[0] == '_' {
+		k = "field_" + strings.TrimLeft(k, "_")
+	}
+	return k
+}
+
+func (r *fileRec) set(k, v string) {
+	if v == "" {
+		return
+	}
+	if r.fields == nil {
+		r.fields = map[string]string{}
+	}
+	k = fieldName(k)
+	if _, ok := r.fields[k]; !ok && len(r.fields) >= maxFileFields {
+		return
+	}
+	if len(v) > maxFieldValue {
+		v = strings.ToValidUTF8(v[:maxFieldValue], "") + "…"
+	}
+	r.fields[k] = v
+}
+
+/* ---------- JSON lines ---------- */
+
+func decodeJSONLine(line string) (map[string]any, bool) {
+	s := strings.TrimSpace(line)
+	if len(s) < 2 || s[0] != '{' {
+		return nil, false
+	}
+	dec := json.NewDecoder(strings.NewReader(s))
+	dec.UseNumber()
+	var obj map[string]any
+	if dec.Decode(&obj) != nil {
+		return nil, false
+	}
+	return obj, true
+}
+
+var (
+	jsonMsgKeys   = []string{"msg", "message", "@message", "log", "text"}
+	jsonTimeKeys  = []string{"time", "ts", "timestamp", "@timestamp", "@t", "t", "datetime", "date"}
+	jsonLevelKeys = []string{"level", "lvl", "severity", "log.level", "@level", "@l", "loglevel", "levelname"}
+)
+
+// takeString removes the first of keys found in obj and returns it as text.
+func takeString(obj map[string]any, keys []string) (string, bool) {
+	for _, k := range keys {
+		if v, ok := obj[k]; ok {
+			if s := scalarString(v); s != "" {
+				delete(obj, k)
+				return s, true
+			}
+		}
+	}
+	return "", false
+}
+
+func scalarString(v any) string {
+	switch v := v.(type) {
+	case string:
+		return v
+	case json.Number:
+		return v.String()
+	case bool:
+		return strconv.FormatBool(v)
+	}
+	return ""
+}
+
+func jsonRec(obj map[string]any, line string, now time.Time) *fileRec {
+	r := &fileRec{t: now, sev: 6, msg: line}
+	for _, k := range jsonTimeKeys {
+		if t, ok := parseLogTime(scalarString(obj[k]), now); ok {
+			r.t, r.dated = t, true
+			delete(obj, k)
+			break
+		}
+	}
+	lvl, hasLevel := takeString(obj, jsonLevelKeys)
+	if hasLevel {
+		if n, ok := levelNum(lvl); ok {
+			r.sev = n
+		} else if n, err := strconv.Atoi(lvl); err == nil {
+			r.sev = numericLevel(n)
+		} else {
+			r.set("level", lvl)
+		}
+	}
+	if s, ok := takeString(obj, jsonMsgKeys); ok {
+		r.msg = strings.TrimRight(s, "\n")
+	}
+	flattenJSON("", obj, r)
+	if !hasLevel {
+		if n, ok := detectLevel(r.msg); ok {
+			r.sev = n
+		}
+	}
+	return r
+}
+
+// numericLevel maps numeric levels: syslog (0-7) or pino/bunyan (10-60).
+func numericLevel(n int) int {
+	switch {
+	case n >= 0 && n <= 7:
+		return n
+	case n >= 60:
+		return 2
+	case n >= 50:
+		return 3
+	case n >= 40:
+		return 4
+	case n >= 30:
+		return 6
+	}
+	return 7
+}
+
+// flattenJSON stores the remaining keys as fields; nested objects give
+// dotted names, arrays stay in JSON.
+func flattenJSON(prefix string, obj map[string]any, r *fileRec) {
+	for k, v := range obj {
+		name := prefix + k
+		switch v := v.(type) {
+		case map[string]any:
+			flattenJSON(name+".", v, r)
+		case []any:
+			b, _ := json.Marshal(v)
+			r.set(name, string(b))
+		case nil:
+		default:
+			r.set(name, scalarString(v))
+		}
+	}
+}
+
+/* ---------- logfmt ---------- */
+
+// parseLogfmt splits `k=v k2="v 2"` (with \" escapes); words without a value
+// are skipped. Nil when the line has no pair.
+func parseLogfmt(s string) [][2]string {
+	var out [][2]string
+	for i := 0; i < len(s); {
+		for i < len(s) && (s[i] == ' ' || s[i] == '\t') {
+			i++
+		}
+		start := i
+		for i < len(s) && s[i] != '=' && s[i] != ' ' && s[i] != '\t' {
+			i++
+		}
+		if i >= len(s) || s[i] != '=' || i == start {
+			for i < len(s) && s[i] != ' ' && s[i] != '\t' {
+				i++
+			}
+			continue
+		}
+		k := s[start:i]
+		i++
+		var v string
+		if i < len(s) && s[i] == '"' {
+			var b strings.Builder
+			i++
+			for i < len(s) && s[i] != '"' {
+				if s[i] == '\\' && i+1 < len(s) {
+					i++
+					switch s[i] {
+					case 'n':
+						b.WriteByte('\n')
+					case 't':
+						b.WriteByte('\t')
+					default:
+						b.WriteByte(s[i])
+					}
+				} else {
+					b.WriteByte(s[i])
+				}
+				i++
+			}
+			i++ // closing quote
+			v = b.String()
+		} else {
+			end := i
+			for end < len(s) && s[end] != ' ' && s[end] != '\t' {
+				end++
+			}
+			v, i = s[i:end], end
+		}
+		out = append(out, [2]string{k, v})
+	}
+	return out
+}
+
+func logfmtRec(kv [][2]string, line string, now time.Time) *fileRec {
+	obj := make(map[string]any, len(kv))
+	for _, p := range kv {
+		obj[p[0]] = p[1]
+	}
+	return jsonRec(obj, line, now)
+}
+
+/* ---------- Regular expression ---------- */
+
+func (p *fileParser) regexRec(m []string, line string, now time.Time) *fileRec {
+	r := &fileRec{t: now, sev: 6, msg: line}
+	hasLevel := false
+	for i, name := range p.re.SubexpNames() {
+		if i == 0 || name == "" || m[i] == "" {
+			continue
+		}
+		v := m[i]
+		switch strings.ToLower(name) {
+		case "time", "timestamp", "ts":
+			if t, ok := parseLogTime(v, now); ok {
+				r.t, r.dated = t, true
+			}
+		case "level", "severity", "lvl":
+			if n, ok := levelNum(v); ok {
+				r.sev, hasLevel = n, true
+			}
+		case "msg", "message":
+			r.msg = v
+		case "host":
+			r.host = v
+		case "app":
+			r.app = v
+		case "status":
+			r.set(name, v)
+			if n, ok := statusLevel(v); ok && !hasLevel {
+				r.sev, hasLevel = n, true
+			}
+		default:
+			r.set(name, v)
+		}
+	}
+	if !hasLevel {
+		if n, ok := detectLevel(r.msg); ok {
+			r.sev = n
+		}
+	}
+	return r
+}
+
+/* ---------- Access logs ---------- */
+
+// statusLevel: 5xx are errors, 4xx warnings, the rest info.
+func statusLevel(s string) (int, bool) {
+	n, err := strconv.Atoi(s)
+	if err != nil || n < 100 || n > 999 {
+		return 0, false
+	}
+	switch {
+	case n >= 500:
+		return 3, true
+	case n >= 400:
+		return 4, true
+	}
+	return 6, true
+}
+
+// clfRe: Common Log Format, the combined extension (referer, user agent) and
+// the Traefik one (request count, router, server URL, duration).
+var clfRe = regexp.MustCompile(`^(\S+) (\S+) (\S+) \[([^\]]+)\] "((?:[^"\\]|\\.)*)" (\d{3}) (\d+|-)(?: "((?:[^"\\]|\\.)*)" "((?:[^"\\]|\\.)*)")?(?: (\d+) "([^"]*)" "([^"]*)" (\d+)ms)?`)
+
+func clfRec(line string, now time.Time) *fileRec {
+	m := clfRe.FindStringSubmatch(line)
+	if m == nil {
+		return nil
+	}
+	r := &fileRec{t: now, sev: 6, msg: line}
+	if t, ok := parseLogTime(m[4], now); ok {
+		r.t, r.dated = t, true
+	}
+	dash := func(s string) string {
+		if s == "-" {
+			return ""
+		}
+		return s
+	}
+	r.set("client_ip", m[1])
+	r.set("http_user", dash(m[3]))
+	if req := strings.Fields(m[5]); len(req) == 3 {
+		r.set("http_method", req[0])
+		r.set("http_path", req[1])
+		r.set("http_proto", req[2])
+	} else {
+		r.set("http_request", m[5])
+	}
+	r.set("http_status", m[6])
+	r.set("http_size", dash(m[7]))
+	r.set("http_referer", dash(m[8]))
+	r.set("user_agent", dash(m[9]))
+	r.set("router", dash(m[11]))
+	r.set("backend", dash(m[12]))
+	r.set("duration_ms", m[13])
+	r.sev, _ = statusLevel(m[6])
+	return r
+}
+
+// traefikRec reads a Traefik access log line in JSON; nil for the other
+// lines (Traefik's own logs), read as generic JSON.
+func traefikRec(obj map[string]any, line string, now time.Time) *fileRec {
+	status := scalarString(obj["DownstreamStatus"])
+	if status == "" {
+		status = scalarString(obj["OriginStatus"])
+	}
+	method := scalarString(obj["RequestMethod"])
+	if status == "" && method == "" {
+		return nil
+	}
+	r := &fileRec{t: now, sev: 6}
+	if t, ok := parseLogTime(firstNonEmpty(scalarString(obj["StartUTC"]), scalarString(obj["time"])), now); ok {
+		r.t, r.dated = t, true
+	}
+	if n, ok := statusLevel(status); ok {
+		r.sev = n
+	}
+	rename := map[string]string{
+		"ClientHost": "client_ip", "ClientUsername": "http_user", "RequestMethod": "http_method",
+		"RequestHost": "http_host", "RequestPath": "http_path", "RequestProtocol": "http_proto",
+		"DownstreamStatus": "http_status", "DownstreamContentSize": "http_size", "OriginStatus": "origin_status",
+		"RouterName": "router", "ServiceName": "service", "ServiceURL": "backend", "entryPointName": "entrypoint",
+		"request_User-Agent": "user_agent", "request_Referer": "http_referer",
+	}
+	dur := ""
+	if d, err := strconv.ParseFloat(scalarString(obj["Duration"]), 64); err == nil {
+		dur = strconv.FormatFloat(d/1e6, 'f', -1, 64)
+		r.set("duration_ms", dur)
+	}
+	for _, k := range []string{"Duration", "StartUTC", "StartLocal", "time", "level", "msg"} {
+		delete(obj, k)
+	}
+	for k, v := range obj {
+		if to, ok := rename[k]; ok {
+			r.set(to, scalarString(v))
+			delete(obj, k)
+		}
+	}
+	flattenJSON("", obj, r)
+	// "GET example.com/path 404 · 12.5 ms · router@docker"
+	var b strings.Builder
+	b.WriteString(firstNonEmpty(method, "-"))
+	b.WriteByte(' ')
+	b.WriteString(r.fields["http_host"])
+	b.WriteString(r.fields["http_path"])
+	b.WriteByte(' ')
+	b.WriteString(firstNonEmpty(status, "-"))
+	if dur != "" {
+		b.WriteString(" · " + dur + " ms")
+	}
+	if rt := r.fields["router"]; rt != "" {
+		b.WriteString(" · " + rt)
+	}
+	if ip := r.fields["client_ip"]; ip != "" {
+		b.WriteString(" · " + ip)
+	}
+	r.msg = b.String()
+	return r
+}
+
+/* ---------- Detection ---------- */
+
+const detectWindow = 64 << 10
+
+// sampleLines returns the last complete lines of a file (50 at most).
+func sampleLines(path string) ([]string, error) {
+	fh, err := os.Open(path)
+	if err != nil {
+		return nil, err
+	}
+	defer fh.Close()
+	fi, err := fh.Stat()
+	if err != nil {
+		return nil, err
+	}
+	off := fi.Size() - detectWindow
+	if off < 0 {
+		off = 0
+	}
+	buf, err := io.ReadAll(io.NewSectionReader(fh, off, detectWindow))
+	if err != nil {
+		return nil, err
+	}
+	end := bytes.LastIndexByte(buf, '\n')
+	if end < 0 {
+		return nil, nil
+	}
+	buf = buf[:end]
+	if off > 0 { // first line cut by the window
+		if nl := bytes.IndexByte(buf, '\n'); nl >= 0 {
+			buf = buf[nl+1:]
+		} else {
+			return nil, nil
+		}
+	}
+	var lines []string
+	for _, b := range bytes.Split(buf, []byte{'\n'}) {
+		if len(bytes.TrimSpace(b)) > 0 {
+			lines = append(lines, string(bytes.TrimRight(b, "\r")))
+		}
+	}
+	if len(lines) > 50 {
+		lines = lines[len(lines)-50:]
+	}
+	return lines, nil
+}
+
+// logfmtKeys: the pairs have one of the usual logfmt keys.
+func logfmtKeys(kv [][2]string) bool {
+	for _, p := range kv {
+		switch strings.ToLower(p[0]) {
+		case "level", "lvl", "msg", "message", "time", "ts", "t":
+			return true
+		}
+	}
+	return false
+}
+
+// detectFormat guesses the format of a file from some of its lines: the
+// structured format most lines follow, else plain text (a regular
+// expression is never guessed).
+func detectFormat(lines []string) string {
+	var js, traefik, clf, logfmt int
+	for _, l := range lines {
+		l = strings.TrimSpace(l)
+		if obj, ok := decodeJSONLine(l); ok {
+			js++
+			if _, ok := obj["RequestMethod"]; ok {
+				traefik++
+			} else if _, ok := obj["DownstreamStatus"]; ok {
+				traefik++
+			}
+			continue
+		}
+		if clfRe.MatchString(l) {
+			clf++
+			continue
+		}
+		if kv := parseLogfmt(l); len(kv) >= 3 && logfmtKeys(kv) {
+			logfmt++
+		}
+	}
+	most := func(n int) bool { return n > 0 && n*5 >= len(lines)*4 } // 80 %
+	switch {
+	case most(js) && traefik*2 >= js:
+		return fmtTraefik
+	case most(js):
+		return fmtJSON
+	case most(clf):
+		return fmtCLF
+	case most(logfmt):
+		return fmtLogfmt
+	}
+	return fmtText
+}

filesource.go

@@ -0,0 +1,639 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"crypto/rand"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"sync"
+	"time"
+)
+
+// fileSourceDef is one log file declared in Settings > Sources.
+type fileSourceDef struct {
+	ID      string `json:"id"`
+	Name    string `json:"name"`            // app of the entries
+	Path    string `json:"path"`            // the file, as mounted in the container
+	Format  string `json:"format"`          // auto (detected), or text, json, logfmt, regex, clf, traefik
+	Regex   string `json:"regex,omitempty"` // format regex: named groups time, level, msg, host, app…
+	Host    string `json:"host,omitempty"`  // host of the entries (empty: the host name of the machine)
+	Enabled bool   `json:"enabled"`
+}
+
+// filesConfig is saved in /data/files.json.
+type filesConfig struct {
+	Sources []fileSourceDef `json:"sources"`
+}
+
+// filePos is where reading resumes in a file: its inode and an offset before
+// which every entry was sent. Until: end of the history found when the
+// source was turned on; entries before it are kept only when dated within
+// FILES_BACKFILL.
+type filePos struct {
+	Ino   uint64 `json:"ino"`
+	Off   int64  `json:"off"`
+	Until int64  `json:"until,omitempty"`
+}
+
+type filesState struct {
+	Pos     map[string]filePos   `json:"pos"`     // by source id
+	Started map[string]time.Time `json:"started"` // when each source was turned on
+}
+
+type fileSourceStatus struct {
+	Open     bool
+	Read     uint64
+	Detected string // format found when the source is in auto
+	Code     string
+	Error    string
+}
+
+// fileSource is a definition with its parser (nil in auto: detected from
+// the file).
+type fileSource struct {
+	fileSourceDef
+	parser *fileParser
+}
+
+// pendingEntry is an entry waiting for its continuation lines.
+type pendingEntry struct {
+	rec     *fileRec
+	off     int64     // offset of its first line
+	seen    time.Time // when its last line was read
+	history bool      // read from the history found when the source was turned on
+	lines   int
+}
+
+// fileTail is the reading state of one file, owned by the Run goroutine.
+type fileTail struct {
+	readOff int64 // where the next read starts (after the lines of the pending entry)
+	known   bool  // readOff is set
+	pending *pendingEntry
+	auto    *fileParser // format detected from the file, in auto
+}
+
+// Files follows log files of other programs (Traefik, nginx, applications…)
+// mounted in the container, one entry per line in the format detected or
+// chosen for each file. Source type "file".
+type Files struct {
+	sink      func(*Entry)
+	hostName  string
+	backfill  time.Duration
+	cfgPath   string
+	statePath string
+	wake      chan struct{}
+
+	// Owned by the Run goroutine.
+	st    filesState
+	tails map[string]*fileTail
+	dirty bool
+
+	mu       sync.Mutex
+	cfg      []*fileSource
+	resets   map[string]bool // sources to read again from FILES_BACKFILL ago
+	redetect map[string]bool // sources whose format changed
+	status   map[string]*fileSourceStatus
+}
+
+func NewFiles(hostRoot, dataDir string, backfill time.Duration, sink func(*Entry)) *Files {
+	f := &Files{
+		sink:      sink,
+		hostName:  readTrim(filepath.Join(hostRoot, "etc/hostname")),
+		backfill:  backfill,
+		cfgPath:   filepath.Join(dataDir, "files.json"),
+		statePath: filepath.Join(dataDir, "files-state.json"),
+		wake:      make(chan struct{}, 1),
+		tails:     map[string]*fileTail{},
+		resets:    map[string]bool{},
+		redetect:  map[string]bool{},
+		status:    map[string]*fileSourceStatus{},
+	}
+	if f.hostName == "" {
+		f.hostName = "localhost"
+	}
+	var cfg filesConfig
+	if b, err := os.ReadFile(f.cfgPath); err == nil {
+		_ = json.Unmarshal(b, &cfg)
+	}
+	for _, d := range cfg.Sources {
+		src, err := compileFileSource(d)
+		if err != nil {
+			logEvent("files", 4, "file %q ignored: %v", d.Name, err)
+			continue
+		}
+		f.cfg = append(f.cfg, src)
+	}
+	if b, err := os.ReadFile(f.statePath); err == nil {
+		_ = json.Unmarshal(b, &f.st)
+	}
+	if f.st.Pos == nil {
+		f.st.Pos = map[string]filePos{}
+	}
+	if f.st.Started == nil {
+		f.st.Started = map[string]time.Time{}
+	}
+	return f
+}
+
+const maxFileSources = 50
+
+var fileIDRe = regexp.MustCompile(`^[0-9a-f]{1,32}$`)
+
+// checkFilePath: one file, absolute, without wildcards.
+func checkFilePath(p string) (string, error) {
+	p = strings.TrimSpace(p)
+	switch {
+	case !filepath.IsAbs(p):
+		return "", errors.New("the path must be absolute (as mounted in the container)")
+	case strings.ContainsAny(p, "*?["):
+		return "", errors.New("give the path of one file (no wildcards)")
+	}
+	return filepath.Clean(p), nil
+}
+
+// compileFileSource checks a definition and builds its parser.
+func compileFileSource(d fileSourceDef) (*fileSource, error) {
+	d.Name = strings.TrimSpace(d.Name)
+	d.Host = strings.TrimSpace(d.Host)
+	if d.Format == "" {
+		d.Format = fmtAuto
+	}
+	if d.Format != fmtRegex {
+		d.Regex = ""
+	}
+	switch {
+	case !fileIDRe.MatchString(d.ID):
+		return nil, errors.New("invalid id")
+	case d.Name == "" || len(d.Name) > 64:
+		return nil, errors.New("the name is required (64 characters at most)")
+	case len(d.Host) > 253:
+		return nil, fmt.Errorf("%s: host name too long", d.Name)
+	}
+	p, err := checkFilePath(d.Path)
+	if err != nil {
+		return nil, fmt.Errorf("%s: %v", d.Name, err)
+	}
+	d.Path = p
+	src := &fileSource{fileSourceDef: d}
+	if d.Format != fmtAuto {
+		if src.parser, err = newFileParser(d.Format, d.Regex); err != nil {
+			return nil, fmt.Errorf("%s: %v", d.Name, err)
+		}
+	}
+	return src, nil
+}
+
+func newFileID() string {
+	b := make([]byte, 6)
+	_, _ = rand.Read(b)
+	return hex.EncodeToString(b)
+}
+
+// Configure checks, saves and applies the list of files. Files turned off,
+// removed or given another path are read again from FILES_BACKFILL ago the
+// next time they are on.
+func (f *Files) Configure(cfg filesConfig) error {
+	if len(cfg.Sources) > maxFileSources {
+		return fmt.Errorf("%d files at most", maxFileSources)
+	}
+	var srcs []*fileSource
+	ids := map[string]bool{}
+	for _, d := range cfg.Sources {
+		if d.ID == "" {
+			d.ID = newFileID()
+		}
+		if ids[d.ID] {
+			return fmt.Errorf("duplicate id %s", d.ID)
+		}
+		ids[d.ID] = true
+		src, err := compileFileSource(d)
+		if err != nil {
+			return err
+		}
+		srcs = append(srcs, src)
+	}
+	saved := filesConfig{Sources: []fileSourceDef{}}
+	for _, s := range srcs {
+		saved.Sources = append(saved.Sources, s.fileSourceDef)
+	}
+	if err := writeJSONFile(f.cfgPath, saved); err != nil {
+		return err
+	}
+	f.mu.Lock()
+	old := map[string]*fileSource{}
+	for _, s := range f.cfg {
+		old[s.ID] = s
+	}
+	for _, s := range srcs {
+		o := old[s.ID]
+		if o != nil && ((o.Enabled && !s.Enabled) || o.Path != s.Path) {
+			f.resets[s.ID] = true
+			delete(f.status, s.ID)
+		}
+		if o != nil && (o.Format != s.Format || o.Regex != s.Regex) {
+			f.redetect[s.ID] = true
+		}
+		delete(old, s.ID)
+	}
+	for id := range old { // removed
+		f.resets[id] = true
+		delete(f.status, id)
+	}
+	f.cfg = srcs
+	f.mu.Unlock()
+	select {
+	case f.wake <- struct{}{}:
+	default:
+	}
+	return nil
+}
+
+// Status is the state shown in Settings > Sources.
+func (f *Files) Status() map[string]any {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	list := make([]map[string]any, 0, len(f.cfg))
+	for _, s := range f.cfg {
+		m := map[string]any{
+			"id": s.ID, "name": s.Name, "path": s.Path, "format": s.Format, "regex": s.Regex,
+			"host": s.Host, "enabled": s.Enabled, "open": false, "read": 0, "detected": "", "code": "", "error": "",
+		}
+		if st := f.status[s.ID]; st != nil && s.Enabled {
+			m["open"], m["read"], m["detected"], m["code"], m["error"] = st.Open, st.Read, st.Detected, st.Code, st.Error
+		}
+		list = append(list, m)
+	}
+	return map[string]any{"sources": list, "defaultHost": f.hostName}
+}
+
+// Run polls the files every second.
+func (f *Files) Run(ctx context.Context) {
+	tick := time.NewTicker(time.Second)
+	defer tick.Stop()
+	n := 0
+	for {
+		select {
+		case <-ctx.Done():
+			now := time.Now()
+			for id := range f.tails {
+				f.flushPending(id, now)
+			}
+			f.checkpointAll()
+			f.saveState()
+			return
+		case <-tick.C:
+		case <-f.wake:
+		}
+		f.poll(time.Now())
+		if n++; n%5 == 0 {
+			f.saveState()
+		}
+	}
+}
+
+func (f *Files) saveState() {
+	if !f.dirty {
+		return
+	}
+	f.dirty = false
+	if err := writeJSONFile(f.statePath, f.st); err != nil {
+		logEvent("files", 4, "saving positions: %v", err)
+	}
+}
+
+// poll reads what was added to every file that is on.
+func (f *Files) poll(now time.Time) {
+	f.mu.Lock()
+	srcs := f.cfg
+	resets, redetect := f.resets, f.redetect
+	f.resets, f.redetect = map[string]bool{}, map[string]bool{}
+	f.mu.Unlock()
+	for id := range resets {
+		delete(f.st.Pos, id)
+		delete(f.st.Started, id)
+		delete(f.tails, id)
+		f.dirty = true
+	}
+	for id := range redetect {
+		if tl := f.tails[id]; tl != nil {
+			tl.auto = nil
+		}
+	}
+	active := map[string]bool{}
+	for _, s := range srcs {
+		if !s.Enabled {
+			continue
+		}
+		active[s.ID] = true
+		f.setStatus(s.ID, f.pollFile(s, now))
+	}
+	// Files turned off: their position is kept until the next poll forgets it.
+	for id := range f.tails {
+		if !active[id] {
+			delete(f.tails, id)
+		}
+	}
+	f.checkpointAll()
+}
+
+const (
+	maxFileRead       = 4 << 20         // per file and per poll
+	fileContinueDelay = 2 * time.Second // an entry waits this long for continuation lines
+)
+
+func (f *Files) setStatus(id string, err error) {
+	code, detail := "", ""
+	if err != nil {
+		code, detail = errCode(err), err.Error()
+		if errors.Is(err, fs.ErrNotExist) {
+			code = "not_found"
+		}
+	}
+	detected := ""
+	if tl := f.tails[id]; tl != nil && tl.auto != nil {
+		detected = tl.auto.format
+	}
+	f.mu.Lock()
+	st := f.status[id]
+	if st == nil {
+		st = &fileSourceStatus{}
+		f.status[id] = st
+	}
+	changed := code != st.Code
+	st.Open, st.Detected, st.Code, st.Error = err == nil, detected, code, detail
+	name := id
+	for _, s := range f.cfg {
+		if s.ID == id {
+			name = s.Name
+		}
+	}
+	f.mu.Unlock()
+	if changed && code != "" && code != "not_found" {
+		logEvent("files", 4, "cannot read the file %s: %v", name, err)
+	}
+}
+
+func (f *Files) count(id string) {
+	f.mu.Lock()
+	if st := f.status[id]; st != nil {
+		st.Read++
+	}
+	f.mu.Unlock()
+}
+
+// parserOf is the parser of a file: the chosen format, or the one detected
+// from its last lines (nil while the file has none).
+func (f *Files) parserOf(s *fileSource, tl *fileTail, path string) *fileParser {
+	if s.parser != nil {
+		return s.parser
+	}
+	if tl.auto == nil {
+		lines, err := sampleLines(path)
+		if err != nil || len(lines) == 0 {
+			return nil
+		}
+		tl.auto, _ = newFileParser(detectFormat(lines), "")
+	}
+	return tl.auto
+}
+
+func (f *Files) pollFile(s *fileSource, now time.Time) error {
+	path := s.Path
+	first := false
+	if _, ok := f.st.Started[s.ID]; !ok {
+		f.st.Started[s.ID], f.dirty, first = now, true, true
+	}
+	fi, err := os.Stat(path)
+	if err != nil {
+		return err
+	}
+	if !fi.Mode().IsRegular() {
+		return fmt.Errorf("%s is not a regular file", path)
+	}
+	tl := f.tails[s.ID]
+	if tl == nil {
+		tl = &fileTail{}
+		f.tails[s.ID] = tl
+	}
+	ino := inodeOf(fi)
+	pos, known := f.st.Pos[s.ID]
+	if !known {
+		pos = filePos{Ino: ino}
+		if first {
+			// Turned on: the last FILES_BACKFILL of what is already there
+			// (0: only new lines).
+			if f.backfill > 0 {
+				pos.Until = fi.Size()
+			} else {
+				pos.Off = fi.Size()
+			}
+		}
+		f.st.Pos[s.ID], f.dirty = pos, true
+	}
+	if !tl.known {
+		tl.readOff, tl.known = pos.Off, true
+	}
+	if pos.Ino != ino {
+		// Rotated (access.log -> access.log.1): finish the old file, then
+		// read the new one from its start.
+		if old, err := os.Stat(path + ".1"); err == nil && inodeOf(old) == pos.Ino {
+			if p := f.parserOf(s, tl, path+".1"); p != nil {
+				for {
+					next, more, err := f.readFrom(s, tl, p, path+".1", tl.readOff, pos.Until, now)
+					if err != nil || !more {
+						break
+					}
+					tl.readOff = next
+				}
+			}
+		}
+		f.flushPending(s.ID, now)
+		pos, tl.readOff = filePos{Ino: ino}, 0
+		f.st.Pos[s.ID], f.dirty = pos, true
+	} else if fi.Size() < tl.readOff {
+		f.flushPending(s.ID, now) // truncated (copytruncate)
+		pos.Until, tl.readOff = 0, 0
+		f.st.Pos[s.ID], f.dirty = pos, true
+	}
+	if fi.Size() > tl.readOff {
+		p := f.parserOf(s, tl, path)
+		if p == nil {
+			return nil // nothing complete to detect the format from yet
+		}
+		next, _, err := f.readFrom(s, tl, p, path, tl.readOff, pos.Until, now)
+		if err != nil {
+			return err
+		}
+		tl.readOff = next
+	}
+	if pe := tl.pending; pe != nil && now.Sub(pe.seen) >= fileContinueDelay {
+		f.flushPending(s.ID, now)
+	}
+	return nil
+}
+
+// readFrom parses the complete lines written after off and returns the new
+// offset; more tells whether the read window was full.
+func (f *Files) readFrom(s *fileSource, tl *fileTail, p *fileParser, path string, off, until int64, now time.Time) (int64, bool, error) {
+	fh, err := os.Open(path)
+	if err != nil {
+		return off, false, err
+	}
+	defer fh.Close()
+	buf, err := io.ReadAll(io.NewSectionReader(fh, off, maxFileRead))
+	if err != nil {
+		return off, false, err
+	}
+	full := len(buf) == maxFileRead
+	end := bytes.LastIndexByte(buf, '\n')
+	if end < 0 {
+		if !full {
+			return off, false, nil // partial line: wait for its end
+		}
+		end = len(buf) - 1 // line longer than the read window: cut it
+	}
+	start := off
+	for _, b := range bytes.Split(buf[:end], []byte{'\n'}) {
+		f.line(s, tl, p, string(b), start, start < until, now)
+		start += int64(len(b)) + 1
+	}
+	return off + int64(end) + 1, full, nil
+}
+
+// line handles one line: a continuation of the pending entry, or a new one.
+func (f *Files) line(s *fileSource, tl *fileTail, p *fileParser, line string, off int64, history bool, now time.Time) {
+	if strings.TrimSpace(line) == "" {
+		return
+	}
+	if pe := tl.pending; p.multiline() && pe != nil && p.continuation(line) {
+		if pe.lines < maxContinuation && len(pe.rec.msg) < maxFileMsg {
+			pe.rec.msg += "\n" + strings.TrimRight(cleanLine(line), " \t")
+			pe.lines++
+		}
+		pe.seen = now
+		return
+	}
+	f.flushPending(s.ID, now)
+	pe := &pendingEntry{rec: p.parse(line, now), off: off, seen: now, history: history}
+	if p.multiline() {
+		tl.pending = pe
+		return
+	}
+	f.emit(s, pe, now)
+}
+
+func (f *Files) flushPending(id string, now time.Time) {
+	tl := f.tails[id]
+	if tl == nil || tl.pending == nil {
+		return
+	}
+	pe := tl.pending
+	tl.pending = nil
+	f.mu.Lock()
+	var src *fileSource
+	for _, s := range f.cfg {
+		if s.ID == id {
+			src = s
+		}
+	}
+	f.mu.Unlock()
+	if src != nil {
+		f.emit(src, pe, now)
+	}
+}
+
+// emit sends an entry; history entries are kept only when dated within
+// FILES_BACKFILL of when the file was turned on.
+func (f *Files) emit(s *fileSource, pe *pendingEntry, now time.Time) {
+	r := pe.rec
+	if pe.history {
+		notBefore := f.st.Started[s.ID].Add(-f.backfill)
+		if !r.dated || r.t.Before(notBefore) {
+			return
+		}
+	}
+	if len(r.msg) > maxFileMsg {
+		r.msg = strings.ToValidUTF8(r.msg[:maxFileMsg], "") + "…"
+	}
+	extra := map[string]string{"log_file": s.Path}
+	for k, v := range r.fields {
+		extra[k] = v
+	}
+	f.sink(&Entry{
+		Time:       r.t,
+		Received:   now,
+		Host:       firstNonEmpty(r.host, s.Host, f.hostName),
+		App:        firstNonEmpty(r.app, s.Name),
+		SevNum:     r.sev,
+		Severity:   severityNames[r.sev],
+		Message:    r.msg,
+		Proto:      "file",
+		SourceType: "file",
+		Extra:      extra,
+		Wait:       true, // the file keeps the lines: wait for room in the queue
+	})
+	f.count(s.ID)
+}
+
+// checkpointAll saves, for each file, an offset before which every entry is
+// sent: the start of its pending entry, or the read offset.
+func (f *Files) checkpointAll() {
+	for id, tl := range f.tails {
+		pos, ok := f.st.Pos[id]
+		if !ok || !tl.known {
+			continue
+		}
+		off := tl.readOff
+		if pe := tl.pending; pe != nil && pe.off < off {
+			off = pe.off
+		}
+		if off != pos.Off {
+			pos.Off, f.dirty = off, true
+			f.st.Pos[id] = pos
+		}
+	}
+}
+
+// DetectFile tells, for the settings, whether a file can be read and its
+// format, with a line as example.
+func DetectFile(path string) map[string]any {
+	res := map[string]any{"format": "", "sample": "", "code": "", "error": ""}
+	p, err := checkFilePath(path)
+	if err != nil {
+		res["code"], res["error"] = "invalid", err.Error()
+		return res
+	}
+	var lines []string
+	if fi, err2 := os.Stat(p); err2 != nil {
+		err = err2
+	} else if !fi.Mode().IsRegular() {
+		err = fmt.Errorf("%s is not a regular file", p)
+	} else {
+		lines, err = sampleLines(p)
+	}
+	switch {
+	case errors.Is(err, fs.ErrNotExist):
+		res["code"], res["error"] = "not_found", err.Error()
+	case err != nil:
+		res["code"], res["error"] = errCode(err), err.Error()
+	case len(lines) == 0:
+		res["code"] = "empty"
+	default:
+		res["format"] = detectFormat(lines)
+		sample := lines[len(lines)-1]
+		if len(sample) > 300 {
+			sample = strings.ToValidUTF8(sample[:300], "") + "…"
+		}
+		res["sample"] = cleanLine(sample)
+	}
+	return res
+}

filesource_test.go

@@ -0,0 +1,384 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+)
+
+func mustParser(t *testing.T, format, expr string) *fileParser {
+	t.Helper()
+	p, err := newFileParser(format, expr)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return p
+}
+
+func TestParseLogTime(t *testing.T) {
+	now := time.Date(2026, 10, 5, 12, 0, 0, 0, time.UTC)
+	utc := func(s string) time.Time { t, _ := time.Parse(time.RFC3339Nano, s); return t }
+	tests := []struct {
+		in   string
+		want time.Time
+	}{
+		{"2026-10-05T10:00:00.123Z", utc("2026-10-05T10:00:00.123Z")},
+		{"2026-10-05T10:00:00+02:00", utc("2026-10-05T08:00:00Z")},
+		{"2026-10-05 10:00:00,250 +0200", utc("2026-10-05T08:00:00.25Z")},
+		{"2026/10/05 10:00:00Z", utc("2026-10-05T10:00:00Z")},
+		{"05/Oct/2026:10:00:00 +0200", utc("2026-10-05T08:00:00Z")},
+		{"1759572000", time.Unix(1759572000, 0)},
+		{"1759572000123", time.UnixMilli(1759572000123)},
+	}
+	for _, tt := range tests {
+		got, ok := parseLogTime(tt.in, now)
+		if !ok || !got.Equal(tt.want) {
+			t.Errorf("%q: got %v %v, want %v", tt.in, got, ok, tt.want)
+		}
+	}
+	// Syslog timestamps: no year, the last one before tomorrow.
+	if got, ok := parseLogTime("Dec 31 23:00:00", now); !ok || got.Year() != 2025 {
+		t.Errorf("Dec 31: %v %v", got, ok)
+	}
+	if got, ok := parseLogTime("Oct  5 11:00:00", now); !ok || got.Year() != 2026 || got.Day() != 5 {
+		t.Errorf("Oct 5: %v %v", got, ok)
+	}
+	for _, bad := range []string{"", "hello", "200", "12345"} {
+		if _, ok := parseLogTime(bad, now); ok {
+			t.Errorf("%q parsed", bad)
+		}
+	}
+}
+
+func TestParseText(t *testing.T) {
+	now := time.Now()
+	p := mustParser(t, fmtText, "")
+	r := p.parse("2026-10-05 10:00:00.5 ERROR db: connection refused", now)
+	if !r.dated || r.msg != "ERROR db: connection refused" || r.sev != 3 {
+		t.Errorf("got %+v", r)
+	}
+	r = p.parse("[2026-10-05T10:00:00Z] - started", now)
+	if !r.dated || r.msg != "started" || r.sev != 6 {
+		t.Errorf("got %+v", r)
+	}
+	r = p.parse("\x1b[33mWARN\x1b[0m no timestamp here", now)
+	if r.dated || r.msg != "WARN no timestamp here" || r.sev != 4 || !r.t.Equal(now) {
+		t.Errorf("got %+v", r)
+	}
+}
+
+func TestParseJSON(t *testing.T) {
+	now := time.Now()
+	p := mustParser(t, fmtJSON, "")
+	r := p.parse(`{"time":"2026-10-05T10:00:00Z","level":"warn","msg":"slow query","host":"db.example","req":{"id":7,"tags":["a"]},"ok":true}`, now)
+	if !r.dated || r.sev != 4 || r.msg != "slow query" {
+		t.Errorf("got %+v", r)
+	}
+	for k, want := range map[string]string{"field_host": "db.example", "req.id": "7", "req.tags": `["a"]`, "ok": "true"} {
+		if r.fields[k] != want {
+			t.Errorf("%s = %q, want %q", k, r.fields[k], want)
+		}
+	}
+	// pino: numeric level, epoch in ms.
+	r = p.parse(`{"level":50,"time":1759572000123,"msg":"boom"}`, now)
+	if r.sev != 3 || !r.t.Equal(time.UnixMilli(1759572000123)) {
+		t.Errorf("got %+v", r)
+	}
+	// No message key: the whole line; not JSON: text.
+	line := `{"event":"login","user":"bob"}`
+	if r = p.parse(line, now); r.msg != line || r.fields["user"] != "bob" {
+		t.Errorf("got %+v", r)
+	}
+	if r = p.parse("plain ERROR line", now); r.msg != "plain ERROR line" || r.sev != 3 {
+		t.Errorf("got %+v", r)
+	}
+}
+
+func TestParseLogfmt(t *testing.T) {
+	p := mustParser(t, fmtLogfmt, "")
+	r := p.parse(`time="2026-10-05T10:00:00Z" level=error msg="Cannot connect: \"docker\" down" providerName=docker flag`, time.Now())
+	if !r.dated || r.sev != 3 || r.msg != `Cannot connect: "docker" down` || r.fields["providerName"] != "docker" {
+		t.Errorf("got %+v", r)
+	}
+}
+
+func TestParseRegex(t *testing.T) {
+	if _, err := newFileParser(fmtRegex, "("); err == nil {
+		t.Error("invalid regex accepted")
+	}
+	if _, err := newFileParser(fmtRegex, ""); err == nil {
+		t.Error("empty regex accepted")
+	}
+	p := mustParser(t, fmtRegex, `^(?P<time>\S+ \S+) \[(?P<level>\w+)\] (?P<module>[\w.]+): (?P<msg>.*)$`)
+	r := p.parse("2026-10-05 10:00:00 [warning] app.db: pool exhausted", time.Now())
+	if !r.dated || r.sev != 4 || r.msg != "pool exhausted" || r.fields["module"] != "app.db" {
+		t.Errorf("got %+v", r)
+	}
+	if !p.continuation("Traceback (most recent call last):") || p.continuation("2026-10-05 10:00:01 [info] a: b") {
+		t.Error("continuation")
+	}
+}
+
+func TestParseCLF(t *testing.T) {
+	p := mustParser(t, fmtCLF, "")
+	r := p.parse(`203.0.113.9 - - [05/Oct/2026:10:00:00 +0200] "GET /api/x?y=1 HTTP/1.1" 502 157 "-" "curl/8.0" 4211 "api@docker" "http://172.18.0.5:8080" 31ms`, time.Now())
+	if !r.dated || r.sev != 3 || !strings.HasPrefix(r.msg, "203.0.113.9 - - [") {
+		t.Errorf("got %+v", r)
+	}
+	for k, want := range map[string]string{"client_ip": "203.0.113.9", "http_method": "GET", "http_path": "/api/x?y=1", "http_status": "502",
+		"user_agent": "curl/8.0", "router": "api@docker", "backend": "http://172.18.0.5:8080", "duration_ms": "31"} {
+		if r.fields[k] != want {
+			t.Errorf("%s = %q, want %q", k, r.fields[k], want)
+		}
+	}
+	if _, ok := r.fields["http_referer"]; ok {
+		t.Error("empty referer kept")
+	}
+	// nginx combined, 404.
+	r = p.parse(`10.0.0.1 - alice [05/Oct/2026:10:00:00 +0000] "POST /login HTTP/2.0" 404 0 "https://example.com/" "Mozilla/5.0"`, time.Now())
+	if r.sev != 4 || r.fields["http_user"] != "alice" || r.fields["http_referer"] != "https://example.com/" {
+		t.Errorf("got %+v", r)
+	}
+}
+
+func TestParseTraefikJSON(t *testing.T) {
+	p := mustParser(t, fmtTraefik, "")
+	r := p.parse(`{"ClientHost":"203.0.113.9","DownstreamStatus":200,"Duration":12500000,"RequestHost":"app.example.com","RequestMethod":"GET","RequestPath":"/","RequestProtocol":"HTTP/2.0","RouterName":"app@docker","StartUTC":"2026-10-05T10:00:00.1Z","entryPointName":"websecure","level":"info","msg":"","time":"2026-10-05T10:00:00Z","TLSVersion":"1.3"}`, time.Now())
+	if !r.dated || r.sev != 6 || r.msg != "GET app.example.com/ 200 · 12.5 ms · app@docker · 203.0.113.9" {
+		t.Errorf("got %+v", r)
+	}
+	for k, want := range map[string]string{"entrypoint": "websecure", "http_status": "200", "TLSVersion": "1.3", "duration_ms": "12.5"} {
+		if r.fields[k] != want {
+			t.Errorf("%s = %q, want %q", k, r.fields[k], want)
+		}
+	}
+	// Traefik's own logs in the same file: generic JSON.
+	r = p.parse(`{"level":"error","msg":"Provider error","time":"2026-10-05T10:00:00Z"}`, time.Now())
+	if r.sev != 3 || r.msg != "Provider error" {
+		t.Errorf("got %+v", r)
+	}
+}
+
+func TestDetectFormat(t *testing.T) {
+	tests := map[string][]string{
+		fmtJSON:    {`{"level":"info","msg":"a"}`, `{"level":"warn","msg":"b"}`},
+		fmtTraefik: {`{"RequestMethod":"GET","DownstreamStatus":200}`, `{"level":"info","msg":"Configuration loaded"}`},
+		fmtCLF:     {`10.0.0.1 - - [05/Oct/2026:10:00:00 +0000] "GET / HTTP/1.1" 200 12 "-" "curl"`},
+		fmtLogfmt:  {`time="2026-10-05T10:00:00Z" level=info msg="Starting provider" providerName=docker`},
+		fmtText:    {`2026-10-05 10:00:00 INFO started`, `user=bob logged in`},
+	}
+	for want, lines := range tests {
+		if got := detectFormat(lines); got != want {
+			t.Errorf("%q: got %s, want %s", lines, got, want)
+		}
+	}
+	// A few lines of another format do not change the answer (80 %).
+	lines := []string{`{"a":1}`, `{"a":2}`, `{"a":3}`, `{"a":4}`, `not json`}
+	if got := detectFormat(lines); got != fmtJSON {
+		t.Errorf("got %s", got)
+	}
+	lines = append(lines, `still not json`)
+	if got := detectFormat(lines); got != fmtText {
+		t.Errorf("got %s", got)
+	}
+}
+
+func TestDetectFile(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "access.log")
+	appendFile(t, path, `10.0.0.1 - - [05/Oct/2026:10:00:00 +0000] "GET / HTTP/1.1" 200 12 "-" "curl"`+"\n"+`partial`)
+	res := DetectFile(path)
+	if res["format"] != fmtCLF || !strings.HasPrefix(res["sample"].(string), "10.0.0.1 - -") || res["code"] != "" {
+		t.Errorf("got %+v", res)
+	}
+	for p, code := range map[string]string{
+		filepath.Join(dir, "nope.log"): "not_found", "relative.log": "invalid", filepath.Join(dir, "*.log"): "invalid", dir: "read",
+	} {
+		if res := DetectFile(p); res["code"] != code {
+			t.Errorf("%s: got %+v, want %s", p, res, code)
+		}
+	}
+	empty := filepath.Join(dir, "empty.log")
+	appendFile(t, empty, "")
+	if res := DetectFile(empty); res["code"] != "empty" {
+		t.Errorf("got %+v", res)
+	}
+}
+
+func TestFileSourceValidation(t *testing.T) {
+	f := NewFiles(t.TempDir(), t.TempDir(), time.Hour, func(*Entry) {})
+	bad := []fileSourceDef{
+		{Name: "", Path: "/logs/a.log"},
+		{Name: "a", Path: "relative.log"},
+		{Name: "a", Path: "/logs/*.log"},
+		{Name: "a", Path: "/logs/a.log", Format: "xml"},
+		{Name: "a", Path: "/logs/a.log", Format: fmtRegex, Regex: "(?P<x"},
+	}
+	for _, d := range bad {
+		if err := f.Configure(filesConfig{Sources: []fileSourceDef{d}}); err == nil {
+			t.Errorf("%+v accepted", d)
+		}
+	}
+	if err := f.Configure(filesConfig{Sources: []fileSourceDef{{Name: " traefik ", Path: "/logs/traefik/access.log"}}}); err != nil {
+		t.Fatal(err)
+	}
+	src := f.Status()["sources"].([]map[string]any)
+	if len(src) != 1 || src[0]["name"] != "traefik" || src[0]["id"] == "" || src[0]["format"] != fmtAuto {
+		t.Errorf("got %+v", src)
+	}
+}
+
+type fileSink struct {
+	mu sync.Mutex
+	e  []*Entry
+}
+
+func (s *fileSink) add(e *Entry) { s.mu.Lock(); s.e = append(s.e, e); s.mu.Unlock() }
+func (s *fileSink) take() []*Entry {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	e := s.e
+	s.e = nil
+	return e
+}
+
+func appendFile(t *testing.T, path, s string) {
+	t.Helper()
+	fh, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, err := fh.WriteString(s); err != nil {
+		t.Fatal(err)
+	}
+	fh.Close()
+}
+
+func msgs(es []*Entry) []string {
+	var out []string
+	for _, e := range es {
+		out = append(out, e.Message)
+	}
+	return out
+}
+
+func TestFilesFollow(t *testing.T) {
+	dir, data := t.TempDir(), t.TempDir()
+	now := time.Now()
+	old := now.Add(-3 * time.Hour).UTC().Format(time.RFC3339)
+	recent := now.Add(-10 * time.Minute).UTC().Format(time.RFC3339)
+	app := filepath.Join(dir, "app.log")
+	appendFile(t, app, old+" INFO too old\n"+recent+" ERROR recent\n  at line 1\nundated history line\n")
+
+	sink := &fileSink{}
+	f := NewFiles(dir, data, time.Hour, sink.add)
+	if err := f.Configure(filesConfig{Sources: []fileSourceDef{{Name: "myapp", Path: app, Host: "web1", Enabled: true}}}); err != nil {
+		t.Fatal(err)
+	}
+	f.poll(now)
+	f.poll(now.Add(3 * time.Second)) // the pending entry waited for continuation lines
+	got := sink.take()
+	if len(got) != 1 || got[0].Message != "ERROR recent\n  at line 1" || got[0].SevNum != 3 || got[0].App != "myapp" ||
+		got[0].Host != "web1" || got[0].SourceType != "file" || got[0].Extra["log_file"] != app {
+		t.Fatalf("history: got %q %+v", msgs(got), got)
+	}
+	if st := f.Status()["sources"].([]map[string]any)[0]; st["detected"] != fmtText || st["open"] != true {
+		t.Errorf("status %+v", st)
+	}
+
+	appendFile(t, app, "plain new line\n\tcontinued\n")
+	f.poll(now.Add(4 * time.Second))
+	appendFile(t, app, "after\n")
+	f.poll(now.Add(5 * time.Second))
+	f.poll(now.Add(8 * time.Second))
+	if got = sink.take(); strings.Join(msgs(got), "|") != "plain new line\n\tcontinued|after" {
+		t.Fatalf("new lines: got %q", msgs(got))
+	}
+
+	appendFile(t, app, "before rotation\n")
+	if err := os.Rename(app, app+".1"); err != nil {
+		t.Fatal(err)
+	}
+	appendFile(t, app, "after rotation\n")
+	f.poll(now.Add(9 * time.Second))
+	f.poll(now.Add(12 * time.Second))
+	if got = sink.take(); strings.Join(msgs(got), "|") != "before rotation|after rotation" {
+		t.Fatalf("rotation: got %q", msgs(got))
+	}
+
+	// Restart: resumes where it stopped, nothing sent twice.
+	f.checkpointAll()
+	f.dirty = true
+	f.saveState()
+	appendFile(t, app, "while stopped\n")
+	f2 := NewFiles(dir, data, time.Hour, sink.add)
+	f2.poll(now.Add(13 * time.Second))
+	f2.poll(now.Add(16 * time.Second))
+	if got = sink.take(); strings.Join(msgs(got), "|") != "while stopped" {
+		t.Fatalf("restart: got %q", msgs(got))
+	}
+
+	// Truncated in place (copytruncate).
+	os.Truncate(app, 0)
+	appendFile(t, app, "x\n")
+	f2.poll(now.Add(17 * time.Second))
+	f2.poll(now.Add(20 * time.Second))
+	if got = sink.take(); strings.Join(msgs(got), "|") != "x" {
+		t.Fatalf("truncate: got %q", msgs(got))
+	}
+}
+
+func TestFilesAutoAndErrors(t *testing.T) {
+	dir, data := t.TempDir(), t.TempDir()
+	path := filepath.Join(dir, "access.json")
+	sink := &fileSink{}
+	f := NewFiles(dir, data, 0, sink.add) // FILES_BACKFILL=0: only new lines
+	cfg := filesConfig{Sources: []fileSourceDef{
+		{ID: "a1", Name: "api", Path: path, Enabled: true}, // auto
+		{ID: "b2", Name: "missing", Path: filepath.Join(dir, "nope.log"), Format: fmtText, Enabled: true},
+	}}
+	if err := f.Configure(cfg); err != nil {
+		t.Fatal(err)
+	}
+	now := time.Now()
+	appendFile(t, path, "")
+	f.poll(now) // empty file: format not known yet
+	appendFile(t, path, `{"msg":"one","level":"error"}`+"\n"+`{"msg":"two"}`+"\n"+`{"msg":"partial`)
+	f.poll(now.Add(time.Second))
+	got := sink.take()
+	if strings.Join(msgs(got), "|") != "one|two" || got[0].SevNum != 3 {
+		t.Fatalf("got %q", msgs(got))
+	}
+	src := f.Status()["sources"].([]map[string]any)
+	if src[0]["detected"] != fmtJSON || src[0]["open"] != true || src[1]["code"] != "not_found" {
+		t.Errorf("status %+v", src)
+	}
+
+	// Forced format: the JSON lines are read as text.
+	cfg.Sources[0].Format = fmtText
+	f.Configure(cfg)
+	appendFile(t, path, `"}`+"\n")
+	f.poll(now.Add(2 * time.Second))
+	f.poll(now.Add(5 * time.Second))
+	if got = sink.take(); strings.Join(msgs(got), "|") != `{"msg":"partial"}` {
+		t.Fatalf("forced: got %q", msgs(got))
+	}
+
+	// Turned off then on: starts over (only new lines here).
+	cfg.Sources[0].Enabled = false
+	f.Configure(cfg)
+	f.poll(now.Add(6 * time.Second))
+	appendFile(t, path, "while off\n")
+	cfg.Sources[0].Enabled = true
+	f.Configure(cfg)
+	f.poll(now.Add(7 * time.Second))
+	appendFile(t, path, "three\n")
+	f.poll(now.Add(8 * time.Second))
+	f.poll(now.Add(11 * time.Second))
+	if got = sink.take(); strings.Join(msgs(got), "|") != "three" {
+		t.Fatalf("re-enabled: got %q", msgs(got))
+	}
+}

guard.go

@@ -0,0 +1,122 @@
+package main
+
+import (
+	"context"
+	"crypto/sha256"
+	"encoding/base64"
+	"io/fs"
+	"net/http"
+	"net/url"
+	"regexp"
+	"strings"
+)
+
+// Request guards shared by every auth mode: security headers, a cross-site
+// request check, and the read-only role.
+
+type viewerKey struct{}
+
+// asViewer marks the request as made by a read-only user.
+func asViewer(r *http.Request) *http.Request {
+	return r.WithContext(context.WithValue(r.Context(), viewerKey{}, true))
+}
+
+func isViewer(r *http.Request) bool {
+	v, _ := r.Context().Value(viewerKey{}).(bool)
+	return v
+}
+
+func roleName(viewer bool) string {
+	if viewer {
+		return "viewer"
+	}
+	return "admin"
+}
+
+func isSafeMethod(m string) bool {
+	return m == http.MethodGet || m == http.MethodHead || m == http.MethodOptions
+}
+
+// readOnly refuses the API calls that change something (tags, sources, purge)
+// to read-only users. Without authentication everyone is admin.
+func readOnly(next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if isViewer(r) && !isSafeMethod(r.Method) && strings.HasPrefix(r.URL.Path, "/api/") {
+			writeErr(w, http.StatusForbidden, &codedError{code: "read_only", msg: "read-only account: changes are reserved to administrators"})
+			return
+		}
+		next.ServeHTTP(w, r)
+	})
+}
+
+// crossSite tells whether a request that changes something comes from another
+// site (a form or script on a third-party page), using the headers browsers
+// add; tools such as curl send neither and are let through.
+func crossSite(r *http.Request) bool {
+	switch r.Header.Get("Sec-Fetch-Site") {
+	case "same-origin", "none":
+		return false
+	case "":
+	default: // same-site, cross-site
+		return true
+	}
+	o := r.Header.Get("Origin")
+	if o == "" {
+		return false
+	}
+	u, err := url.Parse(o)
+	return err != nil || !strings.EqualFold(u.Host, r.Host)
+}
+
+// secure adds the security headers to every answer and refuses cross-site
+// changes. Without authentication it also answers /auth/me, so the UI can
+// warn that anyone on the network has full access.
+func secure(next http.Handler, csp string, authOn bool) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		h := w.Header()
+		h.Set("X-Content-Type-Options", "nosniff")
+		h.Set("X-Frame-Options", "DENY")
+		h.Set("Referrer-Policy", "same-origin")
+		h.Set("Content-Security-Policy", csp)
+		if !isSafeMethod(r.Method) && crossSite(r) {
+			writeErr(w, http.StatusForbidden, &codedError{code: "cross_site", msg: "cross-site request refused"})
+			return
+		}
+		if !authOn && r.URL.Path == "/auth/me" {
+			writeJSON(w, http.StatusOK, map[string]string{"mode": "none", "role": "admin"})
+			return
+		}
+		next.ServeHTTP(w, r)
+	})
+}
+
+var inlineScript = regexp.MustCompile(`(?s)<script>(.*?)</script>`)
+
+// contentSecurityPolicy allows the UI's own files, the inline scripts of the
+// embedded pages (by hash) and the optional Bunny Fonts.
+func contentSecurityPolicy(static fs.FS) string {
+	scripts := []string{"'self'"}
+	for _, page := range []string{"index.html", "login.html"} {
+		b, err := fs.ReadFile(static, page)
+		if err != nil {
+			continue
+		}
+		for _, m := range inlineScript.FindAllSubmatch(b, -1) {
+			sum := sha256.Sum256(m[1])
+			scripts = append(scripts, "'sha256-"+base64.StdEncoding.EncodeToString(sum[:])+"'")
+		}
+	}
+	return strings.Join([]string{
+		"default-src 'self'",
+		"script-src " + strings.Join(scripts, " "),
+		// Inline style attributes carry the tag and project colors.
+		"style-src 'self' 'unsafe-inline' https://fonts.bunny.net",
+		"font-src 'self' https://fonts.bunny.net",
+		"img-src 'self' data:",
+		"connect-src 'self'",
+		"object-src 'none'",
+		"base-uri 'none'",
+		"form-action 'self'",
+		"frame-ancestors 'none'",
+	}, "; ")
+}

guard_test.go

@@ -0,0 +1,168 @@
+package main
+
+import (
+	"errors"
+	"io/fs"
+	"net"
+	"net/http"
+	"net/http/cookiejar"
+	"net/http/httptest"
+	"os"
+	"strings"
+	"testing"
+	"time"
+)
+
+var echo = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("app " + r.URL.Path)) })
+
+func TestSecureHeadersAndCrossSite(t *testing.T) {
+	h := secure(echo, "default-src 'self'", false)
+	cases := []struct {
+		method string
+		hdr    map[string]string
+		want   int
+	}{
+		{"GET", map[string]string{"Sec-Fetch-Site": "cross-site"}, 200}, // reading is fine
+		{"POST", nil, 200}, // curl, scripts
+		{"POST", map[string]string{"Sec-Fetch-Site": "same-origin"}, 200},
+		{"POST", map[string]string{"Sec-Fetch-Site": "cross-site"}, 403},
+		{"DELETE", map[string]string{"Sec-Fetch-Site": "same-site"}, 403},
+		{"POST", map[string]string{"Origin": "http://logs.lan:8080"}, 200},
+		{"POST", map[string]string{"Origin": "https://evil.example"}, 403},
+		{"PUT", map[string]string{"Origin": "null"}, 403},
+	}
+	for _, c := range cases {
+		r := httptest.NewRequest(c.method, "http://logs.lan:8080/api/purge", nil)
+		for k, v := range c.hdr {
+			r.Header.Set(k, v)
+		}
+		rec := httptest.NewRecorder()
+		h.ServeHTTP(rec, r)
+		if rec.Code != c.want {
+			t.Errorf("%s %v: %d, want %d", c.method, c.hdr, rec.Code, c.want)
+		}
+		if rec.Header().Get("Content-Security-Policy") == "" || rec.Header().Get("X-Frame-Options") != "DENY" {
+			t.Errorf("%s %v: security headers missing", c.method, c.hdr)
+		}
+	}
+
+	rec := httptest.NewRecorder()
+	h.ServeHTTP(rec, httptest.NewRequest("GET", "/auth/me", nil))
+	if !strings.Contains(rec.Body.String(), `"mode":"none"`) || !strings.Contains(rec.Body.String(), `"role":"admin"`) {
+		t.Errorf("/auth/me without auth: %s", rec.Body)
+	}
+	rec = httptest.NewRecorder()
+	secure(echo, "", true).ServeHTTP(rec, httptest.NewRequest("GET", "/auth/me", nil))
+	if rec.Body.String() != "app /auth/me" {
+		t.Errorf("/auth/me with auth answered by the guard: %s", rec.Body)
+	}
+}
+
+func TestContentSecurityPolicyHashesInlineScripts(t *testing.T) {
+	static, _ := fs.Sub(webFS, "web")
+	csp := contentSecurityPolicy(static)
+	// index.html and login.html each have inline scripts.
+	if n := strings.Count(csp, "'sha256-"); n < 3 {
+		t.Errorf("%d script hashes in %q", n, csp)
+	}
+	for _, want := range []string{"frame-ancestors 'none'", "connect-src 'self'", "https://fonts.bunny.net"} {
+		if !strings.Contains(csp, want) {
+			t.Errorf("CSP lacks %q", want)
+		}
+	}
+}
+
+func TestLocalViewerIsReadOnly(t *testing.T) {
+	loginFailDelay = 0
+	h, err := newAuth(authConfig{mode: "local", user: "admin", pass: "pw", viewerUser: "guest", viewerPass: "ro", dataDir: t.TempDir()}, readOnly(echo))
+	if err != nil {
+		t.Fatal(err)
+	}
+	jar, _ := cookiejar.New(nil)
+	c := &http.Client{Jar: jar, Transport: hosts{"app.test": h}, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
+	app := "http://app.test"
+
+	login(t, c, app, "guest", "ro", "/")
+	if code, body := get(t, c, app+"/auth/me"); code != 200 || !strings.Contains(body, `"role":"viewer"`) {
+		t.Fatalf("me: %d %s", code, body)
+	}
+	if code, _ := get(t, c, app+"/api/logs"); code != 200 {
+		t.Errorf("viewer reading logs: %d", code)
+	}
+	res, err := c.Post(app+"/api/purge", "application/json", strings.NewReader(`{"confirm":"PURGE"}`))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if res.StatusCode != http.StatusForbidden {
+		t.Errorf("viewer purge: %d, want 403", res.StatusCode)
+	}
+
+	// The admin account still changes things, also through Basic auth.
+	req, _ := http.NewRequest("PUT", app+"/api/syslog", strings.NewReader("{}"))
+	req.SetBasicAuth("admin", "pw")
+	if res, _ := (&http.Client{Transport: hosts{"app.test": h}}).Do(req); res.StatusCode != 200 {
+		t.Errorf("admin change: %d", res.StatusCode)
+	}
+	req, _ = http.NewRequest("PUT", app+"/api/syslog", strings.NewReader("{}"))
+	req.SetBasicAuth("guest", "ro")
+	if res, _ := (&http.Client{Transport: hosts{"app.test": h}}).Do(req); res.StatusCode != http.StatusForbidden {
+		t.Errorf("viewer change through Basic auth: %d", res.StatusCode)
+	}
+}
+
+func TestOIDCAdminGroup(t *testing.T) {
+	for _, tc := range []struct {
+		groups any
+		role   string
+	}{
+		{[]any{"staff", "/logstream-admins"}, "admin"},
+		{[]any{"staff"}, "viewer"},
+		{nil, "viewer"},
+	} {
+		idp := newFakeIdP(t)
+		idp.claims = func(c map[string]any) {
+			if tc.groups != nil {
+				c["groups"] = tc.groups
+			}
+		}
+		h, err := newAuth(authConfig{
+			mode: "oidc", issuer: "http://idp.test/realm", clientID: "logstream", clientSecret: "s3cret",
+			redirectURL: "http://app.test/auth/callback", dataDir: t.TempDir(), adminGroup: "logstream-admins",
+		}, readOnly(echo))
+		if err != nil {
+			t.Fatal(err)
+		}
+		netw := hosts{"app.test": h, "idp.test": idp.mux}
+		h.(*OIDC).client.Transport = netw
+		jar, _ := cookiejar.New(nil)
+		c := &http.Client{Jar: jar, Transport: netw}
+		get(t, c, "http://app.test/")
+		if _, body := get(t, c, "http://app.test/auth/me"); !strings.Contains(body, `"role":"`+tc.role+`"`) {
+			t.Errorf("groups %v: %s, want role %s", tc.groups, body, tc.role)
+		}
+	}
+}
+
+func TestHasGroup(t *testing.T) {
+	if !hasGroup("ops logstream-admins", "/logstream-admins") || !hasGroup([]any{"a", "b"}, "b") || hasGroup(42, "b") {
+		t.Error("hasGroup")
+	}
+}
+
+func TestTCPIdleTimeout(t *testing.T) {
+	a, b := net.Pipe()
+	defer b.Close()
+	c := idleConn{a, 30 * time.Millisecond}
+	go func() { _, _ = b.Write([]byte("x")) }()
+	buf := make([]byte, 1)
+	if _, err := c.Read(buf); err != nil {
+		t.Fatal(err)
+	}
+	start := time.Now()
+	if _, err := c.Read(buf); !errors.Is(err, os.ErrDeadlineExceeded) {
+		t.Fatalf("silent connection: %v, want a deadline error", err)
+	}
+	if time.Since(start) > time.Second {
+		t.Error("deadline not applied")
+	}
+}

histogram_test.go

@@ -153,7 +153,7 @@ func TestHistogramHandler(t *testing.T) {
 {"_time":"2000-01-01T00:00:00Z","severity":"info","hits":"9"}
 `, recent.Format(time.RFC3339), recent.Format(time.RFC3339), older.Format(time.RFC3339))
 	}}
-	store := NewStore("http://vl", 10, 10, time.Second)
+	store := NewStore("http://vl", 10, 10, time.Second, nil)
 	store.streamClient.Transport = vl
 	a := &API{store: store}
 
@@ -207,7 +207,7 @@ func TestHistogramDayInParis(t *testing.T) {
 {"_time":"2026-09-21T20:00:00Z","severity":"info","hits":"100"}
 `
 	}}
-	store := NewStore("http://vl", 10, 10, time.Second)
+	store := NewStore("http://vl", 10, 10, time.Second, nil)
 	store.streamClient.Transport = vl
 	a := &API{store: store}
 	from := time.Date(2026, 9, 10, 0, 0, 0, 0, time.UTC)

hostlogs.go

@@ -0,0 +1,457 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"io"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+	"unicode/utf8"
+)
+
+// hostLogsConfig is saved in /data/hostlogs.json and edited in Settings > Sources.
+type hostLogsConfig struct {
+	Enabled bool `json:"enabled"`
+}
+
+// hostPos is where reading resumes in a journal file (by file ID) or a text
+// file (by name, with its inode to detect rotations). Off -1: archived file
+// read to the end.
+type hostPos struct {
+	Off int64  `json:"off"`
+	Ino uint64 `json:"ino,omitempty"`
+}
+
+// HostLogs collects the system logs of the machine hosting the stack: the
+// systemd journal when there is one, else the text files of /var/log. The
+// host directories are mounted read-only under root (/host by default).
+type HostLogs struct {
+	root      string
+	sink      func(*Entry)
+	backfill  time.Duration
+	cfgPath   string
+	statePath string
+	wake      chan struct{}
+
+	// Owned by the Run goroutine.
+	pos     map[string]hostPos
+	dirty   bool
+	started bool // a first scan was done since enabling: new files are read from their start
+
+	mu         sync.Mutex
+	cfg        hostLogsConfig
+	reset      bool
+	mode       string // "journal", "files" or "" (nothing found)
+	files      int
+	read       uint64
+	compressed uint64
+	errCode    string
+	errDetail  string
+}
+
+func NewHostLogs(root, dataDir string, backfill time.Duration, sink func(*Entry)) *HostLogs {
+	h := &HostLogs{
+		root:      root,
+		sink:      sink,
+		backfill:  backfill,
+		cfgPath:   filepath.Join(dataDir, "hostlogs.json"),
+		statePath: filepath.Join(dataDir, "hostlogs-state.json"),
+		wake:      make(chan struct{}, 1),
+		pos:       map[string]hostPos{},
+	}
+	if b, err := os.ReadFile(h.cfgPath); err == nil {
+		_ = json.Unmarshal(b, &h.cfg)
+	}
+	if b, err := os.ReadFile(h.statePath); err == nil {
+		_ = json.Unmarshal(b, &h.pos)
+		h.started = len(h.pos) > 0
+	}
+	return h
+}
+
+// Run polls the host logs every second while the source is enabled.
+func (h *HostLogs) Run(ctx context.Context) {
+	tick := time.NewTicker(time.Second)
+	defer tick.Stop()
+	n := 0
+	for {
+		select {
+		case <-ctx.Done():
+			h.saveState()
+			return
+		case <-tick.C:
+		case <-h.wake:
+		}
+		h.mu.Lock()
+		enabled, reset := h.cfg.Enabled, h.reset
+		h.reset = false
+		h.mu.Unlock()
+		if reset {
+			// Disabled then enabled again: start over from HOST_LOGS_BACKFILL ago
+			// rather than reading everything written in between.
+			h.pos, h.started, h.dirty = map[string]hostPos{}, false, true
+		}
+		if enabled {
+			h.scan(time.Now())
+		}
+		if n++; n%5 == 0 || reset {
+			h.saveState()
+		}
+	}
+}
+
+func (h *HostLogs) saveState() {
+	if !h.dirty {
+		return
+	}
+	h.dirty = false
+	if err := writeJSONFile(h.statePath, h.pos); err != nil {
+		logEvent("host", 4, "saving positions: %v", err)
+	}
+}
+
+func (h *HostLogs) setStatus(mode string, files int, code, detail string) {
+	h.mu.Lock()
+	changed := code != h.errCode
+	h.mode, h.files, h.errCode, h.errDetail = mode, files, code, detail
+	h.mu.Unlock()
+	if changed && code != "" && code != "empty" {
+		logEvent("host", 4, "cannot read the host system logs (%s): %s", code, detail)
+	}
+}
+
+// scan reads what was added since the previous poll.
+func (h *HostLogs) scan(now time.Time) {
+	notBefore := time.Time{}
+	if !h.started {
+		notBefore = now.Add(-h.backfill)
+	}
+	defer func() { h.started = true }()
+
+	var journals []string
+	for _, dir := range []string{"var/log/journal", "run/log/journal"} {
+		base := filepath.Join(h.root, dir)
+		for _, pat := range []string{"*.journal", "*/*.journal"} {
+			m, _ := filepath.Glob(filepath.Join(base, pat))
+			journals = append(journals, m...)
+		}
+	}
+	if len(journals) > 0 {
+		h.scanJournals(journals, notBefore)
+		return
+	}
+	h.scanFiles(notBefore.IsZero())
+}
+
+func (h *HostLogs) scanJournals(paths []string, notBefore time.Time) {
+	seen := map[string]bool{}
+	var firstErr error
+	for _, p := range paths {
+		f, err := os.Open(p)
+		if err != nil {
+			firstErr = keepFirst(firstErr, err)
+			continue
+		}
+		hdr, err := readJournalHeader(f)
+		f.Close()
+		if err != nil {
+			continue // file being created, or not a journal
+		}
+		key := "j:" + hdr.fileID
+		seen[key] = true
+		pos, known := h.pos[key]
+		if known && pos.Off < 0 {
+			continue
+		}
+		from, nb := pos.Off, time.Time{}
+		if !known {
+			if hdr.archived && !notBefore.IsZero() && time.UnixMicro(int64(hdr.tailRealtimeUS)).Before(notBefore) {
+				h.pos[key], h.dirty = hostPos{Off: -1}, true // archived before the backfill window
+				continue
+			}
+			nb = notBefore
+		}
+		next, hdr, err := readJournal(p, from, nb, h.emitJournal)
+		if err != nil {
+			firstErr = keepFirst(firstErr, err)
+			continue
+		}
+		if hdr.archived {
+			next = -1
+		}
+		if !known || next != pos.Off {
+			h.pos[key], h.dirty = hostPos{Off: next}, true
+		}
+	}
+	h.prune("j:", seen)
+	code, detail := "", ""
+	if firstErr != nil && len(seen) == 0 {
+		code, detail = errCode(firstErr), firstErr.Error()
+	}
+	h.setStatus("journal", len(seen), code, detail)
+}
+
+func keepFirst(first, err error) error {
+	if first != nil {
+		return first
+	}
+	return err
+}
+
+func errCode(err error) string {
+	if errors.Is(err, fs.ErrPermission) {
+		return "permission"
+	}
+	return "read"
+}
+
+func (h *HostLogs) prune(prefix string, seen map[string]bool) {
+	for k := range h.pos {
+		if strings.HasPrefix(k, prefix) && !seen[k] {
+			delete(h.pos, k)
+			h.dirty = true
+		}
+	}
+}
+
+// hostLogFile reports the classic text logs of /var/log (rotated and
+// compressed copies are left out).
+func hostLogFile(name string) bool {
+	return name == "syslog" || name == "messages" || strings.HasSuffix(name, ".log")
+}
+
+const maxHostRead = 4 << 20 // per file and per poll
+
+// scanFiles follows the text files of /var/log, for hosts without journald.
+// Files present at the first scan are read from their end (only new lines).
+func (h *HostLogs) scanFiles(fromStart bool) {
+	dir := filepath.Join(h.root, "var/log")
+	ents, err := os.ReadDir(dir)
+	if err != nil {
+		code := errCode(err)
+		if errors.Is(err, fs.ErrNotExist) {
+			code = "not_mounted"
+		}
+		h.setStatus("", 0, code, err.Error())
+		return
+	}
+	seen := map[string]bool{}
+	var firstErr error
+	for _, de := range ents {
+		if !de.Type().IsRegular() || !hostLogFile(de.Name()) {
+			continue
+		}
+		name := de.Name()
+		key := "f:" + name
+		fi, err := de.Info()
+		if err != nil {
+			continue
+		}
+		var ino uint64
+		if st, ok := fi.Sys().(*syscall.Stat_t); ok {
+			ino = uint64(st.Ino)
+		}
+		pos, known := h.pos[key]
+		switch {
+		case !known && !fromStart:
+			pos = hostPos{Off: fi.Size(), Ino: ino}
+		case !known || pos.Ino != ino || fi.Size() < pos.Off:
+			pos = hostPos{Off: 0, Ino: ino} // new, rotated or truncated file
+		}
+		if fi.Size() > pos.Off {
+			off, err := h.readFile(filepath.Join(dir, name), name, pos.Off)
+			if err != nil {
+				firstErr = keepFirst(firstErr, err)
+				continue // not readable: not counted as followed
+			}
+			pos.Off = off
+		}
+		seen[key] = true
+		if old, ok := h.pos[key]; !ok || old != pos {
+			h.pos[key], h.dirty = pos, true
+		}
+	}
+	h.prune("f:", seen)
+	code, detail := "", ""
+	if firstErr != nil && len(seen) == 0 {
+		code, detail = errCode(firstErr), firstErr.Error()
+	}
+	mode := "files"
+	if len(seen) == 0 && code == "" {
+		mode, code = "", "empty"
+	}
+	h.setStatus(mode, len(seen), code, detail)
+}
+
+// readFile sends the complete lines written after off and returns the new offset.
+func (h *HostLogs) readFile(path, name string, off int64) (int64, error) {
+	f, err := os.Open(path)
+	if err != nil {
+		return off, err
+	}
+	defer f.Close()
+	buf, err := io.ReadAll(io.NewSectionReader(f, off, maxHostRead))
+	if err != nil {
+		return off, err
+	}
+	end := bytes.LastIndexByte(buf, '\n')
+	if end < 0 {
+		if len(buf) < maxHostRead {
+			return off, nil // partial line: wait for its end
+		}
+		end = len(buf) - 1 // line longer than the read window: cut it
+	}
+	now := time.Now()
+	fac := fileFacility(name)
+	for _, line := range bytes.Split(buf[:end+1], []byte{'\n'}) {
+		if len(bytes.TrimSpace(line)) == 0 {
+			continue
+		}
+		e := ParseSyslog(line, "", "file", now)
+		if e.Host == "" {
+			e.Host = "localhost"
+		}
+		if n, ok := detectLevel(e.Message); ok {
+			e.SevNum, e.Severity = n, severityNames[n]
+		} else {
+			e.SevNum, e.Severity = 6, "info"
+		}
+		if fac >= 0 {
+			e.Facility = facilityNames[fac]
+		}
+		e.SourceType = "host"
+		e.Extra = map[string]string{"log_file": "/var/log/" + name}
+		e.Wait = true
+		h.sink(e)
+		h.count(1, 0)
+	}
+	return off + int64(end) + 1, nil
+}
+
+// fileFacility guesses the facility from the usual Debian/RHEL file names.
+func fileFacility(name string) int {
+	switch strings.TrimSuffix(name, ".log") {
+	case "kern":
+		return 0
+	case "mail", "maillog":
+		return 2
+	case "daemon":
+		return 3
+	case "auth", "secure":
+		return 4
+	case "cron":
+		return 9
+	}
+	return -1
+}
+
+func (h *HostLogs) count(read, compressed uint64) {
+	h.mu.Lock()
+	h.read += read
+	h.compressed += compressed
+	h.mu.Unlock()
+}
+
+// emitJournal turns a journal entry into an Entry.
+func (h *HostLogs) emitJournal(je *journalEntry) {
+	f := je.Fields
+	msg := f["MESSAGE"]
+	if msg == "" && je.Compressed > 0 {
+		msg = "(compressed journal entry: read it with journalctl)"
+	}
+	h.count(1, uint64(je.Compressed))
+	if strings.TrimSpace(msg) == "" {
+		return
+	}
+	if !utf8.ValidString(msg) {
+		msg = strings.ToValidUTF8(msg, "�")
+	}
+	sev := 6
+	if n, err := strconv.Atoi(f["PRIORITY"]); err == nil && n >= 0 && n <= 7 {
+		sev = n
+	}
+	fac := 1 // user
+	switch {
+	case f["_TRANSPORT"] == "kernel":
+		fac = 0
+	case f["_SYSTEMD_UNIT"] != "":
+		fac = 3 // daemon
+	}
+	if n, err := strconv.Atoi(f["SYSLOG_FACILITY"]); err == nil && n >= 0 && n < len(facilityNames) {
+		fac = n
+	}
+	app := firstNonEmpty(f["SYSLOG_IDENTIFIER"], f["_COMM"], strings.TrimSuffix(f["_SYSTEMD_UNIT"], ".service"))
+	host := firstNonEmpty(f["_HOSTNAME"], "localhost")
+	h.sink(&Entry{
+		Time:       je.Realtime,
+		Received:   time.Now(),
+		Host:       host,
+		App:        app,
+		ProcID:     firstNonEmpty(f["SYSLOG_PID"], f["_PID"]),
+		Facility:   facilityNames[fac],
+		Severity:   severityNames[sev],
+		SevNum:     sev,
+		Message:    strings.TrimRight(msg, "\n"),
+		Proto:      "journal",
+		SourceType: "host",
+		Extra:      map[string]string{"unit": f["_SYSTEMD_UNIT"]},
+		Wait:       true,
+	})
+}
+
+func firstNonEmpty(vals ...string) string {
+	for _, v := range vals {
+		if v != "" {
+			return v
+		}
+	}
+	return ""
+}
+
+// Configure saves and applies the configuration.
+func (h *HostLogs) Configure(cfg hostLogsConfig) error {
+	h.mu.Lock()
+	if h.cfg.Enabled && !cfg.Enabled {
+		h.reset = true
+		h.mode, h.files, h.errCode, h.errDetail = "", 0, "", ""
+	}
+	h.cfg = cfg
+	h.mu.Unlock()
+	select {
+	case h.wake <- struct{}{}:
+	default:
+	}
+	return writeJSONFile(h.cfgPath, cfg)
+}
+
+// Status is the state shown in Settings > Sources.
+func (h *HostLogs) Status() map[string]any {
+	h.mu.Lock()
+	defer h.mu.Unlock()
+	dirs := []string{}
+	for _, d := range []string{"var/log/journal", "run/log/journal", "var/log"} {
+		if _, err := os.Stat(filepath.Join(h.root, d)); err == nil {
+			dirs = append(dirs, "/"+d)
+		}
+	}
+	sort.Strings(dirs)
+	return map[string]any{
+		"enabled":    h.cfg.Enabled,
+		"mode":       h.mode,
+		"files":      h.files,
+		"read":       h.read,
+		"compressed": h.compressed,
+		"code":       h.errCode,
+		"error":      h.errDetail,
+		"mounted":    dirs,
+	}
+}

hostlogs_test.go

@@ -0,0 +1,206 @@
+package main
+
+import (
+	"encoding/binary"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+// fakeJournal builds a minimal journal file: header, data objects, entries.
+type fakeJournal struct {
+	compact bool
+	buf     []byte
+	seq     uint64
+	tail    uint64
+}
+
+func newFakeJournal(compact bool) *fakeJournal {
+	j := &fakeJournal{compact: compact, buf: make([]byte, 272)}
+	copy(j.buf, jSignature)
+	if compact {
+		binary.LittleEndian.PutUint32(j.buf[12:], jIncompatCompact)
+	}
+	j.buf[16] = 1 // online
+	copy(j.buf[24:40], "0123456789abcdef")
+	binary.LittleEndian.PutUint64(j.buf[88:], 272)
+	return j
+}
+
+func (j *fakeJournal) object(typ, flags byte, body []byte) uint64 {
+	for len(j.buf)%8 != 0 {
+		j.buf = append(j.buf, 0)
+	}
+	off := uint64(len(j.buf))
+	h := make([]byte, jObjHeaderSize)
+	h[0], h[1] = typ, flags
+	binary.LittleEndian.PutUint64(h[8:], uint64(jObjHeaderSize+len(body)))
+	j.buf = append(append(j.buf, h...), body...)
+	j.tail = off
+	binary.LittleEndian.PutUint64(j.buf[136:], off)
+	return off
+}
+
+func (j *fakeJournal) data(field string, flags byte) uint64 {
+	n := 48
+	if j.compact {
+		n = 56
+	}
+	return j.object(jObjData, flags, append(make([]byte, n), field...))
+}
+
+// entry appends an entry; linked=false leaves it unfinished (tail seqnum not updated).
+func (j *fakeJournal) entry(t time.Time, linked bool, fields ...string) {
+	var items []byte
+	for _, f := range fields {
+		flags := byte(0)
+		if strings.HasPrefix(f, "!") { // compressed data object
+			f, flags = f[1:], 4
+		}
+		off := j.data(f, flags)
+		if j.compact {
+			items = binary.LittleEndian.AppendUint32(items, uint32(off))
+		} else {
+			items = binary.LittleEndian.AppendUint64(items, off)
+			items = binary.LittleEndian.AppendUint64(items, 0)
+		}
+	}
+	j.seq++
+	body := make([]byte, 48)
+	binary.LittleEndian.PutUint64(body[0:], j.seq)
+	binary.LittleEndian.PutUint64(body[8:], uint64(t.UnixMicro()))
+	j.object(jObjEntry, 0, append(body, items...))
+	if linked {
+		binary.LittleEndian.PutUint64(j.buf[160:], j.seq)
+		binary.LittleEndian.PutUint64(j.buf[192:], uint64(t.UnixMicro()))
+	}
+}
+
+func (j *fakeJournal) write(t *testing.T, path string) {
+	t.Helper()
+	if err := os.WriteFile(path, j.buf, 0o644); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func TestReadJournal(t *testing.T) {
+	for _, compact := range []bool{false, true} {
+		path := filepath.Join(t.TempDir(), "system.journal")
+		now := time.Now()
+		j := newFakeJournal(compact)
+		j.entry(now.Add(-2*time.Hour), true, "MESSAGE=too old", "PRIORITY=6")
+		j.entry(now.Add(-time.Minute), true, "MESSAGE=Started cron.", "PRIORITY=5", "SYSLOG_IDENTIFIER=systemd", "_HOSTNAME=srv1", "_PID=1")
+		j.write(t, path)
+
+		var got []*journalEntry
+		emit := func(e *journalEntry) { got = append(got, e) }
+		next, _, err := readJournal(path, 0, now.Add(-time.Hour), emit)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if len(got) != 1 || got[0].Fields["MESSAGE"] != "Started cron." || got[0].Fields["_HOSTNAME"] != "srv1" {
+			t.Fatalf("compact=%v: got %+v", compact, got)
+		}
+
+		// A new complete entry and one still being written.
+		j.entry(now, true, "MESSAGE=second", "!MESSAGE=big")
+		j.entry(now, false, "MESSAGE=unfinished")
+		j.write(t, path)
+		got = nil
+		next2, _, err := readJournal(path, next, time.Time{}, emit)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if len(got) != 1 || got[0].Fields["MESSAGE"] != "second" || got[0].Compressed != 1 {
+			t.Fatalf("compact=%v: resumed read got %d entries", compact, len(got))
+		}
+
+		// Once linked, the unfinished entry is read from where reading stopped.
+		binary.LittleEndian.PutUint64(j.buf[160:], j.seq)
+		j.write(t, path)
+		got = nil
+		if _, _, err := readJournal(path, next2, time.Time{}, emit); err != nil {
+			t.Fatal(err)
+		}
+		if len(got) != 1 || got[0].Fields["MESSAGE"] != "unfinished" {
+			t.Fatalf("compact=%v: unfinished entry got %+v", compact, got)
+		}
+	}
+}
+
+func TestHostLogsJournal(t *testing.T) {
+	root := t.TempDir()
+	dir := filepath.Join(root, "var/log/journal/machine")
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	j := newFakeJournal(true)
+	j.entry(time.Now(), true, "MESSAGE=Accepted publickey", "PRIORITY=6", "SYSLOG_FACILITY=10", "SYSLOG_IDENTIFIER=sshd", "_PID=42", "_HOSTNAME=srv1", "_SYSTEMD_UNIT=ssh.service")
+	j.entry(time.Now(), true, "MESSAGE=oops", "PRIORITY=3", "_TRANSPORT=kernel", "_HOSTNAME=srv1")
+	j.write(t, filepath.Join(dir, "system.journal"))
+
+	var got []*Entry
+	h := NewHostLogs(root, t.TempDir(), time.Hour, func(e *Entry) { got = append(got, e) })
+	h.scan(time.Now())
+	if len(got) != 2 {
+		t.Fatalf("got %d entries", len(got))
+	}
+	e := got[0]
+	if e.App != "sshd" || e.ProcID != "42" || e.Facility != "authpriv" || e.Severity != "info" || e.Host != "srv1" || e.SourceType != "host" || e.Extra["unit"] != "ssh.service" {
+		t.Fatalf("entry %+v", e)
+	}
+	if got[1].Facility != "kern" || got[1].Severity != "err" {
+		t.Fatalf("kernel entry %+v", got[1])
+	}
+	h.scan(time.Now()) // nothing new
+	if len(got) != 2 {
+		t.Fatalf("re-read: %d entries", len(got))
+	}
+	if st := h.Status(); st["mode"] != "journal" || st["files"] != 1 {
+		t.Fatalf("status %+v", st)
+	}
+}
+
+func TestHostLogsFiles(t *testing.T) {
+	root := t.TempDir()
+	dir := filepath.Join(root, "var/log")
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	auth := filepath.Join(dir, "auth.log")
+	if err := os.WriteFile(auth, []byte("Oct  3 07:00:00 srv1 sshd[1]: old line\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	_ = os.WriteFile(filepath.Join(dir, "auth.log.1"), []byte("rotated\n"), 0o644)
+
+	var got []*Entry
+	h := NewHostLogs(root, t.TempDir(), time.Hour, func(e *Entry) { got = append(got, e) })
+	h.scan(time.Now()) // existing content is skipped
+	if len(got) != 0 {
+		t.Fatalf("first scan read %d lines", len(got))
+	}
+	f, _ := os.OpenFile(auth, os.O_APPEND|os.O_WRONLY, 0)
+	_, _ = f.WriteString("Oct  3 08:00:00 srv1 sshd[7]: Failed password for root\nOct  3 08:00:01 srv1 sshd[7]: partial")
+	f.Close()
+	h.scan(time.Now())
+	if len(got) != 1 {
+		t.Fatalf("got %d lines", len(got))
+	}
+	e := got[0]
+	if e.Host != "srv1" || e.App != "sshd" || e.ProcID != "7" || e.Facility != "auth" || e.Extra["log_file"] != "/var/log/auth.log" || e.Proto != "file" {
+		t.Fatalf("entry %+v", e)
+	}
+	if st := h.Status(); st["mode"] != "files" || st["files"] != 1 {
+		t.Fatalf("status %+v", st)
+	}
+}
+
+func TestHostLogsNotMounted(t *testing.T) {
+	h := NewHostLogs(filepath.Join(t.TempDir(), "none"), t.TempDir(), time.Hour, func(*Entry) {})
+	h.scan(time.Now())
+	if st := h.Status(); st["code"] != "not_mounted" {
+		t.Fatalf("status %+v", st)
+	}
+}

journal.go

@@ -0,0 +1,174 @@
+package main
+
+import (
+	"bufio"
+	"bytes"
+	"encoding/binary"
+	"encoding/hex"
+	"errors"
+	"io"
+	"os"
+	"time"
+)
+
+// Minimal reader of systemd journal files (*.journal), enough to follow them
+// without journalctl: objects are walked in file order, which is the order
+// entries were written. Format: https://systemd.io/JOURNAL_FILE_FORMAT/
+
+const (
+	jHeaderMin     = 208 // header fields used below end at offset 208
+	jObjHeaderSize = 16
+
+	jObjData  = 1
+	jObjEntry = 3
+
+	jIncompatCompact = 1 << 4
+	jStateArchived   = 2
+	jObjCompressed   = 1<<0 | 1<<1 | 1<<2 // XZ, LZ4, ZSTD: not decoded (no stdlib codec)
+)
+
+var jSignature = []byte("LPKSHHRH")
+
+// jHeader holds the header fields the reader needs.
+type jHeader struct {
+	compact        bool
+	archived       bool
+	fileID         string
+	headerSize     uint64
+	tailObject     uint64 // offset of the last object
+	tailSeqnum     uint64 // seqnum of the last complete entry
+	tailRealtimeUS uint64 // realtime of the last entry (µs since the epoch)
+}
+
+func readJournalHeader(f io.ReaderAt) (jHeader, error) {
+	b := make([]byte, jHeaderMin)
+	if _, err := f.ReadAt(b, 0); err != nil {
+		return jHeader{}, err
+	}
+	if !bytes.Equal(b[:8], jSignature) {
+		return jHeader{}, errors.New("not a journal file")
+	}
+	le := binary.LittleEndian
+	h := jHeader{
+		compact:        le.Uint32(b[12:])&jIncompatCompact != 0,
+		archived:       b[16] == jStateArchived,
+		fileID:         hex.EncodeToString(b[24:40]),
+		headerSize:     le.Uint64(b[88:]),
+		tailObject:     le.Uint64(b[136:]),
+		tailSeqnum:     le.Uint64(b[160:]),
+		tailRealtimeUS: le.Uint64(b[192:]),
+	}
+	if h.headerSize < jHeaderMin {
+		return h, errors.New("journal header too small")
+	}
+	return h, nil
+}
+
+// journalEntry is one entry: its time and its "FIELD=value" pairs.
+type journalEntry struct {
+	Realtime   time.Time
+	Fields     map[string]string
+	Compressed int // fields that could not be read (compressed data)
+}
+
+// readJournal reads the entries complete after offset `from` (0 = start of
+// the file) and returns the offset to resume from. Entries older than
+// `notBefore` are skipped without decoding their fields.
+func readJournal(path string, from int64, notBefore time.Time, emit func(*journalEntry)) (next int64, h jHeader, err error) {
+	f, err := os.Open(path)
+	if err != nil {
+		return from, h, err
+	}
+	defer f.Close()
+	if h, err = readJournalHeader(f); err != nil {
+		return from, h, err
+	}
+	if from < int64(h.headerSize) {
+		from = int64(h.headerSize)
+	}
+	end := int64(h.tailObject)
+	r := bufio.NewReaderSize(io.NewSectionReader(f, from, 1<<62), 64*1024)
+	le := binary.LittleEndian
+	hdr := make([]byte, jObjHeaderSize)
+	off := from
+	for off <= end {
+		if _, err := io.ReadFull(r, hdr); err != nil {
+			return off, h, nil // object still being written
+		}
+		typ, size := hdr[0], int64(le.Uint64(hdr[8:]))
+		if size < jObjHeaderSize {
+			return off, h, nil
+		}
+		body := size - jObjHeaderSize
+		if typ == jObjEntry && body >= 48 && body < 1<<20 {
+			obj := make([]byte, body)
+			if _, err := io.ReadFull(r, obj); err != nil {
+				return off, h, nil
+			}
+			seq := le.Uint64(obj[0:])
+			if seq == 0 || seq > h.tailSeqnum {
+				return off, h, nil // not linked yet: retry at the next poll
+			}
+			rt := time.UnixMicro(int64(le.Uint64(obj[8:])))
+			if !rt.Before(notBefore) {
+				emit(readEntryFields(f, h.compact, rt, obj[48:]))
+			}
+		} else if _, err := r.Discard(int(body)); err != nil {
+			return off, h, nil
+		}
+		next := (off + size + 7) &^ 7 // objects are 8-byte aligned
+		if pad := next - off - size; pad > 0 {
+			if _, err := r.Discard(int(pad)); err != nil {
+				return next, h, nil // the object is complete: resume after it
+			}
+		}
+		off = next
+	}
+	return off, h, nil
+}
+
+// readEntryFields resolves the data objects referenced by an entry.
+func readEntryFields(f io.ReaderAt, compact bool, rt time.Time, items []byte) *journalEntry {
+	le := binary.LittleEndian
+	e := &journalEntry{Realtime: rt, Fields: make(map[string]string, 16)}
+	step := 16
+	if compact {
+		step = 4
+	}
+	payloadAt := int64(64)
+	if compact {
+		payloadAt = 72
+	}
+	hdr := make([]byte, jObjHeaderSize)
+	for i := 0; i+step <= len(items); i += step {
+		var off int64
+		if compact {
+			off = int64(le.Uint32(items[i:]))
+		} else {
+			off = int64(le.Uint64(items[i:]))
+		}
+		if off == 0 {
+			continue
+		}
+		if _, err := f.ReadAt(hdr, off); err != nil || hdr[0] != jObjData {
+			continue
+		}
+		if hdr[1]&jObjCompressed != 0 {
+			e.Compressed++
+			continue
+		}
+		size := int64(le.Uint64(hdr[8:]))
+		n := size - payloadAt
+		if n <= 0 || n > 1<<20 {
+			continue
+		}
+		p := make([]byte, n)
+		if _, err := f.ReadAt(p, off+payloadAt); err != nil {
+			continue
+		}
+		if k := bytes.IndexByte(p, '='); k > 0 {
+			e.Fields[string(p[:k])] = string(p[k+1:])
+		}
+	}
+	return e
+}

main.go

@@ -1,10 +1,9 @@
-// Logstream: a syslog (UDP/TCP) sink stored in VictoriaLogs,
+// Logstream: a syslog (UDP/TCP), Docker and host system logs sink stored in VictoriaLogs,
 // with a web interface to browse and search the logs.
 package main
 
 import (
 	"context"
-	"crypto/subtle"
 	"embed"
 	"errors"
 	"io/fs"
@@ -29,8 +28,7 @@ type config struct {
 	httpAddr   string
 	vlogsURL   string
 	dataDir    string
-	authUser   string
+	auth       authConfig
-	authPass   string
 	rdns       bool
 	dnsServer  string
 	allowPurge bool
@@ -38,9 +36,18 @@ type config struct {
 	dockerLogs bool
 	dockerHost string
 	backfill   time.Duration
+	hostRoot   string
+	hostFill   time.Duration
+	kmsgDevice string
+	kmsgFill   time.Duration
+	auditLog   string
+	auditHost  string
+	auditFill  time.Duration
+	filesFill  time.Duration
 	batchSize  int
 	queueSize  int
 	flushEvery time.Duration
+	spoolMax   int64
 }
 
 func getenv(key, def string) string {
@@ -57,6 +64,14 @@ func getenvInt(key string, def int) int {
 	return def
 }
 
+// getenvIntZero is getenvInt that also accepts 0 (to turn a feature off).
+func getenvIntZero(key string, def int) int {
+	if v, err := strconv.Atoi(os.Getenv(key)); err == nil && v >= 0 {
+		return v
+	}
+	return def
+}
+
 func getenvBool(key string, def bool) bool {
 	switch strings.ToLower(os.Getenv(key)) {
 	case "1", "true", "yes", "on":
@@ -80,20 +95,46 @@ func main() {
 		httpAddr:   getenv("HTTP_ADDR", ":8080"),
 		vlogsURL:   getenv("VLOGS_URL", "http://victorialogs:9428"),
 		dataDir:    getenv("DATA_DIR", "/data"),
-		authUser:   os.Getenv("AUTH_USER"),
+		auth: authConfig{
-		authPass:   os.Getenv("AUTH_PASS"),
+			mode:         getenv("AUTH_MODE", "local"),
+			user:         os.Getenv("AUTH_USER"),
+			pass:         os.Getenv("AUTH_PASS"),
+			viewerUser:   os.Getenv("AUTH_VIEWER_USER"),
+			viewerPass:   os.Getenv("AUTH_VIEWER_PASS"),
+			adminGroup:   os.Getenv("OIDC_ADMIN_GROUP"),
+			groupsClaim:  os.Getenv("OIDC_GROUPS_CLAIM"),
+			issuer:       os.Getenv("OIDC_ISSUER"),
+			clientID:     os.Getenv("OIDC_CLIENT_ID"),
+			clientSecret: os.Getenv("OIDC_CLIENT_SECRET"),
+			redirectURL:  os.Getenv("OIDC_REDIRECT_URL"),
+			scopes:       os.Getenv("OIDC_SCOPES"),
+			// SESSION_TTL applies to both modes; OIDC_SESSION_TTL is its former name.
+			sessionTTL: getenvDuration("SESSION_TTL", getenvDuration("OIDC_SESSION_TTL", 12*time.Hour)),
+			loginLogo:  os.Getenv("LOGIN_LOGO"),
+		},
 		rdns:       getenvBool("RDNS", true),
 		dnsServer:  os.Getenv("DNS_SERVER"),
-		allowPurge: getenvBool("ALLOW_PURGE", true),
+		allowPurge: getenvBool("ALLOW_PURGE", false),
 		exportMax:  getenvInt("EXPORT_MAX", 100000),
 		dockerLogs: getenvBool("DOCKER_LOGS", false),
 		dockerHost: getenv("DOCKER_HOST", "unix:///var/run/docker.sock"),
 		backfill:   getenvDuration("DOCKER_BACKFILL", time.Hour),
+		hostRoot:   getenv("HOST_LOGS_ROOT", "/host"),
+		hostFill:   getenvDuration("HOST_LOGS_BACKFILL", time.Hour),
+		kmsgDevice: getenv("DMESG_DEVICE", "/dev/kmsg"),
+		kmsgFill:   getenvDuration("DMESG_BACKFILL", time.Hour),
+		auditLog:   getenv("AUDIT_LOG", "/var/log/audit/audit.log"),
+		auditHost:  os.Getenv("AUDIT_HOST"),
+		auditFill:  getenvDuration("AUDIT_BACKFILL", time.Hour),
+		filesFill:  getenvDuration("FILES_BACKFILL", time.Hour),
 		batchSize:  getenvInt("BATCH_SIZE", 1000),
 		queueSize:  getenvInt("QUEUE_SIZE", 100000),
 		flushEvery: time.Duration(getenvInt("FLUSH_MS", 1000)) * time.Millisecond,
+		spoolMax:   int64(getenvIntZero("SPOOL_MAX_MB", 1024)) << 20,
 	}
 
+	cfg.auth.dataDir = cfg.dataDir
+
 	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
 	defer stop()
 
@@ -102,10 +143,19 @@ func main() {
 		log.Fatalf("tags: %v", err)
 	}
 
-	store := NewStore(cfg.vlogsURL, cfg.batchSize, cfg.queueSize, cfg.flushEvery)
+	var spool *Spool
+	if cfg.spoolMax > 0 {
+		if spool, err = OpenSpool(filepath.Join(cfg.dataDir, "spool"), cfg.spoolMax); err != nil {
+			log.Printf("disk buffer disabled: %v", err) // before the self-logs exist
+			spool = nil
+		}
+	}
+	store := NewStore(cfg.vlogsURL, cfg.batchSize, cfg.queueSize, cfg.flushEvery, spool)
+	// The store stops after everything else, so the stop event is still stored.
+	storeCtx, stopStore := context.WithCancel(context.Background())
 	storeDone := make(chan struct{})
 	go func() {
-		store.Run(ctx)
+		store.Run(storeCtx)
 		close(storeDone)
 	}()
 
@@ -113,12 +163,33 @@ func main() {
 	rdns := NewReverseDNS(cfg.rdns, cfg.dnsServer)
 	sink := func(e *Entry) {
 		// Host sent as an IP (or no host in the header): replace it with its DNS name.
-		// A new IP waits at most 300 ms; slower lookups finish in the background.
+		// A new IP waits at most 300 ms, without holding up the listener; slower
-		if name := rdns.Lookup(e.Host, 300*time.Millisecond); name != "" {
+		// lookups finish in the background.
+		rdns.Resolve(e.Host, 300*time.Millisecond, func(name string) {
+			if name != "" {
 				e.HostIP, e.Host = e.Host, name
 			}
 			store.Enqueue(e)
 			hub.Publish(e)
+		})
+	}
+	// Internal events of LogStream (host=logstream), WARNING and above unless
+	// changed in Settings > Sources.
+	selfLog = NewSelfLog(cfg.dataDir, func(e *Entry) {
+		store.Enqueue(e)
+		hub.Publish(e)
+	})
+	selfCtx, stopSelf := context.WithCancel(context.Background())
+	selfDone := make(chan struct{})
+	go func() {
+		selfLog.Run(selfCtx)
+		close(selfDone)
+	}()
+	logEvent("main", 5, "LogStream started")
+
+	tcpMaxConns = getenvInt("SYSLOG_TCP_MAX_CONNS", tcpMaxConns)
+	if d := getenvDuration("SYSLOG_TCP_IDLE", tcpIdle); d > 0 {
+		tcpIdle = d
 	}
 	// Listening errors (port already used…) are shown in Settings > Sources.
 	syslogSrv := NewSyslogServer(ctx, cfg.syslogAddr, getenv("SYSLOG_PUBLIC_PORT", ""), cfg.dataDir, sink)
@@ -128,24 +199,54 @@ func main() {
 		log.Fatal(err)
 	}
 	mux := http.NewServeMux()
-	api := &API{store: store, hub: hub, tags: tags, rdns: rdns, allowPurge: cfg.allowPurge, exportMax: cfg.exportMax, syslog: syslogSrv}
+	presets := getenv("PRESETS_FILE", filepath.Join(cfg.dataDir, "presets.json"))
+	api := &API{store: store, hub: hub, tags: tags, presets: presets, rdns: rdns, allowPurge: cfg.allowPurge, exportMax: cfg.exportMax, syslog: syslogSrv}
 	if cfg.dockerLogs {
 		dm, err := NewDockerManager(cfg.dockerHost, cfg.dataDir, cfg.backfill, sink)
 		if err != nil {
-			log.Printf("docker logs disabled: %v", err)
+			logEvent("docker", 3, "Docker logs disabled: %v", err)
 		} else {
 			go dm.Run(ctx)
 			api.docker = dm
 			log.Printf("docker logs enabled through %s", cfg.dockerHost)
 		}
 	}
+	// System logs of the host (journal or /var/log), off until enabled in Settings > Sources.
+	api.host = NewHostLogs(cfg.hostRoot, cfg.dataDir, cfg.hostFill, sink)
+	go api.host.Run(ctx)
+	// Kernel messages of the host (/dev/kmsg), off until enabled in Settings > Sources.
+	api.dmesg = NewDmesg(cfg.kmsgDevice, cfg.dataDir, cfg.kmsgFill, sink)
+	go api.dmesg.Run(ctx)
+	// Linux audit log of the host (auditd), off until enabled in Settings > Sources.
+	api.audit = NewAudit(cfg.auditLog, cfg.hostRoot, cfg.auditHost, cfg.dataDir, cfg.auditFill, sink)
+	go api.audit.Run(ctx)
+	// Log files of other programs (Traefik, nginx…) mounted in the container,
+	// declared in Settings > Sources.
+	api.files = NewFiles(cfg.hostRoot, cfg.dataDir, cfg.filesFill, sink)
+	go api.files.Run(ctx)
+	api.selfLog = selfLog
 	api.Routes(mux)
 	mux.Handle("GET /", http.FileServer(http.FS(static)))
 
+	handler, err := newAuth(cfg.auth, readOnly(mux))
+	if err != nil {
+		log.Fatalf("auth: %v", err)
+	}
+	if o, ok := handler.(*OIDC); ok {
+		go o.checkProvider()
+	}
+	_, local := handler.(*Local)
+	_, oidc := handler.(*OIDC)
+	authOn := local || oidc
+	if !authOn {
+		log.Printf("warning: no authentication (AUTH_USER is empty): anyone who can reach %s can read the logs and change the settings", cfg.httpAddr)
+	}
+	handler = secure(handler, contentSecurityPolicy(static), authOn)
 	srv := &http.Server{
 		Addr:              cfg.httpAddr,
-		Handler:           basicAuth(cfg.authUser, cfg.authPass, mux),
+		Handler:           handler,
 		ReadHeaderTimeout: 10 * time.Second,
+		IdleTimeout:       2 * time.Minute,
 		// Requests inherit the global context so SSE streams end on shutdown.
 		BaseContext: func(net.Listener) context.Context { return ctx },
 	}
@@ -160,28 +261,10 @@ func main() {
 	if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
 		log.Fatalf("http: %v", err)
 	}
+	logEvent("main", 5, "LogStream stopping")
+	stopSelf()
+	<-selfDone
+	stopStore()
 	<-storeDone
 	log.Println("shutdown complete")
 }
-
-// basicAuth protects the UI when AUTH_USER is set (except /healthz).
-func basicAuth(user, pass string, next http.Handler) http.Handler {
-	if user == "" {
-		return next
-	}
-	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		if r.URL.Path == "/healthz" {
-			next.ServeHTTP(w, r)
-			return
-		}
-		u, p, ok := r.BasicAuth()
-		if !ok ||
-			subtle.ConstantTimeCompare([]byte(u), []byte(user)) != 1 ||
-			subtle.ConstantTimeCompare([]byte(p), []byte(pass)) != 1 {
-			w.Header().Set("WWW-Authenticate", `Basic realm="logstream"`)
-			http.Error(w, "authentication required", http.StatusUnauthorized)
-			return
-		}
-		next.ServeHTTP(w, r)
-	})
-}

presets.go

@@ -0,0 +1,110 @@
+package main
+
+import (
+	_ "embed"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"os"
+)
+
+// Built-in tag presets, used when no presets file is found (PRESETS_FILE,
+// /data/presets.json by default). See docs/presets.md.
+//
+//go:embed presets.json
+var builtinPresets []byte
+
+// i18nText is a text per language ({"en": "...", "fr": "..."}) or one plain
+// string for every language.
+type i18nText map[string]string
+
+func (t *i18nText) UnmarshalJSON(b []byte) error {
+	var s string
+	if json.Unmarshal(b, &s) == nil {
+		*t = i18nText{"en": s}
+		return nil
+	}
+	var m map[string]string
+	if err := json.Unmarshal(b, &m); err != nil {
+		return errors.New("expected a string or an object of strings per language")
+	}
+	*t = m
+	return nil
+}
+
+type PresetTag struct {
+	Label         i18nText `json:"label,omitempty"`
+	Pattern       string   `json:"pattern"`
+	Color         string   `json:"color"`
+	Regex         *bool    `json:"regex,omitempty"` // regular expression unless false
+	WholeWord     bool     `json:"wholeWord,omitempty"`
+	CaseSensitive bool     `json:"caseSensitive,omitempty"`
+}
+
+type Preset struct {
+	ID   string      `json:"id"`
+	Name i18nText    `json:"name"`
+	Tags []PresetTag `json:"tags"`
+}
+
+type PresetGroup struct {
+	Group   i18nText `json:"group"`
+	Presets []Preset `json:"presets"`
+}
+
+// parsePresets decodes and checks a presets file with the same rules as the
+// tags, so that every preset can be added as is.
+func parsePresets(b []byte) ([]PresetGroup, error) {
+	var groups []PresetGroup
+	if err := json.Unmarshal(b, &groups); err != nil {
+		return nil, err
+	}
+	seen := map[string]bool{}
+	for _, g := range groups {
+		for _, p := range g.Presets {
+			if p.ID == "" || seen[p.ID] {
+				return nil, fmt.Errorf("preset %q: missing or duplicate id", p.ID)
+			}
+			seen[p.ID] = true
+			if len(p.Tags) == 0 {
+				return nil, fmt.Errorf("preset %q: no tags", p.ID)
+			}
+			for i, pt := range p.Tags {
+				t := Tag{Pattern: pt.Pattern, Color: pt.Color, Regex: pt.Regex == nil || *pt.Regex}
+				if err := t.validate(); err != nil {
+					return nil, fmt.Errorf("preset %q, tag %d: %w", p.ID, i+1, err)
+				}
+			}
+		}
+	}
+	return groups, nil
+}
+
+// presetsResponse is what GET /api/presets returns.
+type presetsResponse struct {
+	Source string        `json:"source"` // "file" or "builtin"
+	File   string        `json:"file"`
+	Error  string        `json:"error,omitempty"` // the file is invalid: built-in presets are used
+	Groups []PresetGroup `json:"groups"`
+}
+
+// loadPresets reads the presets file on every call, so that edits apply
+// without a restart, and falls back to the built-in presets.
+func loadPresets(path string) presetsResponse {
+	res := presetsResponse{Source: "builtin", File: path}
+	if path != "" {
+		b, err := os.ReadFile(path)
+		switch {
+		case err == nil:
+			if res.Groups, err = parsePresets(b); err == nil {
+				res.Source = "file"
+				return res
+			}
+			res.Error = err.Error()
+		case !errors.Is(err, os.ErrNotExist):
+			res.Error = err.Error()
+		}
+	}
+	res.Groups, _ = parsePresets(builtinPresets) // checked by the tests
+	return res
+}

presets.json

@@ -0,0 +1,244 @@
+[
+  {
+    "group": "HTTP/HTTPS",
+    "presets": [
+      {
+        "id": "http_status",
+        "name": { "en": "HTTP status codes", "fr": "Codes HTTP" },
+        "tags": [
+          {
+            "label": "HTTP 2xx",
+            "color": "#86efac",
+            "pattern": "(?:\" |\"(?:status|DownstreamStatus|OriginStatus|status_code)\": ?|(?:-1|\\d+)/(?:-1|\\d+)/(?:-1|\\d+)/(?:-1|\\d+)/\\+?\\d+ )(?<hl>2\\d\\d)\\b"
+          },
+          {
+            "label": "HTTP 3xx",
+            "color": "#93c5fd",
+            "pattern": "(?:\" |\"(?:status|DownstreamStatus|OriginStatus|status_code)\": ?|(?:-1|\\d+)/(?:-1|\\d+)/(?:-1|\\d+)/(?:-1|\\d+)/\\+?\\d+ )(?<hl>3\\d\\d)\\b"
+          },
+          {
+            "label": "HTTP 4xx",
+            "color": "#fdba74",
+            "pattern": "(?:\" |\"(?:status|DownstreamStatus|OriginStatus|status_code)\": ?|(?:-1|\\d+)/(?:-1|\\d+)/(?:-1|\\d+)/(?:-1|\\d+)/\\+?\\d+ )(?<hl>4\\d\\d)\\b"
+          },
+          {
+            "label": "HTTP 5xx",
+            "color": "#f87171",
+            "pattern": "(?:\" |\"(?:status|DownstreamStatus|OriginStatus|status_code)\": ?|(?:-1|\\d+)/(?:-1|\\d+)/(?:-1|\\d+)/(?:-1|\\d+)/\\+?\\d+ )(?<hl>5\\d\\d)\\b"
+          }
+        ]
+      },
+      {
+        "id": "http_methods",
+        "name": { "en": "HTTP methods", "fr": "Méthodes HTTP" },
+        "tags": [
+          {
+            "label": "GET/HEAD/OPTIONS",
+            "color": "#cbd5e1",
+            "caseSensitive": true,
+            "pattern": "\"(?<hl>GET|HEAD|OPTIONS)[ \"]"
+          },
+          {
+            "label": "POST/PUT/PATCH",
+            "color": "#c4b5fd",
+            "caseSensitive": true,
+            "pattern": "\"(?<hl>POST|PUT|PATCH)[ \"]"
+          },
+          {
+            "label": "DELETE",
+            "color": "#f9a8d4",
+            "caseSensitive": true,
+            "pattern": "\"(?<hl>DELETE)[ \"]"
+          }
+        ]
+      },
+      {
+        "id": "http_probes",
+        "name": { "en": "Probes and attacks", "fr": "Sondes et attaques" },
+        "tags": [
+          {
+            "label": { "en": "probes / attacks", "fr": "sondes / attaques" },
+            "color": "#fda4af",
+            "pattern": "(?:wp-login\\.php|xmlrpc\\.php|wp-admin|phpmyadmin|/\\.env|/\\.git|/\\.aws|/cgi-bin/|\\.\\./|%2e%2e|/etc/passwd|<script|union(?:\\s|%20|\\+)+select)"
+          }
+        ]
+      },
+      {
+        "id": "http_bots",
+        "name": { "en": "Bots and scripts", "fr": "Robots et scripts" },
+        "tags": [
+          {
+            "label": { "en": "bots / scripts", "fr": "robots / scripts" },
+            "color": "#fde68a",
+            "pattern": "\\b(?:[a-z]*bot|crawler|spider|curl|wget|python-requests|Go-http-client|zgrab|masscan|nmap|sqlmap|nikto)\\b"
+          }
+        ]
+      },
+      {
+        "id": "http_errors",
+        "name": { "en": "TLS/HTTPS and proxy errors", "fr": "Erreurs TLS/HTTPS et proxy" },
+        "tags": [
+          {
+            "label": { "en": "TLS errors", "fr": "erreurs TLS" },
+            "color": "#f0abfc",
+            "pattern": "(?:TLS handshake error|SSL_do_handshake\\(\\) failed|SSL handshake|SSL routines|certificate (?:has )?expired|certificate verify failed|bad certificate|unknown certificate|x509:)"
+          },
+          {
+            "label": { "en": "proxy errors", "fr": "erreurs proxy" },
+            "color": "#fdba74",
+            "pattern": "(?:upstream timed out|upstream prematurely closed|no live upstreams|connect\\(\\) failed|connection refused|bad gateway|gateway time-?out|service unavailable)"
+          }
+        ]
+      }
+    ]
+  },
+  {
+    "group": { "en": "System", "fr": "Système" },
+    "presets": [
+      {
+        "id": "sys_auth",
+        "name": { "en": "SSH and logins", "fr": "SSH et connexions" },
+        "tags": [
+          {
+            "label": { "en": "login failures", "fr": "échecs de connexion" },
+            "color": "#fca5a5",
+            "pattern": "(?:Failed (?:password|publickey|none)|Invalid user|authentication failures?|Connection closed by (?:invalid|authenticating) user|maximum authentication attempts exceeded|FAILED (?:LOGIN|SU)|incorrect password attempts?|NOT in sudoers)"
+          },
+          {
+            "label": { "en": "logins", "fr": "connexions" },
+            "color": "#86efac",
+            "pattern": "(?:Accepted (?:password|publickey|keyboard-interactive(?:/pam)?)|session opened for user|New session \\S+ of user)"
+          }
+        ]
+      },
+      {
+        "id": "sys_sudo",
+        "name": { "en": "sudo commands", "fr": "Commandes sudo" },
+        "tags": [
+          {
+            "label": { "en": "sudo commands", "fr": "commandes sudo" },
+            "color": "#fde68a",
+            "caseSensitive": true,
+            "pattern": "\\bCOMMAND=\\S+"
+          }
+        ]
+      },
+      {
+        "id": "sys_kernel",
+        "name": { "en": "Kernel: OOM, crashes, disks", "fr": "Noyau : OOM, plantages, disques" },
+        "tags": [
+          {
+            "label": { "en": "out of memory", "fr": "mémoire épuisée" },
+            "color": "#f87171",
+            "pattern": "(?:Out of memory|oom-kill(?:er)?|oom_reaper|Killed process \\d+|invoked oom-killer)"
+          },
+          {
+            "label": { "en": "kernel errors", "fr": "erreurs noyau" },
+            "color": "#fda4af",
+            "pattern": "(?:Kernel panic|\\bBUG: |\\bOops\\b|Call Trace|segfault at|general protection fault|I/O error|EXT4-fs error|Buffer I/O error|blocked for more than \\d+ seconds|Hardware Error|soft lockup|hard LOCKUP)"
+          }
+        ]
+      },
+      {
+        "id": "sys_systemd",
+        "name": { "en": "systemd services", "fr": "Services systemd" },
+        "tags": [
+          {
+            "label": { "en": "failed services", "fr": "services en échec" },
+            "color": "#fca5a5",
+            "pattern": "(?:Failed to start|failed with result|Main process exited, code=(?:exited|killed|dumped)|entered failed state|Start request repeated too quickly|Dependency failed)"
+          },
+          {
+            "label": { "en": "service start/stop", "fr": "démarrage/arrêt de service" },
+            "color": "#bbf7d0",
+            "caseSensitive": true,
+            "pattern": "\\b(?:Started|Starting|Stopped|Stopping|Reloaded|Reloading|Reached target)\\b"
+          }
+        ]
+      },
+      {
+        "id": "sys_firewall",
+        "name": { "en": "Firewall and fail2ban", "fr": "Pare-feu et fail2ban" },
+        "tags": [
+          {
+            "label": { "en": "firewall", "fr": "pare-feu" },
+            "color": "#fdba74",
+            "caseSensitive": true,
+            "pattern": "(?:\\[UFW (?:BLOCK|ALLOW|AUDIT|LIMIT BLOCK)\\]|\\b(?:DROP|REJECT)\\b|\\b(?:Ban|Unban|Found) \\d{1,3}(?:\\.\\d{1,3}){3}\\b)"
+          }
+        ]
+      }
+    ]
+  },
+  {
+    "group": "Applications",
+    "presets": [
+      {
+        "id": "app_docker",
+        "name": { "en": "Docker and containers", "fr": "Docker et conteneurs" },
+        "tags": [
+          {
+            "label": { "en": "container problems", "fr": "problèmes de conteneur" },
+            "color": "#fcd34d",
+            "pattern": "(?:\\bOOMKilled\\b|exited with code [1-9]\\d*|exit code: [1-9]\\d*|health_status: unhealthy|\\bunhealthy\\b|Back-off restarting|CrashLoopBackOff|container (?:die|kill|oom)\\b|restarting \\(\\d+\\))"
+          }
+        ]
+      },
+      {
+        "id": "app_db",
+        "name": { "en": "Databases", "fr": "Bases de données" },
+        "tags": [
+          {
+            "label": { "en": "database errors", "fr": "erreurs base de données" },
+            "color": "#c4b5fd",
+            "pattern": "(?:\\bdeadlock(?: detected| found)?\\b|duplicate key|too many (?:connections|clients)|lock wait timeout|slow query|could not connect to server|server has gone away|out of shared memory|terminating connection|Access denied for user|password authentication failed)"
+          }
+        ]
+      }
+    ]
+  },
+  {
+    "group": { "en": "General", "fr": "Général" },
+    "presets": [
+      {
+        "id": "gen_levels",
+        "name": { "en": "Log levels", "fr": "Niveaux de log" },
+        "tags": [
+          {
+            "label": { "en": "fatal / critical", "fr": "fatal / critique" },
+            "color": "#ef4444",
+            "pattern": "\\b(?:fatal|crit(?:ical)?|panic|emerg(?:ency)?)\\b"
+          },
+          {
+            "label": "info / notice",
+            "color": "#bfdbfe",
+            "pattern": "\\b(?:info|notice)\\b"
+          },
+          {
+            "label": "debug / trace",
+            "color": "#e5e7eb",
+            "pattern": "\\b(?:debug|trace)\\b"
+          },
+          {
+            "label": "ok",
+            "color": "#86efac",
+            "regex": false,
+            "wholeWord": true,
+            "pattern": "ok"
+          }
+        ]
+      },
+      {
+        "id": "gen_ip",
+        "name": { "en": "IPv4 addresses", "fr": "Adresses IPv4" },
+        "tags": [
+          {
+            "label": { "en": "IPv4 addresses", "fr": "adresses IPv4" },
+            "color": "#a5f3fc",
+            "pattern": "\\b(?:(?:25[0-5]|2[0-4]\\d|1?\\d?\\d)\\.){3}(?:25[0-5]|2[0-4]\\d|1?\\d?\\d)\\b"
+          }
+        ]
+      }
+    ]
+  }
+]

presets_test.go

@@ -0,0 +1,46 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+func TestBuiltinPresets(t *testing.T) {
+	groups, err := parsePresets(builtinPresets)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(groups) == 0 {
+		t.Fatal("no built-in presets")
+	}
+}
+
+func TestLoadPresetsFile(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "presets.json")
+
+	if res := loadPresets(path); res.Source != "builtin" || res.Error != "" {
+		t.Fatalf("missing file: got %q, error %q", res.Source, res.Error)
+	}
+
+	os.WriteFile(path, []byte(`[{"group":"Mine","presets":[{"id":"a","name":{"en":"A","fr":"A fr"},
+		"tags":[{"label":"x","pattern":"foo|bar","color":"#112233"},{"pattern":"a.b","color":"#445566","regex":false}]}]}]`), 0o644)
+	res := loadPresets(path)
+	if res.Source != "file" || res.Error != "" || res.Groups[0].Presets[0].Name["fr"] != "A fr" || res.Groups[0].Group["en"] != "Mine" {
+		t.Fatalf("valid file: %+v", res)
+	}
+
+	for _, bad := range []string{
+		`not json`,
+		`[{"group":"G","presets":[{"id":"a","name":"A","tags":[{"pattern":"(","color":"#112233"}]}]}]`,
+		`[{"group":"G","presets":[{"id":"a","name":"A","tags":[{"pattern":"x","color":"red"}]}]}]`,
+		`[{"group":"G","presets":[{"id":"a","name":"A","tags":[]}]}]`,
+		`[{"group":"G","presets":[{"id":"a","name":"A","tags":[{"pattern":"x","color":"#112233"}]},{"id":"a","name":"B","tags":[{"pattern":"y","color":"#112233"}]}]}]`,
+	} {
+		os.WriteFile(path, []byte(bad), 0o644)
+		if res := loadPresets(path); res.Source != "builtin" || res.Error == "" || len(res.Groups) == 0 {
+			t.Errorf("%s: got %q, error %q", bad, res.Source, res.Error)
+		}
+	}
+}

query.go

@@ -17,7 +17,7 @@ type Filter struct {
 	Host     string
 	App      string
 	Severity int    // highest severity number included (0 = emerg … 7 = debug), -1 = all
-	Source   string // "syslog", "docker" or "" for all
+	Source   string // "syslog", "docker", "host" or "" for all
 }
 
 var rangeDurations = map[string]time.Duration{
@@ -119,10 +119,10 @@ func (f Filter) filterExpr() string {
 		parts = append(parts, "app:="+strconv.Quote(f.App))
 	}
 	switch f.Source {
-	case "docker":
+	case "docker", "host", "kernel", "audit", "logstream", "file":
-		parts = append(parts, `source_type:="docker"`)
+		parts = append(parts, `source_type:=`+strconv.Quote(f.Source))
 	case "syslog": // also matches logs stored before source_type existed
-		parts = append(parts, `!(source_type:="docker")`)
+		parts = append(parts, `!(source_type:in("docker","host","kernel","audit","logstream","file"))`)
 	}
 	if f.Severity >= 0 && f.Severity < 7 {
 		names := make([]string, 0, 8)
@@ -209,7 +209,7 @@ func (m *Matcher) Match(e *Entry) bool {
 	if m.f.App != "" && e.App != m.f.App {
 		return false
 	}
-	if (m.f.Source == "docker") != (e.SourceType == "docker") && m.f.Source != "" {
+	if m.f.Source != "" && m.f.Source != e.SourceType {
 		return false
 	}
 	if m.f.Severity >= 0 && e.SevNum > m.f.Severity {

rdns.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"container/list"
 	"context"
 	"net"
 	"strings"
@@ -15,17 +16,26 @@ type ReverseDNS struct {
 	posTTL  time.Duration // cache duration of a found name
 	negTTL  time.Duration // cache duration of "no name"
 
+	lookups chan struct{} // limits the lookups running at once
+	waiters chan struct{} // limits the messages waiting for a lookup (Resolve)
+
 	mu    sync.Mutex
-	cache map[string]*rdnsEntry
+	cache map[string]*list.Element // ip -> element of lru
+	lru   *list.List               // *rdnsEntry, most recently used first
 }
 
 type rdnsEntry struct {
+	ip      string
 	name    string
 	expires time.Time
 	done    chan struct{} // closed once the lookup has finished
 }
 
-const rdnsMaxEntries = 10000
+const (
+	rdnsMaxEntries = 10000
+	rdnsMaxLookups = 64
+	rdnsMaxWaiters = 1024
+)
 
 // NewReverseDNS uses the system resolver, or `server` ("ip" or "ip:port") when set.
 func NewReverseDNS(enabled bool, server string) *ReverseDNS {
@@ -47,7 +57,10 @@ func NewReverseDNS(enabled bool, server string) *ReverseDNS {
 		r:       r,
 		posTTL:  time.Hour,
 		negTTL:  10 * time.Minute,
-		cache:   make(map[string]*rdnsEntry),
+		lookups: make(chan struct{}, rdnsMaxLookups),
+		waiters: make(chan struct{}, rdnsMaxWaiters),
+		cache:   make(map[string]*list.Element),
+		lru:     list.New(),
 	}
 }
 
@@ -60,6 +73,38 @@ func isClosed(ch chan struct{}) bool {
 	}
 }
 
+// entry returns the cache entry of ip, starting its lookup when it is missing
+// or expired, or nil when too many lookups are already running (a flood of
+// unknown addresses). The least recently used entry makes room for a new one.
+func (d *ReverseDNS) entry(ip string) *rdnsEntry {
+	d.mu.Lock()
+	defer d.mu.Unlock()
+	if el := d.cache[ip]; el != nil {
+		e := el.Value.(*rdnsEntry)
+		if !isClosed(e.done) || time.Now().Before(e.expires) {
+			d.lru.MoveToFront(el)
+			return e
+		}
+	}
+	select {
+	case d.lookups <- struct{}{}:
+	default:
+		return nil
+	}
+	if el := d.cache[ip]; el != nil {
+		d.lru.Remove(el)
+	}
+	for d.lru.Len() >= rdnsMaxEntries {
+		old := d.lru.Back()
+		d.lru.Remove(old)
+		delete(d.cache, old.Value.(*rdnsEntry).ip)
+	}
+	e := &rdnsEntry{ip: ip, done: make(chan struct{})}
+	d.cache[ip] = d.lru.PushFront(e)
+	go d.resolve(ip, e)
+	return e
+}
+
 // Lookup returns the name of ip, or "" when ip is not an IP address, has no
 // PTR record, or is not resolved within `wait`. A lookup that takes longer
 // keeps running in the background and fills the cache for later calls.
@@ -67,18 +112,10 @@ func (d *ReverseDNS) Lookup(ip string, wait time.Duration) string {
 	if !d.enabled || net.ParseIP(ip) == nil {
 		return ""
 	}
-	d.mu.Lock()
+	e := d.entry(ip)
-	e := d.cache[ip]
+	if e == nil {
-	if e == nil || (isClosed(e.done) && time.Now().After(e.expires)) {
+		return ""
-		if len(d.cache) >= rdnsMaxEntries {
-			d.cache = make(map[string]*rdnsEntry)
 	}
-		e = &rdnsEntry{done: make(chan struct{})}
-		d.cache[ip] = e
-		go d.resolve(ip, e)
-	}
-	d.mu.Unlock()
-
 	if !isClosed(e.done) {
 		timer := time.NewTimer(wait)
 		defer timer.Stop()
@@ -91,6 +128,43 @@ func (d *ReverseDNS) Lookup(ip string, wait time.Duration) string {
 	return e.name
 }
 
+// Resolve is Lookup without blocking the caller: fn gets the name (or "") at
+// once when it is known, otherwise from a goroutine after at most `wait`. The
+// syslog listeners use it so that a slow DNS server never delays the reading
+// of the next messages.
+func (d *ReverseDNS) Resolve(ip string, wait time.Duration, fn func(name string)) {
+	if !d.enabled || net.ParseIP(ip) == nil {
+		fn("")
+		return
+	}
+	e := d.entry(ip)
+	switch {
+	case e == nil:
+		fn("")
+		return
+	case isClosed(e.done):
+		fn(e.name)
+		return
+	}
+	select {
+	case d.waiters <- struct{}{}:
+	default:
+		fn("") // too many messages waiting already
+		return
+	}
+	go func() {
+		defer func() { <-d.waiters }()
+		timer := time.NewTimer(wait)
+		defer timer.Stop()
+		select {
+		case <-e.done:
+			fn(e.name)
+		case <-timer.C:
+			fn("")
+		}
+	}()
+}
+
 // LookupMany resolves several addresses in parallel; unresolved ones are absent.
 func (d *ReverseDNS) LookupMany(ips []string, wait time.Duration) map[string]string {
 	out := make(map[string]string)
@@ -112,6 +186,7 @@ func (d *ReverseDNS) LookupMany(ips []string, wait time.Duration) map[string]str
 }
 
 func (d *ReverseDNS) resolve(ip string, e *rdnsEntry) {
+	defer func() { <-d.lookups }()
 	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
 	defer cancel()
 	ttl := d.negTTL

rdns_test.go

@@ -0,0 +1,77 @@
+package main
+
+import (
+	"context"
+	"errors"
+	"net"
+	"testing"
+	"time"
+)
+
+// slowDNS never answers before the context ends.
+func slowDNS() *ReverseDNS {
+	d := NewReverseDNS(true, "")
+	d.r = &net.Resolver{PreferGo: true, Dial: func(ctx context.Context, _, _ string) (net.Conn, error) {
+		<-ctx.Done()
+		return nil, errors.New("timeout")
+	}}
+	return d
+}
+
+func TestResolveDoesNotBlock(t *testing.T) {
+	d := slowDNS()
+	got := make(chan string, 1)
+	start := time.Now()
+	d.Resolve("192.0.2.1", 50*time.Millisecond, func(name string) { got <- name })
+	if time.Since(start) > 20*time.Millisecond {
+		t.Fatal("Resolve waited for the DNS server")
+	}
+	if name := <-got; name != "" {
+		t.Errorf("name %q, want none", name)
+	}
+
+	// Not an IP address: the callback runs at once.
+	called := false
+	d.Resolve("router", time.Second, func(name string) { called = name == "" })
+	if !called {
+		t.Error("callback not called synchronously for a host name")
+	}
+}
+
+func TestReverseDNSLimits(t *testing.T) {
+	d := slowDNS()
+	// Lookups beyond the limit are not started (and not cached).
+	for i := 0; i < rdnsMaxLookups+10; i++ {
+		d.Lookup(net.IPv4(10, 0, byte(i>>8), byte(i)).String(), 0)
+	}
+	if n := d.lru.Len(); n != rdnsMaxLookups {
+		t.Errorf("%d entries, want %d", n, rdnsMaxLookups)
+	}
+}
+
+func TestReverseDNSEvictsLeastRecentlyUsed(t *testing.T) {
+	d := slowDNS()
+	done := make(chan struct{})
+	close(done)
+	add := func(ip string) {
+		e := &rdnsEntry{ip: ip, name: ip + ".lan", expires: time.Now().Add(time.Hour), done: done}
+		d.cache[ip] = d.lru.PushFront(e)
+	}
+	for i := 0; i < rdnsMaxEntries; i++ {
+		add(net.IPv4(10, 1, byte(i>>8), byte(i)).String())
+	}
+	first := net.IPv4(10, 1, 0, 0).String()
+	if name := d.Lookup(first, 0); name != first+".lan" { // now the most recent
+		t.Fatalf("cached name %q", name)
+	}
+	d.entry("192.0.2.9")
+	if d.lru.Len() != rdnsMaxEntries {
+		t.Errorf("%d entries, want %d", d.lru.Len(), rdnsMaxEntries)
+	}
+	if d.cache[first] == nil {
+		t.Error("recently used entry evicted")
+	}
+	if d.cache[net.IPv4(10, 1, 0, 1).String()] != nil {
+		t.Error("least recently used entry kept")
+	}
+}

selflog.go

@@ -0,0 +1,187 @@
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"log"
+	"os"
+	"path/filepath"
+	"sync"
+	"time"
+)
+
+// selfLogConfig is saved in /data/selflog.json and edited in Settings > Sources.
+type selfLogConfig struct {
+	Enabled  bool   `json:"enabled"`
+	MinLevel string `json:"min_level"` // severity name: "err", "warning", "notice", "info"…
+}
+
+// SelfLog turns the internal events of LogStream (storage errors, listener
+// failures, sources that stop…) into log entries, with host=logstream and
+// app=component. Events go through a bounded queue and a per-message
+// throttle: an event about a failing Store is itself stored through that
+// Store, so it must never be able to loop or pile up.
+type SelfLog struct {
+	cfgPath  string
+	sink     func(*Entry)
+	queue    chan *Entry
+	interval time.Duration // one event per component and message format per interval
+
+	mu         sync.Mutex
+	cfg        selfLogConfig
+	last       map[string]time.Time
+	suppressed map[string]int
+	emitted    uint64
+	dropped    uint64
+	throttled  uint64
+}
+
+// selfLog receives the events of logEvent; nil (tests) only writes them to
+// the process log.
+var selfLog *SelfLog
+
+func NewSelfLog(dataDir string, sink func(*Entry)) *SelfLog {
+	s := &SelfLog{
+		cfgPath:    filepath.Join(dataDir, "selflog.json"),
+		sink:       sink,
+		queue:      make(chan *Entry, 256),
+		interval:   30 * time.Second,
+		cfg:        selfLogConfig{Enabled: true, MinLevel: "warning"},
+		last:       map[string]time.Time{},
+		suppressed: map[string]int{},
+	}
+	if b, err := os.ReadFile(s.cfgPath); err == nil {
+		var cfg selfLogConfig
+		if json.Unmarshal(b, &cfg) == nil {
+			s.cfg = normSelfLogConfig(cfg)
+		}
+	}
+	return s
+}
+
+func normSelfLogConfig(cfg selfLogConfig) selfLogConfig {
+	if sevNum(cfg.MinLevel) < 0 {
+		cfg.MinLevel = "warning"
+	}
+	return cfg
+}
+
+func sevNum(name string) int {
+	for i, n := range severityNames {
+		if n == name {
+			return i
+		}
+	}
+	return -1
+}
+
+// logEvent writes an internal event to the process log and, when the
+// self-logs source is on and the level is high enough, emits it as an entry.
+// sev is a syslog severity (3 err, 4 warning, 5 notice, 6 info).
+func logEvent(component string, sev int, format string, args ...any) {
+	msg := fmt.Sprintf(format, args...)
+	log.Printf("%s: %s", component, msg)
+	if s := selfLog; s != nil {
+		s.Emit(component, sev, format, msg, time.Now())
+	}
+}
+
+// Emit queues an event without ever blocking. The same component and message
+// format are let through once per interval; the next one says how many were
+// left out in between.
+func (s *SelfLog) Emit(component string, sev int, format, msg string, now time.Time) {
+	sev = min(max(sev, 0), 7)
+	key := component + "\x00" + format
+	s.mu.Lock()
+	if !s.cfg.Enabled || sev > sevNum(s.cfg.MinLevel) {
+		s.mu.Unlock()
+		return
+	}
+	if t, ok := s.last[key]; ok && now.Sub(t) < s.interval {
+		s.suppressed[key]++
+		s.throttled++
+		s.mu.Unlock()
+		return
+	}
+	s.last[key] = now
+	if n := s.suppressed[key]; n > 0 {
+		msg += fmt.Sprintf(" (%d similar events not shown)", n)
+		delete(s.suppressed, key)
+	}
+	if len(s.last) > 1000 { // formats are constants: only a bug could get here
+		clear(s.last)
+	}
+	s.mu.Unlock()
+
+	e := &Entry{
+		Time:       now,
+		Received:   now,
+		Host:       "logstream",
+		App:        component,
+		Facility:   "syslog",
+		Severity:   severityNames[sev],
+		SevNum:     sev,
+		Message:    msg,
+		Proto:      "internal",
+		SourceType: "logstream",
+		// Wait stays false: when the Store queue is full, the event is dropped
+		// instead of blocking the component that reports it.
+	}
+	select {
+	case s.queue <- e:
+	default:
+		s.mu.Lock()
+		s.dropped++
+		s.mu.Unlock()
+	}
+}
+
+// Run hands the queued events to the sink until the context ends, then
+// delivers what is left (the stop event among them).
+func (s *SelfLog) Run(ctx context.Context) {
+	for {
+		select {
+		case e := <-s.queue:
+			s.deliver(e)
+		case <-ctx.Done():
+			for {
+				select {
+				case e := <-s.queue:
+					s.deliver(e)
+				default:
+					return
+				}
+			}
+		}
+	}
+}
+
+func (s *SelfLog) deliver(e *Entry) {
+	s.sink(e)
+	s.mu.Lock()
+	s.emitted++
+	s.mu.Unlock()
+}
+
+// Configure saves and applies the configuration.
+func (s *SelfLog) Configure(cfg selfLogConfig) error {
+	cfg = normSelfLogConfig(cfg)
+	s.mu.Lock()
+	s.cfg = cfg
+	s.mu.Unlock()
+	return writeJSONFile(s.cfgPath, cfg)
+}
+
+// Status is the state shown in Settings > Sources.
+func (s *SelfLog) Status() map[string]any {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	return map[string]any{
+		"enabled":   s.cfg.Enabled,
+		"min_level": s.cfg.MinLevel,
+		"emitted":   s.emitted,
+		"dropped":   s.dropped,
+		"throttled": s.throttled,
+	}
+}

selflog_test.go

@@ -0,0 +1,102 @@
+package main
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestSelfLogEmit(t *testing.T) {
+	now := time.Unix(1759572000, 0)
+	tests := []struct {
+		name   string
+		cfg    selfLogConfig
+		events []int // severities, one second apart, same message format
+		want   int   // entries queued
+		suffix string
+	}{
+		{"warning and above", selfLogConfig{Enabled: true, MinLevel: "warning"}, []int{3}, 1, ""},
+		{"below the level", selfLogConfig{Enabled: true, MinLevel: "warning"}, []int{5, 6}, 0, ""},
+		{"info level", selfLogConfig{Enabled: true, MinLevel: "info"}, []int{6}, 1, ""},
+		{"disabled", selfLogConfig{Enabled: false, MinLevel: "debug"}, []int{0}, 0, ""},
+		{"bad level falls back to warning", normSelfLogConfig(selfLogConfig{Enabled: true, MinLevel: "loud"}), []int{4, 5}, 1, ""},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			s := NewSelfLog(t.TempDir(), func(*Entry) {})
+			s.cfg = tt.cfg
+			for i, sev := range tt.events {
+				s.Emit("store", sev, "fmt %d", "msg", now.Add(time.Duration(i)*time.Second))
+			}
+			if len(s.queue) != tt.want {
+				t.Fatalf("%d queued, want %d", len(s.queue), tt.want)
+			}
+		})
+	}
+
+	// Same component and format: once per interval, then a count of the others.
+	s := NewSelfLog(t.TempDir(), func(*Entry) {})
+	for i := 0; i < 5; i++ {
+		s.Emit("store", 3, "%d messages lost", "x", now.Add(time.Duration(i)*time.Second))
+	}
+	s.Emit("syslog", 3, "%d messages lost", "y", now) // another component
+	s.Emit("store", 3, "%d messages lost", "z", now.Add(s.interval))
+	if len(s.queue) != 3 {
+		t.Fatalf("%d queued", len(s.queue))
+	}
+	<-s.queue
+	if e := <-s.queue; e.App != "syslog" || e.Host != "logstream" || e.SourceType != "logstream" || e.Wait {
+		t.Fatalf("entry %+v", e)
+	}
+	if e := <-s.queue; e.Message != "z (4 similar events not shown)" || e.Severity != "err" {
+		t.Fatalf("message %q", e.Message)
+	}
+	if st := s.Status(); st["throttled"] != uint64(4) {
+		t.Fatalf("status %v", st)
+	}
+
+	// A full queue drops events instead of blocking.
+	s = NewSelfLog(t.TempDir(), func(*Entry) {})
+	for i := 0; i < cap(s.queue)+10; i++ {
+		s.Emit("store", 3, "event %d", "x", now.Add(time.Duration(i)*s.interval))
+	}
+	if st := s.Status(); st["dropped"] != uint64(10) {
+		t.Fatalf("status %v", st)
+	}
+}
+
+func TestSelfLogRunAndConfigure(t *testing.T) {
+	dir := t.TempDir()
+	sink := &entrySink{}
+	s := NewSelfLog(dir, sink.add)
+	if err := s.Configure(selfLogConfig{Enabled: true, MinLevel: "notice"}); err != nil {
+		t.Fatal(err)
+	}
+	if b, _ := os.ReadFile(filepath.Join(dir, "selflog.json")); !strings.Contains(string(b), `"notice"`) {
+		t.Fatalf("saved %s", b)
+	}
+	if s2 := NewSelfLog(dir, nil); s2.cfg.MinLevel != "notice" {
+		t.Fatalf("reloaded %+v", s2.cfg)
+	}
+	old := selfLog
+	selfLog = s
+	defer func() { selfLog = old }()
+
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan struct{})
+	go func() { s.Run(ctx); close(done) }()
+	logEvent("main", 5, "LogStream started")
+	logEvent("main", 6, "too verbose")
+	cancel()
+	<-done
+	logEvent("main", 5, "LogStream stopping") // queued after Run: not lost if Run drains it
+	sink.mu.Lock()
+	n := len(sink.es)
+	sink.mu.Unlock()
+	if n != 1 || s.Status()["emitted"] != uint64(1) {
+		t.Fatalf("%d entries, status %v", n, s.Status())
+	}
+}

spool.go

@@ -0,0 +1,136 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+// Spool keeps on disk the batches VictoriaLogs could not take, so that they
+// are sent later instead of being lost. Each batch is one NDJSON file named
+// <unix nanoseconds>-<lines>.ndjson; files are sent back oldest first.
+type Spool struct {
+	dir string
+	max int64 // maximum total size in bytes
+
+	mu    sync.Mutex
+	size  int64 // bytes on disk
+	lines int64 // messages on disk
+	seq   int64
+}
+
+// errSpoolFull is returned when a batch does not fit within SPOOL_MAX_MB.
+var errSpoolFull = fmt.Errorf("disk buffer full")
+
+// OpenSpool creates the directory if needed and counts the batches already
+// there (left by a previous run).
+func OpenSpool(dir string, max int64) (*Spool, error) {
+	if err := os.MkdirAll(dir, 0o755); err != nil {
+		return nil, err
+	}
+	s := &Spool{dir: dir, max: max}
+	files, err := s.files()
+	if err != nil {
+		return nil, err
+	}
+	for _, f := range files {
+		s.size += f.size
+		s.lines += f.lines
+	}
+	return s, nil
+}
+
+type spoolFile struct {
+	path  string
+	size  int64
+	lines int64
+}
+
+// files lists the batches on disk, oldest first. Unfinished writes (.tmp)
+// are removed.
+func (s *Spool) files() ([]spoolFile, error) {
+	entries, err := os.ReadDir(s.dir)
+	if err != nil {
+		return nil, err
+	}
+	var out []spoolFile
+	for _, e := range entries {
+		name := e.Name()
+		if strings.HasSuffix(name, ".tmp") {
+			_ = os.Remove(filepath.Join(s.dir, name))
+			continue
+		}
+		base, ok := strings.CutSuffix(name, ".ndjson")
+		if !ok {
+			continue
+		}
+		_, n, _ := strings.Cut(base, "-")
+		lines, _ := strconv.ParseInt(n, 10, 64)
+		info, err := e.Info()
+		if err != nil {
+			continue
+		}
+		out = append(out, spoolFile{path: filepath.Join(s.dir, name), size: info.Size(), lines: lines})
+	}
+	// The names start with a fixed-width timestamp: string order is time order.
+	sort.Slice(out, func(i, j int) bool { return out[i].path < out[j].path })
+	return out, nil
+}
+
+// Write saves one batch of `lines` messages.
+func (s *Spool) Write(body []byte, lines int) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if s.size+int64(len(body)) > s.max {
+		return errSpoolFull
+	}
+	s.seq++
+	name := fmt.Sprintf("%020d%04d-%d.ndjson", time.Now().UnixNano(), s.seq%10000, lines)
+	path := filepath.Join(s.dir, name)
+	tmp := path + ".tmp"
+	if err := os.WriteFile(tmp, body, 0o644); err != nil {
+		_ = os.Remove(tmp)
+		return err
+	}
+	if err := os.Rename(tmp, path); err != nil {
+		_ = os.Remove(tmp)
+		return err
+	}
+	s.size += int64(len(body))
+	s.lines += int64(lines)
+	return nil
+}
+
+// Oldest returns the oldest batch, or ok=false when the spool is empty.
+func (s *Spool) Oldest() (f spoolFile, body []byte, ok bool, err error) {
+	files, err := s.files()
+	if err != nil || len(files) == 0 {
+		return f, nil, false, err
+	}
+	f = files[0]
+	body, err = os.ReadFile(f.path)
+	return f, body, err == nil, err
+}
+
+// Remove deletes a batch once VictoriaLogs has taken it.
+func (s *Spool) Remove(f spoolFile) {
+	if err := os.Remove(f.path); err != nil && !os.IsNotExist(err) {
+		return
+	}
+	s.mu.Lock()
+	s.size -= f.size
+	s.lines -= f.lines
+	s.mu.Unlock()
+}
+
+// Pending returns the number of messages and bytes waiting on disk.
+func (s *Spool) Pending() (lines, size int64) {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	return s.lines, s.size
+}

store.go

@@ -8,7 +8,6 @@ import (
 	"errors"
 	"fmt"
 	"io"
-	"log"
 	"net/http"
 	"net/url"
 	"strings"
@@ -17,21 +16,27 @@ import (
 )
 
 // Store sends messages to VictoriaLogs in batches and queries it with LogsQL.
+// Batches VictoriaLogs cannot take go to the disk spool (when enabled) and
+// are sent again, oldest first, once it answers.
 type Store struct {
 	base         string
 	client       *http.Client
 	streamClient *http.Client
 	in           chan *Entry
+	quit         chan struct{} // closed on shutdown: unblocks waiting producers
 	batchSize    int
 	flushEvery   time.Duration
+	spool        *Spool        // nil: no disk buffer
+	spooled      chan struct{} // wakes the replay loop up after a write to the spool
 
 	received atomic.Int64
 	ingested atomic.Int64
 	dropped  atomic.Int64
 	lastErr  atomic.Value // string
+	failing  atomic.Bool  // VictoriaLogs refused or missed the last send
 }
 
-func NewStore(base string, batchSize, queueSize int, flushEvery time.Duration) *Store {
+func NewStore(base string, batchSize, queueSize int, flushEvery time.Duration, spool *Spool) *Store {
 	s := &Store{
 		base:   strings.TrimRight(base, "/"),
 		client: &http.Client{Timeout: 60 * time.Second},
@@ -39,21 +44,34 @@ func NewStore(base string, batchSize, queueSize int, flushEvery time.Duration) *
 		// request context still cancels it when the browser goes away.
 		streamClient: &http.Client{},
 		in:           make(chan *Entry, queueSize),
+		quit:         make(chan struct{}),
 		batchSize:    batchSize,
 		flushEvery:   flushEvery,
+		spool:        spool,
+		spooled:      make(chan struct{}, 1),
 	}
 	s.lastErr.Store("")
 	return s
 }
 
-// Enqueue never blocks: when the queue is full, the message is counted as dropped.
+// Enqueue adds a message to the queue. When the queue is full, a message
+// whose producer can wait (Entry.Wait: Docker, host logs) blocks until there
+// is room; any other one (syslog) is counted as dropped.
 func (s *Store) Enqueue(e *Entry) {
 	s.received.Add(1)
 	select {
 	case s.in <- e:
+		return
 	default:
-		s.dropped.Add(1)
 	}
+	if e.Wait {
+		select {
+		case s.in <- e:
+			return
+		case <-s.quit:
+		}
+	}
+	s.dropped.Add(1)
 }
 
 // Run drains the queue into VictoriaLogs until the context is cancelled,
@@ -62,41 +80,41 @@ func (s *Store) Run(ctx context.Context) {
 	ticker := time.NewTicker(s.flushEvery)
 	defer ticker.Stop()
 	batch := make([]*Entry, 0, s.batchSize)
+	if s.spool != nil {
+		go s.replay(ctx)
+	}
 
-	flush := func() {
+	flush := func(ctx context.Context) {
-		if len(batch) == 0 {
+		if len(batch) > 0 {
-			return
+			s.store(ctx, batch)
-		}
+			clear(batch)
-		if err := s.insert(batch); err != nil {
-			s.dropped.Add(int64(len(batch)))
-			s.lastErr.Store(err.Error())
-			log.Printf("victorialogs: %d messages dropped: %v", len(batch), err)
-		} else {
-			s.ingested.Add(int64(len(batch)))
-			s.lastErr.Store("")
-		}
 			batch = batch[:0]
 		}
+	}
 
 	for {
 		select {
 		case e := <-s.in:
 			batch = append(batch, e)
 			if len(batch) >= s.batchSize {
-				flush()
+				flush(ctx)
 			}
 		case <-ticker.C:
-			flush()
+			flush(ctx)
 		case <-ctx.Done():
+			close(s.quit)
+			// The last batches get one short attempt, then go to the spool.
+			end, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+			defer cancel()
 			for {
 				select {
 				case e := <-s.in:
 					batch = append(batch, e)
 					if len(batch) >= s.batchSize {
-						flush()
+						flush(end)
 					}
 				default:
-					flush()
+					flush(end)
 					return
 				}
 			}
@@ -104,38 +122,179 @@ func (s *Store) Run(ctx context.Context) {
 	}
 }
 
-func (s *Store) insert(batch []*Entry) error {
+// store sends one batch to VictoriaLogs, or to the spool when VictoriaLogs
+// fails or older batches are still waiting there (to keep their order).
+// Entry.Done is called once the batch is stored or spooled.
+func (s *Store) store(ctx context.Context, batch []*Entry) {
+	body, err := encodeBatch(batch)
+	if err == nil {
+		err = s.save(ctx, body, len(batch))
+	}
+	if err != nil {
+		s.dropped.Add(int64(len(batch)))
+		s.lastErr.Store(err.Error())
+		logEvent("store", 3, "%d messages lost: %v", len(batch), err)
+		return
+	}
+	for _, e := range batch {
+		if e.Done != nil {
+			e.Done()
+		}
+	}
+}
+
+func (s *Store) save(ctx context.Context, body []byte, lines int) error {
+	if s.spool == nil {
+		// No disk buffer: retry a few times, then give up.
+		var err error
+		for attempt := 0; attempt < 5; attempt++ {
+			if attempt > 0 && !sleepCtx(ctx, time.Duration(1<<attempt)*500*time.Millisecond) { // 1s, 2s, 4s, 8s
+				break
+			}
+			if err = s.post(ctx, body); err == nil {
+				s.ingested.Add(int64(lines))
+				s.sent()
+				return nil
+			}
+			s.sendFailed(err)
+		}
+		return err
+	}
+
+	var postErr error
+	if waiting, _ := s.spool.Pending(); waiting == 0 {
+		if postErr = s.post(ctx, body); postErr == nil {
+			s.ingested.Add(int64(lines))
+			s.sent()
+			return nil
+		}
+		s.sendFailed(postErr)
+	}
+	err := s.spool.Write(body, lines)
+	if err == nil {
+		select {
+		case s.spooled <- struct{}{}:
+		default:
+		}
+		return nil
+	}
+	if postErr != nil {
+		return fmt.Errorf("%v; %v", postErr, err)
+	}
+	// Spool full while older batches wait: one direct attempt before dropping.
+	if postErr = s.post(ctx, body); postErr == nil {
+		s.ingested.Add(int64(lines))
+		return nil
+	}
+	return fmt.Errorf("%v; %v", err, postErr)
+}
+
+// replay sends the spooled batches back to VictoriaLogs, oldest first, with
+// an increasing pause (up to 30 s) while it keeps failing.
+func (s *Store) replay(ctx context.Context) {
+	if n, size := s.spool.Pending(); n > 0 {
+		logEvent("store", 5, "disk spool: %d messages (%d bytes) waiting from a previous run", n, size)
+	}
+	backoff := time.Second
+	for {
+		f, body, ok, err := s.spool.Oldest()
+		if err != nil {
+			logEvent("store", 4, "disk spool: %v", err)
+		}
+		if !ok {
+			select {
+			case <-ctx.Done():
+				return
+			case <-s.spooled:
+			case <-time.After(time.Minute):
+			}
+			continue
+		}
+		err = s.post(ctx, body)
+		switch {
+		case err == nil:
+			s.spool.Remove(f)
+			s.ingested.Add(f.lines)
+			s.sent()
+			backoff = time.Second
+			continue
+		case isRejected(err):
+			// VictoriaLogs refuses the data itself: sending it again would not help.
+			s.spool.Remove(f)
+			s.dropped.Add(f.lines)
+			logEvent("store", 3, "disk spool: %d messages refused by VictoriaLogs and lost: %v", f.lines, err)
+			continue
+		}
+		s.sendFailed(err)
+		if !sleepCtx(ctx, backoff) {
+			return
+		}
+		backoff = min(2*backoff, 30*time.Second)
+	}
+}
+
+// sendFailed and sent report when VictoriaLogs stops and starts taking the
+// batches again, once per change.
+func (s *Store) sendFailed(err error) {
+	s.lastErr.Store(err.Error())
+	if !s.failing.Swap(true) {
+		if s.spool != nil {
+			logEvent("store", 4, "VictoriaLogs unreachable, batches are kept in the disk spool: %v", err)
+		} else {
+			logEvent("store", 4, "VictoriaLogs unreachable, retrying: %v", err)
+		}
+	}
+}
+
+func (s *Store) sent() {
+	s.lastErr.Store("")
+	if s.failing.Swap(false) {
+		logEvent("store", 5, "VictoriaLogs reachable again")
+	}
+}
+
+func encodeBatch(batch []*Entry) ([]byte, error) {
 	var buf bytes.Buffer
 	enc := json.NewEncoder(&buf)
 	enc.SetEscapeHTML(false)
 	for _, e := range batch {
 		if err := enc.Encode(e.Record()); err != nil {
-			return err
+			return nil, err
 		}
 	}
-	u := s.base + "/insert/jsonline?_stream_fields=host,app&_msg_field=_msg&_time_field=_time"
+	return buf.Bytes(), nil
-
-	var err error
-	for attempt := 0; attempt < 5; attempt++ {
-		if attempt > 0 {
-			time.Sleep(time.Duration(1<<attempt) * 500 * time.Millisecond) // 1s, 2s, 4s, 8s
-		}
-		if err = s.post(u, buf.Bytes()); err == nil {
-			return nil
-		}
-	}
-	return err
 }
 
-func (s *Store) post(u string, body []byte) error {
+// insertError is an error status of VictoriaLogs to an insert.
-	resp, err := s.client.Post(u, "application/stream+json", bytes.NewReader(body))
+type insertError struct {
+	status int
+	msg    string
+}
+
+func (e *insertError) Error() string { return fmt.Sprintf("HTTP %d: %s", e.status, e.msg) }
+
+// isRejected tells whether VictoriaLogs refused the data itself (4xx other
+// than 429), as opposed to being unreachable or overloaded.
+func isRejected(err error) bool {
+	var ie *insertError
+	return errors.As(err, &ie) && ie.status >= 400 && ie.status < 500 && ie.status != http.StatusTooManyRequests
+}
+
+func (s *Store) post(ctx context.Context, body []byte) error {
+	u := s.base + "/insert/jsonline?_stream_fields=host,app&_msg_field=_msg&_time_field=_time"
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost, u, bytes.NewReader(body))
+	if err != nil {
+		return err
+	}
+	req.Header.Set("Content-Type", "application/stream+json")
+	resp, err := s.client.Do(req)
 	if err != nil {
 		return err
 	}
 	defer resp.Body.Close()
 	if resp.StatusCode/100 != 2 {
 		msg, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
-		return fmt.Errorf("HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(msg)))
+		return &insertError{status: resp.StatusCode, msg: strings.TrimSpace(string(msg))}
 	}
 	_, _ = io.Copy(io.Discard, resp.Body)
 	return nil
@@ -265,11 +424,15 @@ func queryStatus(err error) int {
 }
 
 func (s *Store) Stats() map[string]any {
-	return map[string]any{
+	st := map[string]any{
 		"received":  s.received.Load(),
 		"ingested":  s.ingested.Load(),
 		"dropped":   s.dropped.Load(),
 		"queue":     len(s.in),
 		"lastError": s.lastErr.Load(),
 	}
+	if s.spool != nil {
+		st["spooled"], st["spoolBytes"] = s.spool.Pending()
+	}
+	return st
 }

store_test.go

@@ -0,0 +1,216 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"errors"
+	"io"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+)
+
+// insertVL plays VictoriaLogs' insert endpoint: it answers `status` (0 = the
+// connection fails) and keeps the lines it accepted.
+type insertVL struct {
+	mu     sync.Mutex
+	status int
+	lines  []string
+}
+
+func (v *insertVL) set(status int) {
+	v.mu.Lock()
+	v.status = status
+	v.mu.Unlock()
+}
+
+func (v *insertVL) got() []string {
+	v.mu.Lock()
+	defer v.mu.Unlock()
+	return append([]string(nil), v.lines...)
+}
+
+func (v *insertVL) RoundTrip(r *http.Request) (*http.Response, error) {
+	body, _ := io.ReadAll(r.Body)
+	v.mu.Lock()
+	defer v.mu.Unlock()
+	if v.status == 0 {
+		return nil, errors.New("connection refused")
+	}
+	if v.status == http.StatusOK {
+		for _, l := range strings.Split(strings.TrimSpace(string(body)), "\n") {
+			v.lines = append(v.lines, l)
+		}
+	}
+	return &http.Response{StatusCode: v.status, Body: io.NopCloser(strings.NewReader("")), Header: http.Header{}}, nil
+}
+
+func testEntry(msg string, done *atomic.Int64) *Entry {
+	e := &Entry{Received: time.Now(), Time: time.Now(), Host: "h", App: "a", Message: msg}
+	if done != nil {
+		e.Done = func() { done.Add(1) }
+	}
+	return e
+}
+
+func waitFor(t *testing.T, what string, cond func() bool) {
+	t.Helper()
+	for deadline := time.Now().Add(5 * time.Second); time.Now().Before(deadline); time.Sleep(10 * time.Millisecond) {
+		if cond() {
+			return
+		}
+	}
+	t.Fatalf("timed out waiting for %s", what)
+}
+
+func TestSpoolFiles(t *testing.T) {
+	dir := t.TempDir()
+	sp, err := OpenSpool(dir, 100)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := sp.Write([]byte("a\nb\n"), 2); err != nil {
+		t.Fatal(err)
+	}
+	if err := sp.Write([]byte("c\n"), 1); err != nil {
+		t.Fatal(err)
+	}
+	if err := sp.Write(bytes.Repeat([]byte("x"), 100), 1); !errors.Is(err, errSpoolFull) {
+		t.Fatalf("write over the limit: %v, want errSpoolFull", err)
+	}
+	_ = os.WriteFile(filepath.Join(dir, "broken.ndjson.tmp"), []byte("z"), 0o644)
+
+	// A new run finds the batches left on disk and drops unfinished writes.
+	sp, err = OpenSpool(dir, 100)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if n, size := sp.Pending(); n != 3 || size != 6 {
+		t.Fatalf("pending %d lines %d bytes, want 3 and 6", n, size)
+	}
+	f, body, ok, err := sp.Oldest()
+	if !ok || err != nil || string(body) != "a\nb\n" || f.lines != 2 {
+		t.Fatalf("oldest: %q %+v %v %v", body, f, ok, err)
+	}
+	sp.Remove(f)
+	if _, body, _, _ := sp.Oldest(); string(body) != "c\n" {
+		t.Fatalf("next oldest %q", body)
+	}
+	if _, err := os.Stat(filepath.Join(dir, "broken.ndjson.tmp")); !os.IsNotExist(err) {
+		t.Error("unfinished write left in the spool")
+	}
+}
+
+func TestStoreSpoolsWhileVictoriaLogsIsDown(t *testing.T) {
+	sp, err := OpenSpool(t.TempDir(), 1<<20)
+	if err != nil {
+		t.Fatal(err)
+	}
+	vl := &insertVL{status: 0}
+	s := NewStore("http://vl", 2, 100, 20*time.Millisecond, sp)
+	s.client.Transport = vl
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	go s.Run(ctx)
+
+	var done atomic.Int64
+	for _, m := range []string{"one", "two", "three"} {
+		s.Enqueue(testEntry(m, &done))
+	}
+	// VictoriaLogs is down: the messages are kept on disk, and acknowledged.
+	waitFor(t, "3 spooled messages", func() bool { n, _ := sp.Pending(); return n == 3 })
+	if done.Load() != 3 || s.dropped.Load() != 0 {
+		t.Fatalf("done %d dropped %d, want 3 and 0", done.Load(), s.dropped.Load())
+	}
+	// While batches wait on disk, new ones queue behind them.
+	vl.set(http.StatusServiceUnavailable)
+	s.Enqueue(testEntry("four", &done))
+	waitFor(t, "4 spooled messages", func() bool { n, _ := sp.Pending(); return n == 4 })
+
+	vl.set(http.StatusOK)
+	waitFor(t, "the spool to drain", func() bool { n, _ := sp.Pending(); return n == 0 })
+	got := strings.Join(vl.got(), "\n")
+	for i, m := range []string{"one", "two", "three", "four"} {
+		if !strings.Contains(got, `"_msg":"`+m+`"`) {
+			t.Errorf("message %d %q not sent: %s", i, m, got)
+		}
+	}
+	if strings.Index(got, `"one"`) > strings.Index(got, `"four"`) {
+		t.Error("spooled batches sent out of order")
+	}
+	if s.ingested.Load() != 4 || s.lastErr.Load() != "" {
+		t.Errorf("ingested %d lastErr %q", s.ingested.Load(), s.lastErr.Load())
+	}
+
+	// Once the spool is empty, batches go straight to VictoriaLogs again.
+	s.Enqueue(testEntry("five", &done))
+	waitFor(t, "a direct insert", func() bool { return s.ingested.Load() == 5 })
+}
+
+func TestStoreDropsRefusedSpooledBatch(t *testing.T) {
+	sp, _ := OpenSpool(t.TempDir(), 1<<20)
+	_ = sp.Write([]byte("{bad json\n"), 1)
+	vl := &insertVL{status: http.StatusBadRequest}
+	s := NewStore("http://vl", 10, 10, time.Second, sp)
+	s.client.Transport = vl
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	go s.Run(ctx)
+	waitFor(t, "the refused batch to be dropped", func() bool { return s.dropped.Load() == 1 })
+	if n, _ := sp.Pending(); n != 0 {
+		t.Errorf("%d messages left in the spool", n)
+	}
+}
+
+func TestStoreWithoutSpoolDrops(t *testing.T) {
+	vl := &insertVL{status: 0}
+	s := NewStore("http://vl", 10, 10, time.Hour, nil)
+	s.client.Transport = vl
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel() // shutdown: one attempt, no retry pauses
+	var done atomic.Int64
+	s.store(ctx, []*Entry{testEntry("x", &done)})
+	if done.Load() != 0 || s.dropped.Load() != 1 {
+		t.Errorf("done %d dropped %d, want 0 and 1", done.Load(), s.dropped.Load())
+	}
+}
+
+func TestEnqueueWait(t *testing.T) {
+	s := NewStore("http://vl", 10, 1, time.Hour, nil)
+	s.Enqueue(testEntry("fills the queue", nil))
+	s.Enqueue(testEntry("syslog: dropped", nil))
+	if s.dropped.Load() != 1 {
+		t.Fatalf("dropped %d, want 1", s.dropped.Load())
+	}
+	// A producer that can wait blocks until there is room…
+	queued := make(chan struct{})
+	go func() {
+		e := testEntry("docker: waits", nil)
+		e.Wait = true
+		s.Enqueue(e)
+		close(queued)
+	}()
+	select {
+	case <-queued:
+		t.Fatal("did not wait for room in the queue")
+	case <-time.After(50 * time.Millisecond):
+	}
+	<-s.in
+	<-queued
+	if s.dropped.Load() != 1 {
+		t.Errorf("dropped %d, want 1", s.dropped.Load())
+	}
+	// …or until shutdown.
+	close(s.quit)
+	e := testEntry("docker: shutdown", nil)
+	e.Wait = true
+	s.Enqueue(e)
+	if s.dropped.Load() != 2 {
+		t.Errorf("dropped %d after shutdown, want 2", s.dropped.Load())
+	}
+}

syslog.go

@@ -36,8 +36,14 @@ type Entry struct {
 	Proto    string // udp or tcp
 	HostIP   string // original host value when it was an IP resolved through DNS
 
-	SourceType string            // "syslog" or "docker"
+	SourceType string            // "syslog", "docker", "host", "kernel", "audit", "file" or "logstream"
 	Extra      map[string]string // additional fields (docker: container, image, …)
+
+	// Not stored. Wait: the producer can be slowed down when the queue is full
+	// (Docker, host logs) instead of losing the message. Done: called once the
+	// message is stored in VictoriaLogs or in the disk spool.
+	Wait bool
+	Done func()
 }
 
 // Record returns the entry in the shape stored in VictoriaLogs and returned by the API.
@@ -233,7 +239,7 @@ func serveUDP(ctx context.Context, pc net.PacketConn, sink func(*Entry)) {
 			if ctx.Err() != nil {
 				return
 			}
-			log.Printf("syslog udp: %v", err)
+			logEvent("syslog", 4, "UDP read error: %v", err)
 			time.Sleep(100 * time.Millisecond)
 			continue
 		}
@@ -241,19 +247,49 @@ func serveUDP(ctx context.Context, pc net.PacketConn, sink func(*Entry)) {
 	}
 }
 
+// TCP limits: connections open at once, and how long a connection may stay
+// silent before it is closed (senders reconnect on their own).
+var (
+	tcpMaxConns = 512
+	tcpIdle     = 30 * time.Minute
+)
+
 func serveTCP(ctx context.Context, ln net.Listener, sink func(*Entry)) {
+	slots := make(chan struct{}, tcpMaxConns)
 	for {
 		conn, err := ln.Accept()
 		if err != nil {
 			if ctx.Err() != nil {
 				return
 			}
-			log.Printf("syslog tcp: %v", err)
+			logEvent("syslog", 4, "TCP accept error: %v", err)
 			time.Sleep(100 * time.Millisecond)
 			continue
 		}
-		go handleTCP(ctx, conn, sink)
+		select {
+		case slots <- struct{}{}:
+		default:
+			logEvent("syslog", 4, "%d TCP connections already open, refusing %s", tcpMaxConns, conn.RemoteAddr())
+			conn.Close()
+			continue
 		}
+		go func() {
+			defer func() { <-slots }()
+			handleTCP(ctx, conn, sink)
+		}()
+	}
+}
+
+// idleConn pushes the read deadline back before each read, so only a
+// connection that stays silent for tcpIdle is closed.
+type idleConn struct {
+	net.Conn
+	idle time.Duration
+}
+
+func (c idleConn) Read(p []byte) (int, error) {
+	_ = c.Conn.SetReadDeadline(time.Now().Add(c.idle))
+	return c.Conn.Read(p)
 }
 
 const maxFrame = 1 << 20
@@ -266,7 +302,7 @@ func handleTCP(ctx context.Context, conn net.Conn, sink func(*Entry)) {
 	defer stop()
 
 	src := hostOf(conn.RemoteAddr())
-	r := bufio.NewReaderSize(conn, 64*1024)
+	r := bufio.NewReaderSize(idleConn{conn, tcpIdle}, 64*1024)
 	for {
 		c, err := r.ReadByte()
 		if err != nil {

syslogserver.go

@@ -94,7 +94,7 @@ func (s *SyslogServer) apply() {
 		pc, err := net.ListenPacket("udp", s.addr)
 		if err != nil {
 			s.errUDP = err.Error()
-			log.Printf("syslog udp: %v", err)
+			logEvent("syslog", 3, "UDP listener failed: %v", err)
 		} else {
 			if uc, ok := pc.(*net.UDPConn); ok {
 				_ = uc.SetReadBuffer(8 << 20)
@@ -108,7 +108,7 @@ func (s *SyslogServer) apply() {
 		ln, err := net.Listen("tcp", s.addr)
 		if err != nil {
 			s.errTCP = err.Error()
-			log.Printf("syslog tcp: %v", err)
+			logEvent("syslog", 3, "TCP listener failed: %v", err)
 		} else {
 			s.ln = ln
 			go serveTCP(ctx, ln, s.sink)

tags.go

@@ -16,7 +16,9 @@ import (
 // Tag highlights a keyword in displayed messages.
 type Tag struct {
 	ID            string `json:"id"`
+	Code          string `json:"code"` // two digits, shown on matching log lines
 	Pattern       string `json:"pattern"`
+	Label         string `json:"label,omitempty"` // shown instead of the pattern (presets)
 	Color         string `json:"color"`
 	WholeWord     bool   `json:"wholeWord"`
 	CaseSensitive bool   `json:"caseSensitive"`
@@ -26,12 +28,46 @@ type Tag struct {
 
 func defaultTags() []Tag {
 	return []Tag{
-		{ID: "warning", Pattern: "warning", Color: "#fdba74", WholeWord: true, Enabled: true},
+		{ID: "warning", Code: "01", Pattern: "warning", Color: "#fdba74", WholeWord: true, Enabled: true},
-		{ID: "error", Pattern: "error", Color: "#fca5a5", WholeWord: true, Enabled: true},
+		{ID: "error", Code: "02", Pattern: "error", Color: "#fca5a5", WholeWord: true, Enabled: true},
-		{ID: "ok", Pattern: "ok", Color: "#86efac", WholeWord: true, Enabled: true},
 	}
 }
 
+var codeRe = regexp.MustCompile(`^[0-9]{2}$`)
+
+// freeCode returns the lowest code from 01 to 99 not used by tags, or "" when
+// all are taken. A code stays with its tag until the tag is deleted.
+func freeCode(tags []Tag) string {
+	used := map[string]bool{}
+	for _, t := range tags {
+		used[t.Code] = true
+	}
+	for n := 1; n <= 99; n++ {
+		if c := fmt.Sprintf("%02d", n); !used[c] {
+			return c
+		}
+	}
+	return ""
+}
+
+// assignCodes gives a code to the tags that have none (files written before
+// codes existed) or share one with an earlier tag.
+func assignCodes(tags []Tag) bool {
+	changed := false
+	seen := map[string]bool{}
+	for i := range tags {
+		if codeRe.MatchString(tags[i].Code) && !seen[tags[i].Code] {
+			seen[tags[i].Code] = true
+			continue
+		}
+		tags[i].Code = ""
+		tags[i].Code = freeCode(tags)
+		seen[tags[i].Code] = true
+		changed = true
+	}
+	return changed
+}
+
 // Colors of the default tags in earlier versions: still unchanged, they are
 // switched to the new pastel defaults when the file is loaded.
 var oldDefaultColors = map[string]string{"warning": "#f59e0b", "error": "#ef4444", "ok": "#22c55e"}
@@ -67,11 +103,13 @@ func (e *codedError) Error() string {
 
 var (
 	errTagNotFound = &codedError{code: "tag_not_found", msg: "tag not found"}
+	errTooManyTags = &codedError{code: "too_many_tags", msg: "too many tags (99 at most)"}
 	colorRe        = regexp.MustCompile(`^#[0-9a-fA-F]{6}$`)
 )
 
 func (t *Tag) validate() error {
 	t.Pattern = strings.TrimSpace(t.Pattern)
+	t.Label = strings.TrimSpace(t.Label)
 	if t.Pattern == "" {
 		return &codedError{code: "pattern_required", msg: "the keyword is required"}
 	}
@@ -106,7 +144,7 @@ func LoadTagStore(path string) (*TagStore, error) {
 	if err := json.Unmarshal(b, &s.tags); err != nil {
 		return nil, fmt.Errorf("%s: %w", path, err)
 	}
-	if migrateDefaultColors(s.tags) {
+	if c1, c2 := migrateDefaultColors(s.tags), assignCodes(s.tags); c1 || c2 {
 		if err := s.save(); err != nil {
 			return nil, err
 		}
@@ -143,6 +181,9 @@ func (s *TagStore) Create(t Tag) (Tag, error) {
 	t.ID = newID()
 	s.mu.Lock()
 	defer s.mu.Unlock()
+	if t.Code = freeCode(s.tags); t.Code == "" {
+		return t, errTooManyTags
+	}
 	s.tags = append(s.tags, t)
 	return t, s.save()
 }
@@ -156,6 +197,7 @@ func (s *TagStore) Update(id string, t Tag) (Tag, error) {
 	defer s.mu.Unlock()
 	for i := range s.tags {
 		if s.tags[i].ID == id {
+			t.Code = s.tags[i].Code // assigned by the server, never changed
 			s.tags[i] = t
 			return t, s.save()
 		}

tags_test.go

@@ -0,0 +1,54 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"slices"
+	"testing"
+)
+
+func TestTagCodes(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "tags.json")
+	// File written before codes existed, with a duplicate code.
+	old := `[{"id":"a","pattern":"x","color":"#000000"},{"id":"b","code":"07","pattern":"y","color":"#000000"},{"id":"c","code":"07","pattern":"z","color":"#000000"}]`
+	if err := os.WriteFile(path, []byte(old), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	s, err := LoadTagStore(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	got := []string{}
+	for _, tg := range s.List() {
+		got = append(got, tg.Code)
+	}
+	if want := []string{"01", "07", "02"}; !slices.Equal(got, want) {
+		t.Fatalf("migrated codes = %v, want %v", got, want)
+	}
+
+	n, err := s.Create(Tag{Pattern: "w", Color: "#000000"})
+	if err != nil || n.Code != "03" {
+		t.Fatalf("Create code = %q, %v; want 03", n.Code, err)
+	}
+	// The client cannot change a code.
+	u, err := s.Update("b", Tag{Code: "42", Pattern: "y2", Color: "#000000"})
+	if err != nil || u.Code != "07" {
+		t.Fatalf("Update code = %q, %v; want 07", u.Code, err)
+	}
+	// A deleted tag frees its code; the others keep theirs.
+	if err := s.Delete("a"); err != nil {
+		t.Fatal(err)
+	}
+	if n, _ := s.Create(Tag{Pattern: "v", Color: "#000000"}); n.Code != "01" {
+		t.Fatalf("code after delete = %q, want 01", n.Code)
+	}
+
+	// Codes are saved in the file.
+	s2, err := LoadTagStore(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got := s2.List()[0].Code; got != "07" {
+		t.Fatalf("reloaded code = %q, want 07", got)
+	}
+}

web/fonts/OFL-Inconsolata.txt

@@ -0,0 +1,93 @@
+Copyright 2006 The Inconsolata Project Authors (https://github.com/cyrealtype/Inconsolata)
+
+This Font Software is licensed under the SIL Open Font License, Version 1.1.
+This license is copied below, and is also available with a FAQ at:
+http://scripts.sil.org/OFL
+
+
+-----------------------------------------------------------
+SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
+-----------------------------------------------------------
+
+PREAMBLE
+The goals of the Open Font License (OFL) are to stimulate worldwide
+development of collaborative font projects, to support the font creation
+efforts of academic and linguistic communities, and to provide a free and
+open framework in which fonts may be shared and improved in partnership
+with others.
+
+The OFL allows the licensed fonts to be used, studied, modified and
+redistributed freely as long as they are not sold by themselves. The
+fonts, including any derivative works, can be bundled, embedded,
+redistributed and/or sold with any software provided that any reserved
+names are not used by derivative works. The fonts and derivatives,
+however, cannot be released under any other type of license. The
+requirement for fonts to remain under this license does not apply
+to any document created using the fonts or their derivatives.
+
+DEFINITIONS
+"Font Software" refers to the set of files released by the Copyright
+Holder(s) under this license and clearly marked as such. This may
+include source files, build scripts and documentation.
+
+"Reserved Font Name" refers to any names specified as such after the
+copyright statement(s).
+
+"Original Version" refers to the collection of Font Software components as
+distributed by the Copyright Holder(s).
+
+"Modified Version" refers to any derivative made by adding to, deleting,
+or substituting -- in part or in whole -- any of the components of the
+Original Version, by changing formats or by porting the Font Software to a
+new environment.
+
+"Author" refers to any designer, engineer, programmer, technical
+writer or other person who contributed to the Font Software.
+
+PERMISSION & CONDITIONS
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of the Font Software, to use, study, copy, merge, embed, modify,
+redistribute, and sell modified and unmodified copies of the Font
+Software, subject to the following conditions:
+
+1) Neither the Font Software nor any of its individual components,
+in Original or Modified Versions, may be sold by itself.
+
+2) Original or Modified Versions of the Font Software may be bundled,
+redistributed and/or sold with any software, provided that each copy
+contains the above copyright notice and this license. These can be
+included either as stand-alone text files, human-readable headers or
+in the appropriate machine-readable metadata fields within text or
+binary files as long as those fields can be easily viewed by the user.
+
+3) No Modified Version of the Font Software may use the Reserved Font
+Name(s) unless explicit written permission is granted by the corresponding
+Copyright Holder. This restriction only applies to the primary font name as
+presented to the users.
+
+4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
+Software shall not be used to promote, endorse or advertise any
+Modified Version, except to acknowledge the contribution(s) of the
+Copyright Holder(s) and the Author(s) or with their explicit written
+permission.
+
+5) The Font Software, modified or unmodified, in part or in whole,
+must be distributed entirely under this license, and must not be
+distributed under any other license. The requirement for fonts to
+remain under this license does not apply to any document created
+using the Font Software.
+
+TERMINATION
+This license becomes null and void if any of the above conditions are
+not met.
+
+DISCLAIMER
+THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
+OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
+DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
+OTHER DEALINGS IN THE FONT SOFTWARE.

web/fonts/OFL-Iosevka.txt

@@ -0,0 +1,110 @@
+Copyright (c) 2015-2023, Renzhi Li (aka. Belleve Invis, belleve@typeof.net)
+
+This Font Software is licensed under the SIL Open Font License, Version 1.1.
+
+This license is copied below, and is also available with a FAQ at:
+http://scripts.sil.org/OFL
+
+--------------------------
+
+
+SIL Open Font License v1.1
+====================================================
+
+
+Preamble
+----------
+
+The goals of the Open Font License (OFL) are to stimulate worldwide
+development of collaborative font projects, to support the font creation
+efforts of academic and linguistic communities, and to provide a free and
+open framework in which fonts may be shared and improved in partnership
+with others.
+
+The OFL allows the licensed fonts to be used, studied, modified and
+redistributed freely as long as they are not sold by themselves. The
+fonts, including any derivative works, can be bundled, embedded, 
+redistributed and/or sold with any software provided that any reserved
+names are not used by derivative works. The fonts and derivatives,
+however, cannot be released under any other type of license. The
+requirement for fonts to remain under this license does not apply
+to any document created using the fonts or their derivatives.
+
+
+Definitions
+-------------
+
+`"Font Software"` refers to the set of files released by the Copyright
+Holder(s) under this license and clearly marked as such. This may
+include source files, build scripts and documentation.
+
+`"Reserved Font Name"` refers to any names specified as such after the
+copyright statement(s).
+
+`"Original Version"` refers to the collection of Font Software components as
+distributed by the Copyright Holder(s).
+
+`"Modified Version"` refers to any derivative made by adding to, deleting,
+or substituting -- in part or in whole -- any of the components of the
+Original Version, by changing formats or by porting the Font Software to a
+new environment.
+
+`"Author"` refers to any designer, engineer, programmer, technical
+writer or other person who contributed to the Font Software.
+
+
+Permission & Conditions
+------------------------
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of the Font Software, to use, study, copy, merge, embed, modify,
+redistribute, and sell modified and unmodified copies of the Font
+Software, subject to the following conditions:
+
+1. Neither the Font Software nor any of its individual components,
+   in Original or Modified Versions, may be sold by itself.
+
+2. Original or Modified Versions of the Font Software may be bundled,
+   redistributed and/or sold with any software, provided that each copy
+   contains the above copyright notice and this license. These can be
+   included either as stand-alone text files, human-readable headers or
+   in the appropriate machine-readable metadata fields within text or
+   binary files as long as those fields can be easily viewed by the user.
+
+3. No Modified Version of the Font Software may use the Reserved Font
+   Name(s) unless explicit written permission is granted by the corresponding
+   Copyright Holder. This restriction only applies to the primary font name as
+   presented to the users.
+
+4. The name(s) of the Copyright Holder(s) or the Author(s) of the Font
+   Software shall not be used to promote, endorse or advertise any
+   Modified Version, except to acknowledge the contribution(s) of the
+   Copyright Holder(s) and the Author(s) or with their explicit written
+   permission.
+
+5. The Font Software, modified or unmodified, in part or in whole,
+   must be distributed entirely under this license, and must not be
+   distributed under any other license. The requirement for fonts to
+   remain under this license does not apply to any document created
+   using the Font Software.
+
+
+
+Termination
+-----------
+
+This license becomes null and void if any of the above conditions are
+not met.
+
+
+    DISCLAIMER
+    
+    THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+    EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
+    MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
+    OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
+    COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+    INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
+    DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+    FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
+    OTHER DEALINGS IN THE FONT SOFTWARE.

web/fonts/UFL-Ubuntu-Mono.txt

@@ -0,0 +1,96 @@
+-------------------------------
+UBUNTU FONT LICENCE Version 1.0
+-------------------------------
+
+PREAMBLE
+This licence allows the licensed fonts to be used, studied, modified and
+redistributed freely. The fonts, including any derivative works, can be
+bundled, embedded, and redistributed provided the terms of this licence
+are met. The fonts and derivatives, however, cannot be released under
+any other licence. The requirement for fonts to remain under this
+licence does not require any document created using the fonts or their
+derivatives to be published under this licence, as long as the primary
+purpose of the document is not to be a vehicle for the distribution of
+the fonts.
+
+DEFINITIONS
+"Font Software" refers to the set of files released by the Copyright
+Holder(s) under this licence and clearly marked as such. This may
+include source files, build scripts and documentation.
+
+"Original Version" refers to the collection of Font Software components
+as received under this licence.
+
+"Modified Version" refers to any derivative made by adding to, deleting,
+or substituting -- in part or in whole -- any of the components of the
+Original Version, by changing formats or by porting the Font Software to
+a new environment.
+
+"Copyright Holder(s)" refers to all individuals and companies who have a
+copyright ownership of the Font Software.
+
+"Substantially Changed" refers to Modified Versions which can be easily
+identified as dissimilar to the Font Software by users of the Font
+Software comparing the Original Version with the Modified Version.
+
+To "Propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy. Propagation includes copying,
+distribution (with or without modification and with or without charging
+a redistribution fee), making available to the public, and in some
+countries other activities as well.
+
+PERMISSION & CONDITIONS
+This licence does not grant any rights under trademark law and all such
+rights are reserved.
+
+Permission is hereby granted, free of charge, to any person obtaining a
+copy of the Font Software, to propagate the Font Software, subject to
+the below conditions:
+
+1) Each copy of the Font Software must contain the above copyright
+notice and this licence. These can be included either as stand-alone
+text files, human-readable headers or in the appropriate machine-
+readable metadata fields within text or binary files as long as those
+fields can be easily viewed by the user.
+
+2) The font name complies with the following:
+(a) The Original Version must retain its name, unmodified.
+(b) Modified Versions which are Substantially Changed must be renamed to
+avoid use of the name of the Original Version or similar names entirely.
+(c) Modified Versions which are not Substantially Changed must be
+renamed to both (i) retain the name of the Original Version and (ii) add
+additional naming elements to distinguish the Modified Version from the
+Original Version. The name of such Modified Versions must be the name of
+the Original Version, with "derivative X" where X represents the name of
+the new work, appended to that name.
+
+3) The name(s) of the Copyright Holder(s) and any contributor to the
+Font Software shall not be used to promote, endorse or advertise any
+Modified Version, except (i) as required by this licence, (ii) to
+acknowledge the contribution(s) of the Copyright Holder(s) or (iii) with
+their explicit written permission.
+
+4) The Font Software, modified or unmodified, in part or in whole, must
+be distributed entirely under this licence, and must not be distributed
+under any other licence. The requirement for fonts to remain under this
+licence does not affect any document created using the Font Software,
+except any version of the Font Software extracted from a document
+created using the Font Software may only be distributed under this
+licence.
+
+TERMINATION
+This licence becomes null and void if any of the above conditions are
+not met.
+
+DISCLAIMER
+THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF
+COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
+DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM OTHER
+DEALINGS IN THE FONT SOFTWARE.

web/index.html

@@ -3,7 +3,7 @@
 <head>
   <meta charset="utf-8">
   <meta name="viewport" content="width=device-width, initial-scale=1">
-  <title>Logstream</title>
+  <title>LogStream</title>
   <link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'%3E%3Crect width='32' height='32' rx='8' fill='%236366f1'/%3E%3Cpath d='M8 10h16M8 16h11M8 22h14' stroke='white' stroke-width='3' stroke-linecap='round'/%3E%3C/svg%3E">
   <link rel="stylesheet" href="style.css">
   <script>
@@ -20,7 +20,7 @@
   <header class="topbar">
     <div class="brand">
       <svg viewBox="0 0 32 32" aria-hidden="true"><rect width="32" height="32" rx="8"/><path d="M8 10h16M8 16h11M8 22h14"/></svg>
-      <span>Logstream</span>
+      <span>LogStream</span>
     </div>
 
     <div class="search">
@@ -41,11 +41,43 @@
         <svg class="sun" viewBox="0 0 24 24"><circle cx="12" cy="12" r="4"/><path d="M12 2v2M12 20v2M4.9 4.9l1.4 1.4M17.7 17.7l1.4 1.4M2 12h2M20 12h2M4.9 19.1l1.4-1.4M17.7 6.3l1.4-1.4"/></svg>
         <svg class="moon" viewBox="0 0 24 24"><path d="M21 12.8A9 9 0 1 1 11.2 3a7 7 0 0 0 9.8 9.8z"/></svg>
       </button>
+      <span id="whoami" class="whoami" hidden><span class="who-name"></span><span class="who-badge"></span></span>
+      <a id="logoutBtn" class="icon-btn" href="/auth/logout" hidden data-i18n-title="logout" data-i18n-aria="logout">
+        <!-- Log out icon (Lucide "log-out", ISC license) -->
+        <svg viewBox="0 0 24 24"><path d="M9 21H5a2 2 0 0 1-2-2V5a2 2 0 0 1 2-2h4"/><path d="m16 17 5-5-5-5"/><path d="M21 12H9"/></svg>
+      </a>
     </div>
     <div class="progress" aria-hidden="true"></div>
   </header>
 
   <section class="filters">
+    <div id="viewMode" class="seg view-mode" role="radiogroup" data-i18n-aria="viewAria">
+      <button type="button" role="radio" data-view="stream" data-i18n-title="viewStreamTitle">
+        <!-- Lucide "radio" (ISC license) -->
+        <svg viewBox="0 0 24 24" aria-hidden="true"><path d="M4.9 19.1C1 15.2 1 8.8 4.9 4.9"/><path d="M7.8 16.2c-2.3-2.3-2.3-6.1 0-8.5"/><circle cx="12" cy="12" r="2"/><path d="M16.2 7.8c2.3 2.3 2.3 6.1 0 8.5"/><path d="M19.1 4.9C23 8.8 23 15.1 19.1 19"/></svg>
+        <span data-i18n="viewStream">Stream</span>
+      </button>
+      <button type="button" role="radio" data-view="period" data-i18n-title="viewPeriodTitle">
+        <!-- Lucide "calendar-range" (ISC license) -->
+        <svg viewBox="0 0 24 24" aria-hidden="true"><rect width="18" height="18" x="3" y="4" rx="2"/><path d="M16 2v4M3 10h18M8 2v4M17 14h-6M13 18H7M7 14h.01M17 18h.01"/></svg>
+        <span data-i18n="viewPeriod">Time range</span>
+      </button>
+    </div>
+    <div id="period" class="period" hidden>
+      <button id="perPrev" class="btn tool" type="button" data-i18n-title="perPrev" data-i18n-aria="perPrev">
+        <svg viewBox="0 0 24 24" aria-hidden="true"><path d="m15 18-6-6 6-6"/></svg>
+      </button>
+      <input id="perFrom" type="datetime-local" data-i18n-title="perFrom" data-i18n-aria="perFrom">
+      <span class="muted" aria-hidden="true">→</span>
+      <input id="perTo" type="datetime-local" data-i18n-title="perTo" data-i18n-aria="perTo">
+      <button id="perNext" class="btn tool" type="button" data-i18n-title="perNext" data-i18n-aria="perNext">
+        <svg viewBox="0 0 24 24" aria-hidden="true"><path d="m9 18 6-6-6-6"/></svg>
+      </button>
+      <button id="perOut" class="btn tool" type="button" data-i18n-title="perOut" data-i18n-aria="perOut">
+        <!-- Lucide "zoom-out" (ISC license) -->
+        <svg viewBox="0 0 24 24" aria-hidden="true"><circle cx="11" cy="11" r="8"/><path d="m21 21-4.3-4.3M8 11h6"/></svg>
+      </button>
+    </div>
     <select id="range" data-i18n-aria="rangeAria">
       <option value="5m" data-i18n="r5m">5 min</option>
       <option value="15m" data-i18n="r15m">15 min</option>
@@ -70,6 +102,11 @@
       <option value="" data-i18n="srcAll">All sources</option>
       <option value="syslog">Syslog</option>
       <option value="docker">Docker</option>
+      <option value="host" data-i18n="srcHost">Host system</option>
+      <option value="kernel" data-i18n="srcKernel">Kernel (dmesg)</option>
+      <option value="audit">Audit</option>
+      <option value="file" data-i18n="srcFile">Files</option>
+      <option value="logstream">LogStream</option>
     </select>
     <span class="spacer"></span>
     <span id="count" class="muted"></span>
@@ -92,6 +129,12 @@
 
   <section id="histo" class="histo" data-i18n-aria="histoAria" hidden></section>
 
+  <div id="authWarn" class="auth-warn" role="status" hidden>
+    <span data-i18n="authOff"></span>
+    <button id="authWarnClose" class="icon-btn" type="button" data-i18n-aria="close">
+      <svg viewBox="0 0 24 24"><path d="M6 6l12 12M18 6 6 18"/></svg>
+    </button>
+  </div>
   <div id="error" class="error-banner" hidden></div>
   <button id="newPill" class="pill" type="button" hidden></button>
 
@@ -102,6 +145,7 @@
       <span data-col="sev"><span class="lbl" data-i18n="colSev">Severity</span><i class="grip" data-i18n-title="colGrip"></i></span>
       <span data-col="host"><span class="lbl" data-i18n="colHost">Host</span><i class="grip" data-i18n-title="colGrip"></i></span>
       <span data-col="app"><span class="lbl" data-i18n="colApp">App</span><i class="grip" data-i18n-title="colGrip"></i></span>
+      <span data-col="codes"><span class="lbl" data-i18n="colCodes" data-i18n-title="codesTitle">Pattern</span></span>
       <span data-col="msg"><span class="lbl" data-i18n="colMsg">Message</span></span>
     </div>
     <main id="list" class="list"></main>
@@ -137,19 +181,19 @@
           </button>
           <button type="button" role="tab" data-tab="filters">
             <svg viewBox="0 0 24 24"><path d="M3 12V4a1 1 0 0 1 1-1h8l9 9-9 9z"/><circle cx="8" cy="8" r="1.5"/></svg>
-            <span data-i18n="tabFilters">Filters</span>
+            <span data-i18n="tabFilters">Patterns</span>
           </button>
           <button type="button" role="tab" data-tab="sources">
             <svg viewBox="0 0 24 24"><path d="M21 8 12 3 3 8v8l9 5 9-5z"/><path d="m3 8 9 5 9-5M12 13v8"/></svg>
-            <span data-i18n="tabSources">Sources</span>
+            <span data-i18n="tabSources">Log sources</span>
           </button>
           <button type="button" role="tab" data-tab="interface">
             <svg viewBox="0 0 24 24"><rect x="3" y="4" width="18" height="12" rx="2"/><path d="M8 20h8M12 16v4"/></svg>
-            <span data-i18n="tabInterface">Interface</span>
+            <span data-i18n="tabInterface">Appearance</span>
           </button>
           <button type="button" role="tab" data-tab="data">
             <svg viewBox="0 0 24 24"><ellipse cx="12" cy="5" rx="8" ry="3"/><path d="M4 5v14c0 1.7 3.6 3 8 3s8-1.3 8-3V5M4 12c0 1.7 3.6 3 8 3s8-1.3 8-3"/></svg>
-            <span data-i18n="tabData">Data</span>
+            <span data-i18n="tabData">Database</span>
           </button>
         </nav>
 
@@ -176,16 +220,23 @@
             </section>
           </div>
 
-          <!-- Filters -->
+          <!-- Pattern (color tags; the "filters" tab id is kept for saved settings) -->
-          <div class="set-panel" role="tabpanel" data-panel="filters" hidden>
+          <div class="set-panel wide" role="tabpanel" data-panel="filters" hidden>
-            <section class="set-section">
+            <section class="set-section tags-section">
+              <div class="sec-head">
                 <h3 data-i18n="tagsTitle">Color tags</h3>
+                <span id="tagCount" class="muted small"></span>
+              </div>
               <p class="muted small" data-i18n="tagsHelp"></p>
-              <div id="tagList" class="tag-list"></div>
+              <div class="tag-toolbar">
-              <footer>
                 <button id="addTag" class="btn primary" type="button" data-i18n="addTag">+ Add tag</button>
+                <select id="presetTags" class="field" aria-label="Presets" data-i18n-aria="presetAria">
+                  <option value="" data-i18n="presetPick">+ Preset…</option>
+                </select>
+                <input id="tagFilter" type="search" class="dk-filter" autocomplete="off" spellcheck="false" data-i18n-ph="tagFilter" data-i18n-aria="tagFilter">
                 <button id="resetTags" class="btn ghost" type="button" data-i18n="resetTags">Restore default tags</button>
-              </footer>
+              </div>
+              <div id="tagList" class="tag-list"></div>
             </section>
           </div>
 
@@ -210,6 +261,68 @@
             </section>
 
             <section class="set-section">
+              <h3 data-i18n="hostTitle">Host system logs</h3>
+              <p id="hostStatus" class="docker-status"></p>
+              <label class="switch-row">
+                <input id="hostEnabled" type="checkbox" class="switch">
+                <span data-i18n="hostEnabled">Collect the system logs of this machine</span>
+              </label>
+              <p class="muted small" data-i18n="hostHelp"></p>
+            </section>
+
+            <section class="set-section">
+              <h3 data-i18n="dmesgTitle">Kernel messages (dmesg)</h3>
+              <p id="dmesgStatus" class="docker-status"></p>
+              <label class="switch-row">
+                <input id="dmesgEnabled" type="checkbox" class="switch">
+                <span data-i18n="dmesgEnabled">Collect the kernel messages of this machine</span>
+              </label>
+              <p class="muted small" data-i18n="dmesgHelp"></p>
+            </section>
+
+            <section class="set-section">
+              <h3 data-i18n="auditTitle">Linux audit (auditd)</h3>
+              <p id="auditStatus" class="docker-status"></p>
+              <label class="switch-row">
+                <input id="auditEnabled" type="checkbox" class="switch">
+                <span data-i18n="auditEnabled">Collect the audit events of this machine</span>
+              </label>
+              <div class="sl-grid">
+                <label class="lbl" for="auditPath" data-i18n="auditPath">File</label>
+                <input id="auditPath" type="text" class="dk-filter" autocomplete="off" spellcheck="false">
+              </div>
+              <p class="muted small src-help" data-i18n="auditHelp"></p>
+            </section>
+
+            <section class="set-section">
+              <h3 data-i18n="selfTitle">LogStream internal logs</h3>
+              <p id="selfStatus" class="docker-status"></p>
+              <label class="switch-row">
+                <input id="selfEnabled" type="checkbox" class="switch">
+                <span data-i18n="selfEnabled">Record the events of LogStream itself</span>
+              </label>
+              <div class="sl-grid">
+                <label class="lbl" for="selfLevel" data-i18n="selfLevel">Minimum level</label>
+                <select id="selfLevel" class="field self-level">
+                  <option value="err" data-i18n="sev3">Errors and above</option>
+                  <option value="warning" data-i18n="sev4">Warnings and above</option>
+                  <option value="notice" data-i18n="sev5">Notice and above</option>
+                  <option value="info" data-i18n="sev6">Info and above</option>
+                </select>
+              </div>
+              <p class="muted small src-help" data-i18n="selfHelp"></p>
+            </section>
+
+            <section class="set-section span">
+              <h3 data-i18n="filesTitle">Log files</h3>
+              <div id="fileList" class="file-list"></div>
+              <div class="file-toolbar">
+                <button id="addFile" class="btn primary" type="button" data-i18n="addFile">+ Add a file</button>
+              </div>
+              <p class="muted small src-help" data-i18n="filesHelp"></p>
+            </section>
+
+            <section class="set-section span">
               <h3 data-i18n="dockerTitle">Docker containers</h3>
               <p id="dockerStatus" class="docker-status"></p>
               <div id="dockerBody" hidden>
@@ -246,7 +359,7 @@
               </div>
               <p class="muted small hint" data-i18n="themeHelp"></p>
             </section>
-            <section class="set-section">
+            <section class="set-section span">
               <h3 data-i18n="logDisplay">Log display</h3>
               <div class="field-grid">
                 <span class="lbl" data-i18n="fontSize">Font size</span>
@@ -256,6 +369,11 @@
                   <button type="button" role="radio" data-size="medium" data-i18n="sizeMedium">Medium</button>
                   <button type="button" role="radio" data-size="large" data-i18n="sizeLarge">Large</button>
                 </div>
+                <span class="lbl" data-i18n="density">Density</span>
+                <div id="densitySwitch" class="seg" role="radiogroup" data-i18n-aria="density">
+                  <button type="button" role="radio" data-density="normal" data-i18n="densityNormal">Normal</button>
+                  <button type="button" role="radio" data-density="compact" data-i18n="densityCompact">Compact</button>
+                </div>
                 <label for="fontSelect" data-i18n="fontLabel">Font</label>
                 <select id="fontSelect" class="field"></select>
               </div>
@@ -302,9 +420,14 @@
           </div>
 
           <!-- Data -->
-          <div class="set-panel" role="tabpanel" data-panel="data" hidden>
+          <div class="set-panel wide" role="tabpanel" data-panel="data" hidden>
             <section class="set-section danger">
               <h3 data-i18n="dangerZone">Danger zone</h3>
+              <div class="db-head">
+                <span class="lbl" data-i18n="dbTitle">Database</span>
+                <button id="dbRefresh" class="link-btn" type="button" data-i18n="dbRefresh">Refresh</button>
+              </div>
+              <dl id="dbStats" class="db-stats"></dl>
               <p class="muted small" data-i18n="purgeHelp"></p>
               <div class="purge-row">
                 <button id="purgeBtn" class="btn danger" type="button" data-i18n="purgeBtn">Delete all logs…</button>

web/login.html

@@ -0,0 +1,95 @@
+<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1">
+  <title>LogStream</title>
+  <link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'%3E%3Crect width='32' height='32' rx='8' fill='%236366f1'/%3E%3Cpath d='M8 10h16M8 16h11M8 22h14' stroke='white' stroke-width='3' stroke-linecap='round'/%3E%3C/svg%3E">
+  <link rel="stylesheet" href="style.css">
+  <script>
+    // Same saved theme and language as the UI, applied before first paint.
+    try {
+      var t = localStorage.getItem('logstream.theme');
+      if (t === 'light' || t === 'dark') document.documentElement.dataset.theme = t;
+      var l = localStorage.getItem('logstream.lang');
+      if (l) document.documentElement.lang = l;
+    } catch (e) {}
+  </script>
+</head>
+<body class="login-page">
+  <div class="login-tools">
+    <div class="seg" role="radiogroup" id="langSwitch">
+      <button type="button" role="radio" data-lang="fr">FR</button>
+      <button type="button" role="radio" data-lang="en">EN</button>
+    </div>
+    <button id="themeBtn" class="icon-btn" type="button" data-i18n-title="theme" data-i18n-aria="theme">
+      <svg class="sun" viewBox="0 0 24 24"><circle cx="12" cy="12" r="4"/><path d="M12 2v2M12 20v2M4.9 4.9l1.4 1.4M17.7 17.7l1.4 1.4M2 12h2M20 12h2M4.9 19.1l1.4-1.4M17.7 6.3l1.4-1.4"/></svg>
+      <svg class="moon" viewBox="0 0 24 24"><path d="M21 12.8A9 9 0 1 1 11.2 3a7 7 0 0 0 9.8 9.8z"/></svg>
+    </button>
+  </div>
+
+  <main class="login-card">
+    <img id="logo" class="login-logo" src="auth/logo" alt="" hidden>
+    <div class="brand">
+      <svg viewBox="0 0 32 32" aria-hidden="true"><rect width="32" height="32" rx="8"/><path d="M8 10h16M8 16h11M8 22h14"/></svg>
+      <span>LogStream</span>
+    </div>
+    <p class="muted" data-i18n="intro">Sign in to view the logs.</p>
+
+    <form method="post" action="auth/login">
+      <input type="hidden" name="r" id="ret">
+      <label for="user" data-i18n="user">User</label>
+      <input id="user" name="user" type="text" autocomplete="username" autocapitalize="none" spellcheck="false" required autofocus>
+      <label for="pass" data-i18n="pass">Password</label>
+      <input id="pass" name="pass" type="password" autocomplete="current-password" required>
+      <p id="err" class="login-err" role="alert" data-i18n="error" hidden>Wrong user or password.</p>
+      <button class="btn primary" type="submit" data-i18n="submit">Sign in</button>
+    </form>
+  </main>
+
+  <script>
+    (function () {
+      var I18N = {
+        en: { intro: 'Sign in to view the logs.', user: 'User', pass: 'Password', submit: 'Sign in',
+              error: 'Wrong user or password.', theme: 'Light / dark theme' },
+        fr: { intro: 'Connectez-vous pour consulter les logs.', user: 'Utilisateur', pass: 'Mot de passe',
+              submit: 'Se connecter', error: 'Utilisateur ou mot de passe incorrect.', theme: 'Thème clair / sombre' },
+      };
+      function get(k) { try { return localStorage.getItem('logstream.' + k); } catch (e) { return null; } }
+      function set(k, v) { try { localStorage.setItem('logstream.' + k, v); } catch (e) {} }
+      var lang = get('lang');
+      if (!I18N[lang]) lang = /^fr\b/i.test(navigator.language || '') ? 'fr' : 'en';
+
+      function applyLang() {
+        var d = I18N[lang];
+        document.documentElement.lang = lang;
+        document.querySelectorAll('[data-i18n]').forEach(function (el) { el.textContent = d[el.dataset.i18n]; });
+        document.querySelectorAll('[data-i18n-title]').forEach(function (el) { el.title = d[el.dataset.i18nTitle]; });
+        document.querySelectorAll('[data-i18n-aria]').forEach(function (el) { el.setAttribute('aria-label', d[el.dataset.i18nAria]); });
+        document.querySelectorAll('#langSwitch [data-lang]').forEach(function (b) { b.setAttribute('aria-checked', String(b.dataset.lang === lang)); });
+      }
+      document.querySelectorAll('#langSwitch [data-lang]').forEach(function (b) {
+        b.addEventListener('click', function () { lang = b.dataset.lang; set('lang', lang); applyLang(); });
+      });
+
+      document.getElementById('themeBtn').addEventListener('click', function () {
+        var root = document.documentElement;
+        var dark = root.dataset.theme ? root.dataset.theme === 'dark' : matchMedia('(prefers-color-scheme: dark)').matches;
+        root.dataset.theme = dark ? 'light' : 'dark';
+        set('theme', root.dataset.theme);
+      });
+
+      var q = new URLSearchParams(location.search);
+      document.getElementById('ret').value = q.get('r') || '/';
+      document.getElementById('err').hidden = q.get('e') !== '1';
+
+      // LOGIN_LOGO: shown only when the server has one.
+      var logo = document.getElementById('logo');
+      logo.addEventListener('load', function () { logo.hidden = false; });
+      if (logo.complete && logo.naturalWidth) logo.hidden = false;
+
+      applyLang();
+    })();
+  </script>
+</body>
+</html>

web/style.css

@@ -1,3 +1,13 @@
+/* Narrow monospace fonts served by LogStream itself, so they also work offline (licences in
+   web/fonts). Latin subset only; other scripts fall back to --mono. Inconsolata Condensed is
+   Inconsolata's variable font pinned at width 75 (0.4em per character, others 0.5em). */
+@font-face { font-family: "Iosevka"; src: url("fonts/iosevka-400.woff2") format("woff2"); font-weight: 400 500; font-display: swap; }
+@font-face { font-family: "Iosevka"; src: url("fonts/iosevka-700.woff2") format("woff2"); font-weight: 600 800; font-display: swap; }
+@font-face { font-family: "Ubuntu Mono"; src: url("fonts/ubuntu-mono-400.woff2") format("woff2"); font-weight: 400 500; font-display: swap; }
+@font-face { font-family: "Ubuntu Mono"; src: url("fonts/ubuntu-mono-700.woff2") format("woff2"); font-weight: 600 800; font-display: swap; }
+@font-face { font-family: "Inconsolata Condensed"; src: url("fonts/inconsolata-condensed-400.woff2") format("woff2"); font-weight: 400 500; font-display: swap; }
+@font-face { font-family: "Inconsolata Condensed"; src: url("fonts/inconsolata-condensed-700.woff2") format("woff2"); font-weight: 600 800; font-display: swap; }
+
 /* ---------- Theme: everything goes through these variables ---------- */
 :root {
   --bg: #f6f7f9;
@@ -12,6 +22,10 @@
   --hit: rgba(99, 102, 241, .18);
   --shadow: 0 10px 30px rgba(15, 23, 42, .12);
   --chip-l: 38%;            /* lightness of project-colored text (labels, Docker apps) */
+  --src-audit: #7c4dbd;     /* audit events (auditd) */
+  --src-ls: #1f9aa3;        /* LogStream internal logs */
+  --src-file: #5f8a1f;      /* log files (Settings > Sources > Log files) */
+  --src-tint: 6%;           /* background tint of their rows */
 
   --sev-crit: #be123c;
   --sev-err: #dc2626;
@@ -38,6 +52,8 @@
   --tag-warning-text: #111827;
 
   --log-size: 12.5px;   /* Settings > Interface > Font size */
+  --codes-w: 4.9rem;    /* tag codes column: room for 3 badges */
+  --code-bg: #858c97;   /* tag code badges */
   /* --log-font is set by Settings > Interface > Font (defaults to --mono) */
   --mono: ui-monospace, "SF Mono", "JetBrains Mono", "Cascadia Code", Menlo, Consolas, monospace;
   --sans: system-ui, -apple-system, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
@@ -58,6 +74,10 @@
   --hit: rgba(129, 140, 248, .28);
   --shadow: 0 10px 30px rgba(0, 0, 0, .5);
   --chip-l: 70%;
+  --src-audit: #a983e6;
+  --src-ls: #3cc3cc;
+  --src-file: #9ccc54;
+  --src-tint: 9%;
   --sev-crit: #fb7185;
   --sev-err: #f87171;
   --sev-warning: #fbbf24;
@@ -90,6 +110,11 @@
     --hit: rgba(129, 140, 248, .28);
     --shadow: 0 10px 30px rgba(0, 0, 0, .5);
   --chip-l: 70%;
+    --src-audit: #a983e6;
+    --src-ls: #3cc3cc;
+    --src-file: #9ccc54;
+  --src-file: #9ccc54;
+    --src-tint: 9%;
     --sev-crit: #fb7185;
     --sev-err: #f87171;
     --sev-warning: #fbbf24;
@@ -168,6 +193,16 @@ svg { fill: none; stroke: currentColor; stroke-width: 2; stroke-linecap: round;
 .chip.on { color: var(--accent); border-color: var(--accent); background: var(--accent-soft); }
 
 .actions { display: flex; align-items: center; gap: 6px; margin-left: auto; }
+/* Logged-in user, with how they logged in (oidc / local) */
+.whoami { display: flex; align-items: center; gap: 6px; min-width: 0; margin-left: 6px; font-size: 13px; color: var(--muted); }
+.whoami .who-name { font-weight: 600; color: var(--text); max-width: 18ch; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
+.whoami .who-badge {
+  flex: none; border: 1px solid var(--border); border-radius: 6px;
+  padding: 1px 6px; font-size: 10.5px; font-weight: 650; letter-spacing: .02em;
+}
+.whoami .who-badge.oidc { color: var(--accent); border-color: var(--accent); background: var(--accent-soft); }
+.whoami .who-badge:empty { display: none; }
+.whoami.none .who-name { font-weight: 400; color: var(--muted); }
 .icon-btn {
   width: 36px; height: 36px; display: grid; place-items: center;
   border: 1px solid transparent; background: transparent; border-radius: 9px; color: var(--muted);
@@ -228,6 +263,19 @@ body.busy .progress::after {
   font-size: 13px; max-width: 220px;
 }
 .filters select.set { border-color: var(--accent); background-color: var(--accent-soft); }
+/* Display mode: Stream or Time range, then the start → end fields of the range */
+.view-mode { padding: 2px; gap: 2px; border-radius: 9px; }
+.view-mode button { display: inline-flex; align-items: center; gap: 6px; height: 26px; padding: 0 10px; }
+.view-mode svg { width: 15px; height: 15px; }
+.period { display: inline-flex; align-items: center; gap: 6px; }
+.period .btn.tool { width: 32px; padding: 0; justify-content: center; }
+.period input {
+  height: 32px; padding: 0 8px;
+  border: 1px solid var(--accent); border-radius: 8px; background: var(--accent-soft);
+  font-size: 13px; font-variant-numeric: tabular-nums; color-scheme: light dark;
+}
+.period input:focus { outline: 0; box-shadow: 0 0 0 3px var(--accent-soft); }
+.period input.bad { border-color: var(--sev-err); background: color-mix(in srgb, var(--sev-err) 10%, transparent); }
 .spacer { flex: 1; }
 #count { font-size: 12.5px; font-variant-numeric: tabular-nums; }
 
@@ -296,13 +344,13 @@ body.busy .progress::after {
 .list { margin: 6px 20px 0; background: var(--panel); border: 1px solid var(--border); border-radius: var(--radius); overflow: hidden; }
 .list:empty { display: none; }
 
-/* Columns: received, message time, severity, host, app, message. Widths come from
+/* Columns: received, message time, severity, host, app, tag codes, message. Widths come from
    --col-* (set on #table when a column has been resized, see app.js), shared by
    the header and every row through subgrid so that the columns line up. */
 .table {
   display: grid;
   grid-template-columns: var(--col-rcv, max-content) var(--col-mt, max-content) var(--col-sev, 4.6rem)
-    var(--col-host, minmax(5rem, 9rem)) var(--col-app, minmax(4rem, 8rem)) minmax(0, 1fr);
+    var(--col-host, minmax(5rem, 9rem)) var(--col-app, minmax(4rem, 8rem)) var(--codes-w) minmax(0, 1fr);
   column-gap: 12px;
   margin: 6px 20px 0; background: var(--panel); border: 1px solid var(--border); border-radius: var(--radius);
   overflow: clip; /* rounded corners without breaking the sticky header (hidden would) */
@@ -331,7 +379,7 @@ body.col-resizing, body.col-resizing * { cursor: col-resize !important; user-sel
 
 .row {
   display: grid;
-  grid-template-columns: max-content max-content 4.6rem minmax(5rem, 9rem) minmax(4rem, 8rem) 1fr;
+  grid-template-columns: max-content max-content 4.6rem minmax(5rem, 9rem) minmax(4rem, 8rem) var(--codes-w) 1fr;
   gap: 0 12px; align-items: baseline;
   padding: 5px 14px 5px 11px;
   border-left: 3px solid transparent;
@@ -341,6 +389,10 @@ body.col-resizing, body.col-resizing * { cursor: col-resize !important; user-sel
   cursor: pointer;
 }
 .row:last-child { border-bottom: 0; }
+/* Settings > Interface > Density: compact fits about 50% more lines on screen */
+:root[data-density="compact"] .row { padding-top: 1px; padding-bottom: 1px; line-height: 1.25; }
+:root[data-density="compact"] .row .sev { padding-top: 0; padding-bottom: 0; line-height: 1.3; }
+:root[data-density="compact"] .row .codes b { padding-top: 0; padding-bottom: 0; line-height: 1.2; }
 .row:hover { background: var(--row-hover); }
 .row:focus-visible { outline: 2px solid var(--accent); outline-offset: -2px; }
 .row.new { animation: flash 1.2s ease-out; }
@@ -372,6 +424,37 @@ body.col-resizing, body.col-resizing * { cursor: col-resize !important; user-sel
 .row.sev-crit, .row.sev-err, .row.sev-warning { border-left-color: var(--sev); }
 .row.sev-crit, .row.sev-err { background: color-mix(in srgb, var(--sev) 7%, transparent); }
 .row .host[data-act], .row .app[data-act] { cursor: pointer; }
+/* Audit events and LogStream internal logs: colored edge (over the severity
+   one), light tint, badge before the app name. */
+.row.src-audit { --src: var(--src-audit); }
+.row.src-logstream { --src: var(--src-ls); }
+.row.src-file { --src: var(--src-file); }
+.row.src-audit, .row.src-logstream, .row.src-file { border-left-color: var(--src); background: color-mix(in srgb, var(--src) var(--src-tint), transparent); }
+.row.src-audit:hover, .row.src-logstream:hover, .row.src-file:hover { background: color-mix(in srgb, var(--src) 13%, var(--row-hover)); }
+.src-badge {
+  display: inline-block; margin-right: 5px; padding: 0 4px; border-radius: 4px; vertical-align: 1px;
+  font: 700 9.5px/1.45 var(--sans); letter-spacing: .04em;
+  color: var(--src); background: color-mix(in srgb, var(--src) 16%, transparent);
+  box-shadow: inset 0 0 0 1px color-mix(in srgb, var(--src) 40%, transparent);
+}
+.row .raw {
+  margin-top: 2px; font-size: .85em; line-height: 1.35; color: var(--muted);
+  white-space: pre-wrap; word-break: break-all;
+}
+:root[data-density="compact"] .row:not(.open) .raw { display: none; }
+.src-help { margin-top: 10px !important; }
+.self-level { width: auto; justify-self: start; }
+/* Codes of the tags found in the message: grey badges, fixed size */
+.row .codes { white-space: nowrap; overflow: hidden; }
+.row .codes > span { display: inline-flex; gap: 3px; vertical-align: 1px; }
+.row .codes b, .tag-code {
+  min-width: 2.2ch; padding: 1px 4px; border-radius: 5px; text-align: center;
+  /* same font as the severity badges (the log font) */
+  font-family: var(--log-font, var(--mono)); font-size: 10.5px; font-weight: 700; letter-spacing: .03em;
+  line-height: 1.35; font-variant-numeric: tabular-nums;
+  background: var(--code-bg); color: #0b0f17;
+}
+.row .codes b.more { background: none; box-shadow: inset 0 0 0 1px var(--code-bg); color: var(--muted); }
 .row .host[data-act]:hover, .row .app[data-act]:hover { color: var(--accent); text-decoration: underline; text-underline-offset: 2px; }
 
 mark.tag {
@@ -406,6 +489,22 @@ mark.hit { background: var(--hit); color: inherit; border-radius: 3px; padding:
   font-family: var(--mono); font-size: 12.5px; white-space: pre-wrap;
 }
 
+.auth-warn {
+  display: flex; align-items: center; gap: 10px;
+  margin: 6px 20px 0; padding: 6px 8px 6px 14px;
+  border: 1px solid color-mix(in srgb, var(--sev-warning) 45%, transparent);
+  background: color-mix(in srgb, var(--sev-warning) 12%, transparent);
+  color: var(--text); border-radius: var(--radius); font-size: 12.5px;
+}
+.auth-warn[hidden] { display: none; }
+.auth-warn span { flex: 1; }
+.auth-warn .icon-btn { width: 26px; height: 26px; flex: none; }
+.ro-note {
+  margin: 0 0 12px; padding: 8px 12px; border-radius: var(--radius); font-size: 12.5px;
+  background: color-mix(in srgb, var(--accent) 10%, transparent); color: var(--text);
+}
+.read-only .set-panel .set-section[inert] { opacity: .55; }
+
 .pill {
   position: fixed; top: 70px; left: 50%; transform: translateX(-50%); z-index: 30;
   border: 0; border-radius: 999px; padding: 7px 16px;
@@ -425,6 +524,7 @@ mark.hit { background: var(--hit); color: inherit; border-radius: 3px; padding:
 .conn.ok::before { color: var(--sev-info); }
 .conn.ko::before { color: var(--sev-err); }
 #stats .bad { color: var(--sev-err); }
+#stats .warn { color: var(--sev-warning); }
 /* "Back to top", at the right end of the status bar: never over a log row */
 .to-top { width: 24px; height: 24px; margin: -5px -6px -5px auto; flex: none; } /* no taller bar */
 .to-top svg { width: 16px; height: 16px; }
@@ -438,12 +538,12 @@ mark.hit { background: var(--hit); color: inherit; border-radius: 3px; padding:
 
 /* ---------- Settings dialog ---------- */
 dialog.settings {
-  width: min(900px, calc(100vw - 32px));
+  width: min(1280px, calc(100vw - 32px));
-  height: min(660px, calc(100vh - 64px)); max-height: none;
+  height: calc(100vh - 48px); max-height: 960px;
 }
 dialog.settings .dlg { display: flex; flex-direction: column; height: 100%; padding: 0; }
 dialog.settings .dlg > header { padding: 14px 16px 14px 22px; border-bottom: 1px solid var(--border); }
-.set-body { flex: 1; min-height: 0; display: grid; grid-template-columns: 200px minmax(0, 1fr); }
+.set-body { flex: 1; min-height: 0; display: grid; grid-template-columns: minmax(176px, max-content) minmax(0, 1fr); }
 .set-nav {
   display: flex; flex-direction: column; gap: 2px;
   padding: 12px 10px; border-right: 1px solid var(--border);
@@ -458,8 +558,19 @@ dialog.settings .dlg > header { padding: 14px 16px 14px 22px; border-bottom: 1px
 .set-nav button:hover { background: var(--panel-2); color: var(--text); }
 .set-nav button[aria-selected="true"] { background: var(--accent-soft); color: var(--accent); }
 .set-nav svg { width: 18px; height: 18px; flex: none; }
-.set-panels { overflow-y: auto; padding: 2px 24px 22px; }
+.set-panels { overflow-y: auto; padding: 16px 20px 20px; }
-.set-panel > .set-section:first-child { margin-top: 16px; }
+/* Sections are cards laid out in columns when there is room; .span ones take the full width. */
+.set-panel { display: grid; grid-template-columns: repeat(auto-fit, minmax(380px, 1fr)); gap: 14px; align-items: start; grid-auto-flow: row dense; }
+.set-panel[hidden] { display: none; }
+.set-panel.wide { grid-template-columns: minmax(0, 1fr); }
+.set-panel > .set-section, .set-panel > .set-section + .set-section {
+  margin: 0; padding: 14px 16px; min-width: 0;
+  border: 1px solid var(--border); border-radius: 12px;
+  background: color-mix(in srgb, var(--panel-2) 35%, var(--panel));
+}
+.set-panel > .set-section.span { grid-column: 1 / -1; }
+.set-panel > .ro-note { grid-column: 1 / -1; margin: 0; }
+.sec-head { display: flex; align-items: baseline; justify-content: space-between; gap: 12px; }
 .hint { margin-top: 8px !important; }
 .lbl { font-size: 13px; font-weight: 550; }
 .set-panel .preview-list { margin: 14px 0 10px; pointer-events: none; }
@@ -475,6 +586,7 @@ dialog.settings .dlg > header { padding: 14px 16px 14px 22px; border-bottom: 1px
 .preview-list .row .host { grid-area: host; }
 .preview-list .row .app { grid-area: app; display: block; }
 .preview-list .row .msg { grid-area: msg; }
+.preview-list .row .codes { display: none; }
 
 dialog {
   width: min(720px, calc(100vw - 32px)); max-height: calc(100vh - 64px);
@@ -503,6 +615,17 @@ select.field:focus { outline: 0; border-color: var(--accent); box-shadow: 0 0 0
   background: var(--panel-2); font-family: var(--mono); font-size: 12.5px;
 }
 .set-section.danger h3 { color: var(--sev-err); }
+.db-head { display: flex; align-items: baseline; justify-content: space-between; gap: 10px; margin: 4px 0 6px; }
+.db-stats {
+  display: grid; grid-template-columns: max-content minmax(0, 1fr); gap: 5px 14px;
+  margin: 0 0 14px; padding: 10px 12px; border-radius: 8px; background: var(--panel-2); font-size: 12.5px;
+}
+.db-stats dt { color: var(--muted); }
+.db-stats dd { margin: 0; min-width: 0; overflow-wrap: anywhere; }
+.db-stats dd b { font-weight: 600; font-variant-numeric: tabular-nums; }
+.db-msg { grid-column: 1 / -1; color: var(--muted); }
+.db-msg.bad { color: var(--sev-err); }
+.link-btn:disabled { opacity: .5; cursor: default; text-decoration: none; }
 .purge-row { display: flex; flex-wrap: wrap; align-items: center; gap: 10px; margin-top: 10px; }
 .btn.danger { color: var(--sev-err); border-color: color-mix(in srgb, var(--sev-err) 45%, var(--border)); }
 .btn.danger:hover:not(:disabled) { background: var(--sev-err); border-color: var(--sev-err); color: #fff; }
@@ -577,7 +700,19 @@ input.switch:checked::after { transform: translateX(14px); }
 input.switch:disabled { cursor: not-allowed; }
 input.switch:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
 
-.set-section footer { display: flex; flex-wrap: wrap; gap: 8px; justify-content: space-between; margin-top: 14px; }
+/* Settings > Pattern: toolbar kept in view while the list scrolls, dense rows. */
+.tags-section { padding-top: 12px !important; }
+.tag-toolbar {
+  position: sticky; top: -16px; z-index: 2;
+  display: flex; flex-wrap: wrap; align-items: center; gap: 8px;
+  margin: 10px -16px 0; padding: 8px 16px;
+  background: color-mix(in srgb, var(--panel-2) 35%, var(--panel));
+  border-bottom: 1px solid var(--border);
+}
+.tag-toolbar .btn { height: 30px; }
+.tag-toolbar select.field { width: auto; height: 30px; }
+.tag-toolbar .dk-filter { flex: 1 1 220px; height: 30px; }
+.tag-toolbar #resetTags { margin-left: auto; }
 
 .seg { display: inline-flex; padding: 3px; gap: 3px; background: var(--panel-2); border: 1px solid var(--border); border-radius: 10px; }
 .seg button {
@@ -587,58 +722,100 @@ input.switch:focus-visible { outline: 2px solid var(--accent); outline-offset: 2
 .seg button:hover { color: var(--text); }
 .seg button[aria-checked="true"] { background: var(--panel); color: var(--accent); box-shadow: 0 1px 3px rgba(0, 0, 0, .12); }
 
-.tag-list { display: flex; flex-direction: column; gap: 8px; margin-top: 12px; }
+/* One wide column: long regular expressions stay readable. */
+.tag-list { display: flex; flex-direction: column; gap: 3px; margin-top: 10px; }
 .tag-row {
-  display: grid; align-items: center; gap: 8px 10px;
+  display: grid; align-items: center; gap: 4px 6px;
-  grid-template-columns: 38px minmax(8rem, 1fr) 7.5rem auto 36px;
+  grid-template-columns: 2.4em 30px minmax(7rem, 1fr) 10rem auto 26px;
-  padding: 8px 10px; border: 1px solid var(--border); border-radius: 10px;
+  padding: 3px 4px 3px 6px; border: 1px solid var(--border); border-radius: 8px;
-  background: var(--panel-2);
+  background: var(--panel);
 }
+.tag-row[hidden] { display: none; }
 .tag-row.off { opacity: .55; }
+.tag-row .tag-code { font-size: 11px; padding: 2px 4px; cursor: default; text-align: center; }
 .tag-row input[type="color"] {
-  width: 38px; height: 32px; padding: 0; border: 1px solid var(--border); border-radius: 8px;
+  width: 30px; height: 26px; padding: 0; border: 1px solid var(--border); border-radius: 6px;
   background: none; cursor: pointer;
 }
-.tag-row input[type="color"]::-webkit-color-swatch-wrapper { padding: 3px; }
+.tag-row input[type="color"]::-webkit-color-swatch-wrapper { padding: 2px; }
 .tag-row input[type="color"]::-webkit-color-swatch { border: 0; border-radius: 5px; }
 .tag-row input[type="text"] {
-  height: 32px; padding: 0 10px; min-width: 0;
+  height: 26px; padding: 0 8px; min-width: 0;
-  border: 1px solid var(--border); border-radius: 8px; background: var(--panel);
+  border: 1px solid var(--border); border-radius: 6px; background: var(--panel);
-  font-family: var(--mono); font-size: 13px; outline: 0;
+  font-family: var(--mono); font-size: 12.5px; outline: 0;
 }
 .tag-row input[type="text"]:focus { border-color: var(--accent); box-shadow: 0 0 0 3px var(--accent-soft); }
-.tag-row .preview { font-family: var(--mono); font-size: 12.5px; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; text-align: center; }
+.tag-row .preview { font-family: var(--mono); font-size: 12px; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; text-align: center; }
-.tag-row .opts { display: flex; gap: 4px; }
+.tag-row .opts { display: flex; gap: 3px; }
 .opt {
-  display: inline-flex; align-items: center; height: 28px; padding: 0 8px;
+  display: inline-flex; align-items: center; height: 24px; padding: 0 6px;
-  border: 1px solid var(--border); border-radius: 7px; background: var(--panel);
+  border: 1px solid var(--border); border-radius: 6px; background: var(--panel);
-  font-size: 11.5px; font-weight: 600; color: var(--muted); cursor: pointer; user-select: none;
+  font-size: 11px; font-weight: 600; color: var(--muted); cursor: pointer; user-select: none;
 }
+.tag-row .icon-btn { width: 26px; height: 26px; border-radius: 6px; }
+.tag-row .icon-btn svg { width: 15px; height: 15px; }
 .opt input { display: none; }
 .opt:has(input:checked) { color: var(--accent); border-color: var(--accent); background: var(--accent-soft); }
+/* Log files (Settings > Sources) */
+.file-list { display: flex; flex-direction: column; gap: 4px; }
+.file-row {
+  display: grid; align-items: center; gap: 6px;
+  grid-template-columns: auto minmax(7rem, 1fr) minmax(10rem, 2fr) minmax(9rem, 1.4fr) minmax(6rem, .8fr) 26px;
+  padding: 6px 6px 6px 10px; border: 1px solid var(--border); border-radius: 8px; background: var(--panel);
+}
+.file-row.off > input[type="text"], .file-row.off > select { opacity: .6; }
+.file-row input[type="text"] {
+  height: 28px; padding: 0 8px; min-width: 0;
+  border: 1px solid var(--border); border-radius: 6px; background: var(--panel); font-size: 12.5px; outline: 0;
+}
+.file-row input.mono { font-family: var(--mono); }
+.file-row input[type="text"]:focus { border-color: var(--accent); box-shadow: 0 0 0 3px var(--accent-soft); }
+.file-row select.field { height: 28px; }
+.file-row .icon-btn { width: 26px; height: 26px; border-radius: 6px; }
+.file-row .icon-btn svg { width: 15px; height: 15px; }
+.file-row .fr-regex, .file-row .fr-status, .file-row .fr-sample { grid-column: 2 / -1; }
+.file-row .fr-sample {
+  margin: 0; font: 11.5px var(--mono); color: var(--muted);
+  white-space: nowrap; overflow: hidden; text-overflow: ellipsis;
+}
+.file-row .fr-sample[hidden] { display: none; }
+.file-row .fr-regex[hidden] { display: none; }
+.file-row .fr-status { margin: 0 !important; font-size: 12px; }
+.file-toolbar { margin-top: 10px; }
 .tag-row .err { grid-column: 1 / -1; color: var(--sev-err); font-size: 12px; }
 .tag-row .err:empty { display: none; }
 
 /* ---------- Mobile ---------- */
 @media (max-width: 760px) {
+  .file-row { grid-template-columns: auto minmax(0, 1fr) 26px; }
+  .file-row [data-del] { grid-column: 3; grid-row: 1; }
+  .file-row > [data-f="path"], .file-row > [data-f="format"], .file-row > [data-f="host"] { grid-column: 2 / -1; }
   body { font-size: 13.5px; }
   .topbar { flex-wrap: wrap; padding: 8px 16px; gap: 8px 10px; }
   .search { order: 3; flex-basis: 100%; max-width: none; }
   .live .lbl { display: none; }
+  .whoami .who-name { max-width: 12ch; }
   .filters { padding: 10px 16px 6px; }
   .filters select { flex: 1 1 calc(50% - 8px); max-width: none; min-width: 0; }
+  .view-mode { flex: 1 1 100%; }
+  .view-mode button { flex: 1; justify-content: center; }
+  /* Phones: the two dates on one line, the ◀ ▶ − buttons below */
+  .period { flex: 1 1 100%; flex-wrap: wrap; }
+  .period > span { display: none; }
+  .period input { flex: 1 1 calc(50% - 3px); min-width: 0; }
+  .period .btn.tool { order: 1; flex: 1; }
   .spacer { display: none; }
   #count { flex: 1 1 50%; min-width: 0; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
   .histo { padding: 0 16px 6px; }
   .h-leg { display: none; }
-  .list, .error-banner, .table { margin-left: 16px; margin-right: 16px; }
+  .list, .error-banner, .auth-warn, .table { margin-left: 16px; margin-right: 16px; }
   /* Phones: no columns, two lines per log (layout below); the widths do not apply */
   .table { display: block; }
   .table .list { display: block; margin: 0; }
   .list-head { display: none; }
   .row, .table .row {
-    grid-template-columns: auto auto 1fr;
+    grid-template-columns: auto auto 1fr auto;
-    grid-template-areas: "rcv sev host" "msg msg msg";
+    grid-template-areas: "rcv sev host codes" "msg msg msg msg";
     gap: 3px 8px; padding: 8px 12px 8px 10px;
   }
   .row time.rcv { grid-area: rcv; }
@@ -646,6 +823,7 @@ input.switch:focus-visible { outline: 2px solid var(--accent); outline-offset: 2
   .row .sev { grid-area: sev; }
   .row .host { grid-area: host; justify-self: end; max-width: 100%; }
   .row .app { display: none; }
+  .row .codes { grid-area: codes; }
   .row .msg { grid-area: msg; }
   .details { grid-column: 1 / -1; }
   .details dl { grid-template-columns: 1fr; }
@@ -661,14 +839,45 @@ input.switch:focus-visible { outline: 2px solid var(--accent); outline-offset: 2
     flex-direction: column; justify-content: center; height: auto; padding: 6px 2px; gap: 3px;
     font-size: 11.5px; text-align: center;
   }
-  .set-nav button span { max-width: 100%; overflow: hidden; text-overflow: ellipsis; }
+  .set-nav button span { max-width: 100%; white-space: normal; line-height: 1.15; overflow: hidden; text-overflow: ellipsis; }
-  .set-panels { padding: 0 16px 18px; }
+  .set-panels { padding: 12px 12px 18px; }
-  #sizeSwitch { display: flex; }
+  .set-panel { grid-template-columns: minmax(0, 1fr); gap: 10px; }
-  #sizeSwitch button { flex: 1; padding: 0 4px; white-space: nowrap; font-size: 12px; }
+  .set-panel > .set-section, .set-panel > .set-section + .set-section { padding: 12px; }
+  .tag-toolbar { position: static; margin: 10px -12px 0; padding: 8px 12px; }
+  .tag-toolbar #resetTags { margin-left: 0; }
+  #sizeSwitch, #densitySwitch { display: flex; }
+  #sizeSwitch button, #densitySwitch button { flex: 1; padding: 0 4px; white-space: nowrap; font-size: 12px; }
+  :root[data-density="compact"] .row { padding-top: 3px; padding-bottom: 3px; }
   .field-grid select { margin-bottom: 6px; }
   .status { padding: 6px 16px; }
-  .tag-row { grid-template-columns: 38px 1fr 36px; }
+  .tag-row { grid-template-columns: auto 30px 1fr 26px; }
   .tag-row .preview { display: none; }
   .tag-row .opts { grid-column: 1 / -1; grid-row: 2; flex-wrap: wrap; }
-  .tag-row [data-del] { grid-column: 3; grid-row: 1; }
+  .tag-row [data-del] { grid-column: 4; grid-row: 1; }
 }
+
+/* ---------- Login page (AUTH_MODE=local) ---------- */
+body.login-page {
+  min-height: 100vh; padding: 16px;
+  display: grid; place-items: center;
+}
+.login-tools { position: fixed; top: 12px; right: 16px; display: flex; align-items: center; gap: 8px; }
+.login-card {
+  width: 100%; max-width: 360px;
+  display: flex; flex-direction: column; align-items: center; gap: 10px;
+  padding: 32px 28px 28px;
+  background: var(--panel); border: 1px solid var(--border); border-radius: 14px; box-shadow: var(--shadow);
+}
+.login-logo { max-width: 200px; max-height: 96px; object-fit: contain; margin-bottom: 6px; }
+.login-card .brand { font-size: 20px; }
+.login-card .brand svg { width: 32px; height: 32px; }
+.login-card > p { margin: 0 0 8px; text-align: center; }
+.login-card form { width: 100%; display: flex; flex-direction: column; gap: 6px; }
+.login-card label { font-size: 13px; font-weight: 550; }
+.login-card input {
+  height: 38px; padding: 0 11px; margin-bottom: 6px;
+  border: 1px solid var(--border); border-radius: 9px; background: var(--bg);
+}
+.login-card input:focus { outline: 0; border-color: var(--accent); box-shadow: 0 0 0 3px var(--accent-soft); }
+.login-card .btn.primary { height: 38px; justify-content: center; margin-top: 6px; font-size: 14px; }
+.login-err { margin: 0; color: var(--sev-err); font-size: 13px; }