UI files: ETag and revalidation on each load

The embedded UI was served without any cache header, so a browser or a
proxy could keep an old app.js after an update (seen with the API token
display). Each file now has an ETag (hash of its content, computed at
startup); the page, app.js and style.css are sent with "no-cache", so
each load costs a 304 without body until the next version. Fonts are
cached for 30 days.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

README.fr.md

@@ -92,6 +92,10 @@ une base propre.
 Les badges de sévérité `err`/`crit` et `warning` reprennent les couleurs des tags `error` et
 `warning`.
 
+Chaque ligne commence par un badge de largeur fixe qui indique sa source, avec une couleur par
+source : **SYSLOG**, **DOCKER**, **KERNEL** (dmesg), **HOST** (fichiers de `/var/log`), **AUDIT**,
+**FILE** et **LS** (logs de LogStream lui-même).
+
 Raccourcis : `/` place le curseur dans la recherche, `Esc` la vide. Un clic sur une ligne
 affiche tous ses champs.
 
@@ -278,7 +282,7 @@ source est **désactivée par défaut** : activez-la dans **Réglages > Sources
   ne peut pas être résolu), `exe`, `res` (`success` / `failed`, d'après `res=` ou `success=`) et
   `key` (clé de la règle). La sévérité est `warning` en cas d'échec, `info` sinon ; facility
   `authpriv` ; source `audit`. Dans la liste, les lignes d'audit ont un liseré gauche violet,
-  un fond légèrement teinté et un badge **AUDIT** devant l'app ; le filtre **Source** permet de
+  un fond légèrement teinté et le badge de source **AUDIT** ; le filtre **Source** permet de
   les afficher seules.
 - **Noms** : `user` vient du format enrichi (`log_format = ENRICHED`, le défaut des versions
   récentes d'auditd), sinon du `/etc/passwd` de l'hôte s'il est monté en `/host/etc/passwd`.
@@ -338,11 +342,13 @@ conteneur, un **format** (*Automatique* par défaut), un **hôte** facultatif (p
   qui portent le nom d'un champ de LogStream (`host`, `app`…) sont renommés `field_host`,
   `field_app`…
 - **Affichage** : source `file`, le chemin dans `log_file`. Dans la liste, ces lignes ont un
-  liseré gauche vert, une légère teinte et un badge **FILE** avant l'app ; le filtre **Source**
+  liseré gauche vert, une légère teinte et le badge de source **FILE** ; le filtre **Source**
   les affiche seules.
 - À l'activation d'un fichier, les lignes de la dernière heure déjà présentes sont lues d'abord
   (`FILES_BACKFILL`, `0` = seulement les nouvelles lignes ; les lignes sans horodatage sont alors
-  ignorées) ; un fichier créé ensuite est lu depuis son début. L'inode et une
+  ignorées). Le début de cette heure est trouvé par dichotomie sur les dates des lignes : un gros
+  fichier qui contient des mois de logs n'est pas relu depuis son début ; un fichier créé ensuite
+  est lu depuis son début, jusqu'à 64 Mo par seconde, l'état indiquant ce qu'il reste à lire. L'inode et une
   position sont enregistrés par fichier dans `/data/files-state.json` : après un redémarrage la
   lecture reprend là ; après une rotation (`access.log` renommé `access.log.1`), la fin de
   l'ancien fichier est lue avant le nouveau ; un fichier vidé sur place (`copytruncate`) est
@@ -384,6 +390,79 @@ charge (pas de `| pipes`).
 
 En ligne de commande : `curl -o logs.csv 'localhost:8080/api/export.csv?q=error&range=24h&tz=Europe/Paris'`.
 
+## API de lecture (n8n, scripts)
+
+`GET /api/v1/logs` renvoie les logs stockés qui suivent un curseur, du plus ancien au plus
+récent : un outil appelé à intervalle régulier (n8n, un script, un connecteur SIEM) reçoit ainsi
+chaque nouveau log une fois, sans trou ni doublon, même quand beaucoup de logs ont le même
+horodatage.
+
+L'accès passe par un **jeton d'API**, créé dans *Paramètres > API* par un administrateur et
+envoyé dans `Authorization: Bearer lst_…`. Un jeton est en lecture seule et n'ouvre que
+`/api/v1/` ; il fonctionne quel que soit le mode d'[authentification](#authentification)
+(`local`, `oidc` ou aucune). Il n'est affiché qu'une fois, à sa création : LogStream n'en garde
+que l'empreinte SHA-256, dans `/data/api_tokens.json`. Le révoquer (puis Valider) coupe l'accès immédiatement ;
+la liste indique quand et d'où chaque jeton a servi pour la dernière fois. Une session ouverte
+dans le navigateur peut aussi appeler la route.
+
+| Paramètre | Rôle |
+|---|---|
+| `query` | filtre LogsQL (par défaut `*`), par exemple `severity:in(err,crit) app:sshd`. Les pipes (`\| ...`) sont refusés. |
+| `since` | point de départ : le `next_cursor` de la réponse précédente, une date RFC 3339 (`2026-11-01T08:00:00Z`) ou une durée avant maintenant (`15m`, `6h`, `7d`). Par défaut : `1h`. |
+| `limit` | lignes par réponse, de 1 à 10 000 (1 000 par défaut). |
+| `lag` | seuls les logs reçus depuis au moins cette durée sont renvoyés (`30s` par défaut, `1h` au plus), pour ne pas sauter les logs encore dans la file ou en attente de leur nom DNS. |
+
+```json
+{
+  "rows": [{"_time": "2026-11-01T14:02:11.52Z", "_msg": "…", "host": "fw01", "app": "sshd", "severity": "err", …}],
+  "next_cursor": "eyJ0IjoiMjAyNi0xMS0wMVQxNDowMjoxMS41MjAwMDAwMDFaIiwicSI6IjJlYjE4YzNhIn0",
+  "has_more": false,
+  "until": "2026-11-01T14:05:30.1Z"
+}
+```
+
+Les lignes contiennent les champs stockés (comme la vue JSON de l'interface), triées par
+`_time`, l'heure de réception. Renvoyez toujours `next_cursor`, même quand `rows` est vide.
+`has_more: true` signifie que d'autres logs sont prêts : rappelez tout de suite.
+`waiting: true` signifie que des logs du [tampon disque](#configuration) sont encore en cours
+d'envoi à VictoriaLogs : rien n'est renvoyé tant qu'ils ne sont pas stockés, pour n'en manquer
+aucun. Le curseur est lié à sa `query` : changer la requête garde la position dans le temps. Les
+erreurs arrivent sous la forme `{"error", "code"}` avec le statut 400 (paramètre ou LogsQL
+invalide), 401 (jeton invalide ou révoqué) ou 502 (VictoriaLogs injoignable).
+
+```bash
+curl -H "Authorization: Bearer lst_…" \
+  "https://logs.example.org/api/v1/logs?query=severity:err&since=15m&limit=1000"
+```
+
+### Exemple : workflow n8n
+
+Quatre nœuds, le curseur gardé dans les données statiques du workflow d'une exécution à l'autre :
+
+1. **Schedule Trigger** : toutes les minutes.
+2. **Code** (*Run Once for All Items*), lit le curseur :
+   ```js
+   const data = $getWorkflowStaticData('global');
+   return [{ json: { since: data.cursor || '15m' } }];
+   ```
+3. **HTTP Request** : méthode `GET`, URL `https://logs.example.org/api/v1/logs`.
+   *Authentication* : Generic Credential Type > Header Auth, avec le nom `Authorization` et la
+   valeur `Bearer lst_…`. *Send Query Parameters* : `query` = `severity:in(emerg,alert,crit,err)`,
+   `since` = `{{ $json.since }}`, `limit` = `1000`.
+4. **Code** (*Run Once for All Items*), enregistre le curseur et fait de chaque log un item :
+   ```js
+   const data = $getWorkflowStaticData('global');
+   const page = $input.first().json;
+   data.cursor = page.next_cursor;
+   return page.rows.map((row) => ({ json: row }));
+   ```
+
+Branchez ensuite ce que les logs doivent alimenter (Slack, mail, ticket, base de données…). n8n ne
+garde les données statiques que pour les exécutions du workflow actif, pas pour les tests
+manuels : pendant les essais, chaque exécution repart de `15m`. Au-delà de 1 000 logs par minute,
+`has_more` vaut true et les exécutions suivantes rattrapent le retard ; augmentez `limit` ou la
+fréquence du déclencheur si besoin.
+
 ## Paramètres
 
 L'icône en forme d'engrenage ouvre les paramètres, organisés en onglets. Les modifications
@@ -446,6 +525,11 @@ et la réinitialisation des colonnes agissent immédiatement.
   désactivée par défaut : mettez `ALLOW_PURGE=true` pour l'autoriser. Tout administrateur peut
   alors purger : activez l'[authentification](#authentification) si l'interface est accessible
   à d'autres.
+- **API** : jetons de l'[API de lecture](#api-de-lecture-n8n-scripts) : en ajouter un avec un
+  nom, en révoquer un, voir quand et d'où chacun a servi pour la dernière fois. Ajouts et
+  révocations attendent **Valider** comme les autres réglages (**Annuler** les abandonne). Après
+  Valider, chaque nouveau jeton est affiché une seule fois avec un bouton Copier, et l'exemple
+  curl en dessous l'utilise, surligné, avec son propre bouton Copier. Réservé aux administrateurs.
 
 ## Authentification
 
@@ -625,6 +709,8 @@ Pour mettre à jour l'une d'elles :
 | `hub.go` | envoie les nouveaux messages aux navigateurs (SSE) |
 | `rdns.go` | résolutions DNS inverses avec cache |
 | `export.go` | export CSV en flux |
+| `logsapi.go` | API de lecture `/api/v1/logs` : curseur, groupes de logs de même heure |
+| `apitokens.go` | jetons d'API : stockage (haché), contrôle Bearer, `/api/tokens` |
 | `docker.go` | logs des conteneurs Docker (API, lecteurs, positions, détection du niveau) |
 | `syslogserver.go` | écoutes syslog ouvertes et fermées depuis Paramètres > Sources de logs |
 | `hostlogs.go`, `journal.go` | logs système de l'hôte : lecteur du journal systemd (sans `journalctl`) et suivi de `/var/log` |
@@ -635,6 +721,7 @@ Pour mettre à jour l'une d'elles :
 | `tags.go` | stockage des tags de couleur |
 | `presets.go`, `presets.json` | préréglages de tags (`/api/presets`), liste intégrée |
 | `api.go` | routes HTTP `/api/*` |
+| `static.go` | fichiers de l'interface embarqués : ETag par fichier, revérifiés à chaque chargement (`no-cache`, 304 s'ils n'ont pas changé) pour toujours prendre une mise à jour ; polices en cache 30 jours |
 | `web/` | interface (HTML, CSS, JavaScript simple, sans étape de build), embarquée dans le binaire ; les traductions sont dans `web/app.js` (`I18N`), et dans `web/login.html` pour la page de connexion |
 
 ## Remarque

README.md

@@ -83,6 +83,9 @@ versions before this change are indexed by their message timestamp; purge them f
 Settings to start clean.
 
 Severity badges `err`/`crit` and `warning` use the colors of the `error` and `warning` tags.
+Each row starts with a fixed-width badge naming its source, one color per source: **SYSLOG**,
+**DOCKER**, **KERNEL** (dmesg), **HOST** (files of `/var/log`), **AUDIT**, **FILE** and **LS**
+(LogStream's own logs).
 
 Shortcuts: `/` focuses the search box, `Esc` clears it. Clicking a row shows all its fields.
 
@@ -254,7 +257,7 @@ also be changed (saved in `/data/audit.json`; `AUDIT_LOG` sets the default path)
   `auid` (login user ID), `user` (its name, or the uid when it cannot be resolved), `exe`, `res`
   (`success` / `failed`, from `res=` or `success=`) and `key` (rule key). Severity is `warning`
   when the result is a failure, `info` otherwise; facility `authpriv`; source `audit`. In the
-  list, audit lines have a violet left edge, a light tint and an **AUDIT** badge before the app;
+  list, audit lines have a violet left edge, a light tint and the **AUDIT** source badge;
   the **Source** filter shows them alone.
 - **Names**: `user` comes from the enriched format (`log_format = ENRICHED`, the default of
   recent auditd versions), else from the host's `/etc/passwd` when it is mounted as
@@ -310,10 +313,12 @@ optional **host** (default: `/host/etc/hostname`, else `localhost`) and a switch
   not fit the chosen format is kept as plain text. Fields named like LogStream's own ones
   (`host`, `app`…) are renamed `field_host`, `field_app`…
 - **Display**: source `file`, the path in `log_file`. In the list these lines have a green left
-  edge, a light tint and a **FILE** badge before the app; the **Source** filter shows them alone.
+  edge, a light tint and the **FILE** source badge; the **Source** filter shows them alone.
 - When a file is turned on, the lines of the last hour already in it are read first
-  (`FILES_BACKFILL`, `0` = only new lines; lines without a timestamp are then left out); a file
-  created later is read from its start. The inode and an offset are saved per file in
+  (`FILES_BACKFILL`, `0` = only new lines; lines without a timestamp are then left out). The
+  start of that hour is found by bisection on the dates of the lines, so a large file that holds
+  months of logs is not read from its start; a file created later is read from its start, up to
+  64 MB per second, the status showing what is left to read. The inode and an offset are saved per file in
   `/data/files-state.json`: after a restart reading resumes there; after a rotation
   (`access.log` renamed `access.log.1`), the end of the old file is read before the new one;
   a file truncated in place (`copytruncate`) is read again from its start.
@@ -349,6 +354,75 @@ chosen in Settings), `severity`, `facility`, `host`, `host_ip`, `app`, `pid`, `s
 
 From the command line: `curl -o logs.csv 'localhost:8080/api/export.csv?q=error&range=24h&tz=Europe/Paris'`.
 
+## Reading API (n8n, scripts)
+
+`GET /api/v1/logs` returns the stored logs that follow a cursor, oldest first, so that a tool
+called on a schedule (n8n, a script, a SIEM connector) gets every new log once, without gaps
+or duplicates, even when many logs share the same timestamp.
+
+Access goes through an **API token**, created in *Settings > API* by an administrator and sent
+as `Authorization: Bearer lst_…`. A token is read-only and only opens `/api/v1/`; it works
+whatever the [authentication](#authentication) mode (`local`, `oidc`, or none). It is shown once,
+when it is created: LogStream only keeps its SHA-256 hash, in `/data/api_tokens.json`.
+Revoking it (then Save) cuts off access at once; the list shows when and from where each token was last
+used. A logged-in browser session can call the route too.
+
+| Parameter | Meaning |
+|---|---|
+| `query` | LogsQL filter (default `*`), e.g. `severity:in(err,crit) app:sshd`. Pipes (`\| ...`) are refused. |
+| `since` | where to start: the `next_cursor` of the previous answer, an RFC 3339 time (`2026-11-01T08:00:00Z`) or a duration back from now (`15m`, `6h`, `7d`). Default: `1h`. |
+| `limit` | rows per answer, 1 to 10,000 (default 1,000). |
+| `lag` | only logs received at least this long ago are returned (default `30s`, at most `1h`), so that the logs still in the queue or waiting for their DNS name are not skipped. |
+
+```json
+{
+  "rows": [{"_time": "2026-11-01T14:02:11.52Z", "_msg": "…", "host": "fw01", "app": "sshd", "severity": "err", …}],
+  "next_cursor": "eyJ0IjoiMjAyNi0xMS0wMVQxNDowMjoxMS41MjAwMDAwMDFaIiwicSI6IjJlYjE4YzNhIn0",
+  "has_more": false,
+  "until": "2026-11-01T14:05:30.1Z"
+}
+```
+
+Rows have the stored fields (as in the UI's JSON view), sorted by `_time`, the reception time.
+Always send back `next_cursor`, even when `rows` is empty. `has_more: true` means more logs are
+ready: call again at once. `waiting: true` means logs from the [disk buffer](#configuration) are
+still being sent to VictoriaLogs: nothing is returned until they are stored, so none is missed.
+The cursor belongs to its `query`: changing the query keeps the position in time. Errors come
+as `{"error", "code"}` with status 400 (bad parameter or LogsQL), 401 (invalid or revoked token)
+or 502 (VictoriaLogs unreachable).
+
+```bash
+curl -H "Authorization: Bearer lst_…" \
+  "https://logs.example.org/api/v1/logs?query=severity:err&since=15m&limit=1000"
+```
+
+### Example: n8n workflow
+
+Four nodes, the cursor kept in the workflow's static data between runs:
+
+1. **Schedule Trigger**: every minute.
+2. **Code** (*Run Once for All Items*), reads the cursor:
+   ```js
+   const data = $getWorkflowStaticData('global');
+   return [{ json: { since: data.cursor || '15m' } }];
+   ```
+3. **HTTP Request**: method `GET`, URL `https://logs.example.org/api/v1/logs`.
+   *Authentication*: Generic Credential Type > Header Auth, with name `Authorization` and value
+   `Bearer lst_…`. *Send Query Parameters*: `query` = `severity:in(emerg,alert,crit,err)`,
+   `since` = `{{ $json.since }}`, `limit` = `1000`.
+4. **Code** (*Run Once for All Items*), saves the cursor and turns each log into an item:
+   ```js
+   const data = $getWorkflowStaticData('global');
+   const page = $input.first().json;
+   data.cursor = page.next_cursor;
+   return page.rows.map((row) => ({ json: row }));
+   ```
+
+Then connect what the logs should feed (Slack, mail, ticket, database…). n8n only keeps static
+data for runs of the active workflow, not for manual test runs: while testing, each run starts
+again from `15m`. When more than 1,000 logs arrive per minute, `has_more` is true and the
+following runs catch up; raise `limit` or run the trigger more often if needed.
+
 ## Settings
 
 The gear icon opens the settings, organized in tabs. Changes show at once but are only kept
@@ -403,6 +477,11 @@ and log sources are saved on the server. Purging and resetting column widths act
   (already set in `docker-compose.yml`). The feature is off by default: set `ALLOW_PURGE=true`
   to allow it. Any admin can then purge: turn on [authentication](#authentication) if the UI
   is reachable by others.
+- **API**: API tokens for the [reading API](#reading-api-n8n-scripts): add one with a name,
+  revoke one, see when and from where each was last used. Additions and revocations wait for
+  **Save** like the other settings (**Cancel** drops them). After Save, each new token is shown
+  once with a copy button, and the curl example below uses it, highlighted, with its own copy
+  button. Reserved to administrators.
 
 ## Authentication
 
@@ -572,6 +651,8 @@ To update one of them:
 | `hub.go` | pushes new messages to browsers (SSE) |
 | `rdns.go` | cached reverse DNS lookups |
 | `export.go` | streamed CSV export |
+| `logsapi.go` | reading API `/api/v1/logs`: cursor, groups of logs with the same time |
+| `apitokens.go` | API tokens: storage (hashed), Bearer check, `/api/tokens` |
 | `docker.go` | Docker container logs (API, followers, positions, level detection) |
 | `syslogserver.go` | syslog listeners opened and closed from Settings > Log sources |
 | `hostlogs.go`, `journal.go` | host system logs: systemd journal reader (no `journalctl`) and `/var/log` follower |
@@ -582,6 +663,7 @@ To update one of them:
 | `tags.go` | color tag storage |
 | `presets.go`, `presets.json` | color tag presets (`/api/presets`), built-in list |
 | `api.go` | `/api/*` HTTP routes |
+| `static.go` | embedded UI files: ETag per file, revalidated on each load (`no-cache`, 304 when unchanged) so an update is always picked up; fonts cached 30 days |
 | `web/` | UI (HTML, CSS, plain JavaScript, no build step), embedded in the binary; translations live in `web/app.js` (`I18N`), and in `web/login.html` for the login page |
 
 ## Note

api.go

@@ -28,11 +28,16 @@ type API struct {
 	files      *Files
 	selfLog    *SelfLog
 	dbstats    dbStatsCache
+	tokens     *TokenStore
 }
 
 func (a *API) Routes(mux *http.ServeMux) {
 	mux.HandleFunc("GET /healthz", a.health)
 	mux.HandleFunc("GET /api/logs", a.logs)
+	mux.HandleFunc("GET /api/v1/logs", a.logsV1)
+	mux.HandleFunc("GET /api/tokens", a.listTokens)
+	mux.HandleFunc("POST /api/tokens", a.createToken)
+	mux.HandleFunc("DELETE /api/tokens/{id}", a.deleteToken)
 	mux.HandleFunc("GET /api/histogram", a.histogram)
 	mux.HandleFunc("GET /api/facets", a.facets)
 	mux.HandleFunc("GET /api/stream", a.stream)

apitokens.go

@@ -0,0 +1,254 @@
+package main
+
+import (
+	"crypto/rand"
+	"crypto/sha256"
+	"crypto/subtle"
+	"encoding/base64"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"log"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"time"
+	"unicode/utf8"
+)
+
+// API tokens: read-only access to /api/v1/ for scripts and automation tools
+// (n8n…), sent as "Authorization: Bearer lst_…". They are created and revoked
+// in Settings > API by an administrator, and work whatever the AUTH_MODE,
+// including without authentication. Only a SHA-256 of each token is kept, in
+// DATA_DIR/api_tokens.json: a token is shown once, when it is created.
+
+const (
+	tokenPrefix    = "lst_"
+	maxTokens      = 100
+	lastUsedPeriod = time.Minute // last use of a token is written to disk at most this often (or on a new IP)
+)
+
+var errTokenNotFound = errors.New("token not found")
+
+type apiToken struct {
+	ID       string    `json:"id"`
+	Name     string    `json:"name"`
+	Hint     string    `json:"hint"` // start of the token, to recognize it
+	Hash     string    `json:"hash"` // hex SHA-256 of the whole token
+	Created  time.Time `json:"created"`
+	LastUsed time.Time `json:"last_used,omitzero"`
+	LastIP   string    `json:"last_ip,omitempty"`
+}
+
+// apiTokenInfo is what the UI sees of a token (never its hash).
+type apiTokenInfo struct {
+	ID       string    `json:"id"`
+	Name     string    `json:"name"`
+	Hint     string    `json:"hint"`
+	Created  time.Time `json:"created"`
+	LastUsed time.Time `json:"last_used,omitzero"`
+	LastIP   string    `json:"last_ip,omitempty"`
+}
+
+type TokenStore struct {
+	path   string
+	mu     sync.Mutex
+	tokens []apiToken
+}
+
+func LoadTokenStore(path string) (*TokenStore, error) {
+	s := &TokenStore{path: path}
+	b, err := os.ReadFile(path)
+	if errors.Is(err, os.ErrNotExist) {
+		return s, nil
+	}
+	if err != nil {
+		return nil, err
+	}
+	if err := json.Unmarshal(b, &s.tokens); err != nil {
+		return nil, fmt.Errorf("%s: %w", path, err)
+	}
+	return s, nil
+}
+
+// save writes the file atomically, readable by its owner only; the caller holds the lock.
+func (s *TokenStore) save() error {
+	b, err := json.MarshalIndent(s.tokens, "", "  ")
+	if err != nil {
+		return err
+	}
+	if err := os.MkdirAll(filepath.Dir(s.path), 0o755); err != nil {
+		return err
+	}
+	tmp := s.path + ".tmp"
+	if err := os.WriteFile(tmp, b, 0o600); err != nil {
+		return err
+	}
+	return os.Rename(tmp, s.path)
+}
+
+func hashToken(token string) string {
+	sum := sha256.Sum256([]byte(token))
+	return hex.EncodeToString(sum[:])
+}
+
+func (s *TokenStore) List() []apiTokenInfo {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	out := make([]apiTokenInfo, len(s.tokens))
+	for i, t := range s.tokens {
+		out[i] = apiTokenInfo{ID: t.ID, Name: t.Name, Hint: t.Hint, Created: t.Created, LastUsed: t.LastUsed, LastIP: t.LastIP}
+	}
+	return out
+}
+
+// Create makes a new token and returns it in clear, the only time it exists so.
+func (s *TokenStore) Create(name string) (apiTokenInfo, string, error) {
+	name = strings.TrimSpace(name)
+	if name == "" || utf8.RuneCountInString(name) > 80 {
+		return apiTokenInfo{}, "", &codedError{code: "token_name", msg: "give the token a name of 1 to 80 characters"}
+	}
+	b := make([]byte, 32)
+	if _, err := rand.Read(b); err != nil {
+		return apiTokenInfo{}, "", err
+	}
+	secret := tokenPrefix + base64.RawURLEncoding.EncodeToString(b)
+	t := apiToken{ID: newID(), Name: name, Hint: secret[:len(tokenPrefix)+6], Hash: hashToken(secret), Created: time.Now().UTC()}
+
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if len(s.tokens) >= maxTokens {
+		return apiTokenInfo{}, "", &codedError{code: "token_max", msg: fmt.Sprintf("at most %d API tokens", maxTokens)}
+	}
+	s.tokens = append(s.tokens, t)
+	if err := s.save(); err != nil {
+		s.tokens = s.tokens[:len(s.tokens)-1]
+		return apiTokenInfo{}, "", err
+	}
+	return apiTokenInfo{ID: t.ID, Name: t.Name, Hint: t.Hint, Created: t.Created}, secret, nil
+}
+
+func (s *TokenStore) Delete(id string) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	for i, t := range s.tokens {
+		if t.ID == id {
+			s.tokens = append(s.tokens[:i:i], s.tokens[i+1:]...)
+			return s.save()
+		}
+	}
+	return errTokenNotFound
+}
+
+// Check tells whether token is a valid API token, and notes its use.
+func (s *TokenStore) Check(token, ip string) (name string, ok bool) {
+	if !strings.HasPrefix(token, tokenPrefix) {
+		return "", false
+	}
+	h := []byte(hashToken(token))
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	for i := range s.tokens {
+		t := &s.tokens[i]
+		if subtle.ConstantTimeCompare(h, []byte(t.Hash)) != 1 {
+			continue
+		}
+		now := time.Now().UTC()
+		stale := now.Sub(t.LastUsed) >= lastUsedPeriod || t.LastIP != ip
+		t.LastUsed, t.LastIP = now, ip
+		if stale {
+			if err := s.save(); err != nil {
+				log.Printf("api tokens: %v", err)
+			}
+		}
+		return t.Name, true
+	}
+	return "", false
+}
+
+// bearerToken returns the token of an "Authorization: Bearer …" header.
+func bearerToken(r *http.Request) (string, bool) {
+	scheme, token, ok := strings.Cut(r.Header.Get("Authorization"), " ")
+	if !ok || !strings.EqualFold(scheme, "Bearer") {
+		return "", false
+	}
+	return strings.TrimSpace(token), true
+}
+
+// tokenAuth serves the requests carrying an API token itself, read-only and
+// limited to /api/v1/, before the session login (auth); api is the handler
+// behind the login.
+func tokenAuth(tokens *TokenStore, auth, api http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		token, ok := bearerToken(r)
+		if !ok {
+			auth.ServeHTTP(w, r)
+			return
+		}
+		ip := clientIP(r)
+		if _, valid := tokens.Check(token, ip); !valid {
+			log.Printf("api: invalid or revoked token from %s", ip)
+			time.Sleep(loginFailDelay)
+			w.Header().Set("WWW-Authenticate", `Bearer realm="logstream"`)
+			writeErr(w, http.StatusUnauthorized, &codedError{code: "token", msg: "invalid or revoked API token"})
+			return
+		}
+		if !strings.HasPrefix(r.URL.Path, "/api/v1/") || !isSafeMethod(r.Method) {
+			writeErr(w, http.StatusForbidden, &codedError{code: "token_scope", msg: "API tokens only give read access to /api/v1/"})
+			return
+		}
+		api.ServeHTTP(w, asViewer(r))
+	})
+}
+
+// GET /api/tokens: the API tokens (Settings > API), for administrators.
+func (a *API) listTokens(w http.ResponseWriter, r *http.Request) {
+	if isViewer(r) {
+		writeErr(w, http.StatusForbidden, &codedError{code: "read_only", msg: "read-only account: changes are reserved to administrators"})
+		return
+	}
+	writeJSON(w, http.StatusOK, a.tokens.List())
+}
+
+// POST /api/tokens {"name": "n8n"}: returns the new token, with "token" in clear.
+func (a *API) createToken(w http.ResponseWriter, r *http.Request) {
+	var body struct {
+		Name string `json:"name"`
+	}
+	if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&body); err != nil {
+		writeErr(w, http.StatusBadRequest, err)
+		return
+	}
+	info, secret, err := a.tokens.Create(body.Name)
+	var ce *codedError
+	switch {
+	case errors.As(err, &ce):
+		writeErr(w, http.StatusBadRequest, err)
+		return
+	case err != nil:
+		writeErr(w, http.StatusInternalServerError, err)
+		return
+	}
+	log.Printf("api token %q created from %s", info.Name, clientIP(r))
+	writeJSON(w, http.StatusCreated, struct {
+		apiTokenInfo
+		Token string `json:"token"`
+	}{info, secret})
+}
+
+// DELETE /api/tokens/{id}: revokes a token at once.
+func (a *API) deleteToken(w http.ResponseWriter, r *http.Request) {
+	err := a.tokens.Delete(r.PathValue("id"))
+	switch {
+	case errors.Is(err, errTokenNotFound):
+		writeErr(w, http.StatusNotFound, err)
+	case err != nil:
+		writeErr(w, http.StatusInternalServerError, err)
+	default:
+		log.Printf("api token %s revoked from %s", r.PathValue("id"), clientIP(r))
+		w.WriteHeader(http.StatusNoContent)
+	}
+}

apitokens_test.go

@@ -0,0 +1,176 @@
+package main
+
+import (
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestTokenStore(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "api_tokens.json")
+	s, err := LoadTokenStore(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, _, err := s.Create("  "); err == nil {
+		t.Fatal("a token without a name should be refused")
+	}
+	info, secret, err := s.Create("n8n")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !strings.HasPrefix(secret, tokenPrefix) || len(secret) < 40 || !strings.HasPrefix(secret, info.Hint) {
+		t.Fatalf("token %q, hint %q", secret, info.Hint)
+	}
+	b, _ := os.ReadFile(path)
+	if strings.Contains(string(b), secret) || !strings.Contains(string(b), hashToken(secret)) {
+		t.Fatal("the file must hold the hash of the token, not the token")
+	}
+	if st, _ := os.Stat(path); st.Mode().Perm() != 0o600 {
+		t.Fatalf("file mode %v, want 0600", st.Mode().Perm())
+	}
+	if _, ok := s.Check(secret+"x", "10.0.0.1"); ok {
+		t.Fatal("a wrong token was accepted")
+	}
+	if name, ok := s.Check(secret, "10.0.0.1"); !ok || name != "n8n" {
+		t.Fatal("the token was refused")
+	}
+
+	// Reloaded from disk (restart): still valid, last use kept.
+	s2, err := LoadTokenStore(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if l := s2.List(); len(l) != 1 || l[0].LastIP != "10.0.0.1" || l[0].LastUsed.IsZero() {
+		t.Fatalf("after reload: %+v", l)
+	}
+	if _, ok := s2.Check(secret, "10.0.0.2"); !ok {
+		t.Fatal("token refused after reload")
+	}
+	if err := s2.Delete(info.ID); err != nil {
+		t.Fatal(err)
+	}
+	if _, ok := s2.Check(secret, "10.0.0.2"); ok {
+		t.Fatal("a revoked token was accepted")
+	}
+	if err := s2.Delete(info.ID); err != errTokenNotFound {
+		t.Fatalf("second delete: %v", err)
+	}
+}
+
+func TestTokenAuth(t *testing.T) {
+	loginFailDelay = 0
+	tokens, _ := LoadTokenStore(filepath.Join(t.TempDir(), "api_tokens.json"))
+	_, secret, _ := tokens.Create("n8n")
+
+	var reached string
+	mux := http.NewServeMux()
+	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+		reached = r.URL.Path
+		if isViewer(r) {
+			reached += " viewer"
+		}
+	})
+	app := readOnly(mux)
+	login := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { writeAuthRequired(w) })
+	h := tokenAuth(tokens, login, app)
+
+	call := func(method, path, auth string) int {
+		reached = ""
+		req := httptest.NewRequest(method, path, nil)
+		if auth != "" {
+			req.Header.Set("Authorization", auth)
+		}
+		rec := httptest.NewRecorder()
+		h.ServeHTTP(rec, req)
+		return rec.Code
+	}
+	if code := call("GET", "/api/v1/logs", "Bearer "+secret); code != http.StatusOK || reached != "/api/v1/logs viewer" {
+		t.Fatalf("valid token: %d, reached %q", code, reached)
+	}
+	if code := call("GET", "/api/v1/logs", "bearer "+secret); code != http.StatusOK {
+		t.Fatalf("lower-case scheme: %d", code)
+	}
+	for _, tc := range []struct {
+		method, path, auth string
+		want               int
+	}{
+		{"GET", "/api/v1/logs", "Bearer lst_wrong", http.StatusUnauthorized},
+		{"GET", "/api/v1/logs", "Bearer ", http.StatusUnauthorized},
+		{"GET", "/api/tags", "Bearer " + secret, http.StatusForbidden},
+		{"GET", "/api/tokens", "Bearer " + secret, http.StatusForbidden},
+		{"POST", "/api/v1/logs", "Bearer " + secret, http.StatusForbidden},
+		{"GET", "/api/v1/logs", "", http.StatusUnauthorized},           // no token: the login decides
+		{"GET", "/api/v1/logs", "Basic dTpw", http.StatusUnauthorized}, // Basic goes to the login
+	} {
+		if code := call(tc.method, tc.path, tc.auth); code != tc.want || reached != "" {
+			t.Errorf("%s %s %q: %d (reached %q), want %d", tc.method, tc.path, tc.auth, code, reached, tc.want)
+		}
+	}
+}
+
+// With local authentication, a token opens /api/v1/ without a session.
+func TestTokenWithLocalAuth(t *testing.T) {
+	loginFailDelay = 0
+	dir := t.TempDir()
+	tokens, _ := LoadTokenStore(filepath.Join(dir, "api_tokens.json"))
+	_, secret, _ := tokens.Create("script")
+	app := readOnly(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("ok")) }))
+	auth, err := newAuth(authConfig{mode: "local", user: "admin", pass: "pw", dataDir: dir, sessionTTL: time.Hour}, app)
+	if err != nil {
+		t.Fatal(err)
+	}
+	h := tokenAuth(tokens, auth, app)
+	req := httptest.NewRequest("GET", "/api/v1/logs", nil)
+	rec := httptest.NewRecorder()
+	h.ServeHTTP(rec, req)
+	if rec.Code != http.StatusUnauthorized {
+		t.Fatalf("without token: %d", rec.Code)
+	}
+	req.Header.Set("Authorization", "Bearer "+secret)
+	rec = httptest.NewRecorder()
+	h.ServeHTTP(rec, req)
+	if rec.Code != http.StatusOK || rec.Body.String() != "ok" {
+		t.Fatalf("with token: %d %s", rec.Code, rec.Body)
+	}
+}
+
+func TestTokenHandlers(t *testing.T) {
+	tokens, _ := LoadTokenStore(filepath.Join(t.TempDir(), "api_tokens.json"))
+	a := &API{tokens: tokens}
+	mux := http.NewServeMux()
+	a.Routes(mux)
+	do := func(r *http.Request) *httptest.ResponseRecorder {
+		rec := httptest.NewRecorder()
+		readOnly(mux).ServeHTTP(rec, r)
+		return rec
+	}
+	rec := do(httptest.NewRequest("POST", "/api/tokens", strings.NewReader(`{"name":"n8n"}`)))
+	var created struct {
+		ID, Token, Hint string
+	}
+	if err := json.Unmarshal(rec.Body.Bytes(), &created); rec.Code != http.StatusCreated || err != nil || created.Token == "" {
+		t.Fatalf("create: %d %s", rec.Code, rec.Body)
+	}
+	rec = do(httptest.NewRequest("GET", "/api/tokens", nil))
+	if rec.Code != http.StatusOK || strings.Contains(rec.Body.String(), created.Token) || strings.Contains(rec.Body.String(), "hash") {
+		t.Fatalf("list must not show the token or its hash: %s", rec.Body)
+	}
+	if rec := do(asViewer(httptest.NewRequest("GET", "/api/tokens", nil))); rec.Code != http.StatusForbidden {
+		t.Fatalf("viewer list: %d", rec.Code)
+	}
+	if rec := do(asViewer(httptest.NewRequest("DELETE", "/api/tokens/"+created.ID, nil))); rec.Code != http.StatusForbidden {
+		t.Fatalf("viewer delete: %d", rec.Code)
+	}
+	if rec := do(httptest.NewRequest("DELETE", "/api/tokens/"+created.ID, nil)); rec.Code != http.StatusNoContent {
+		t.Fatalf("delete: %d", rec.Code)
+	}
+	if rec := do(httptest.NewRequest("DELETE", "/api/tokens/"+created.ID, nil)); rec.Code != http.StatusNotFound {
+		t.Fatalf("delete again: %d", rec.Code)
+	}
+}

docker-compose.yml

@@ -6,10 +6,10 @@ services:
     depends_on:
       - victorialogs
       - docker-proxy
-    ports:
-      - "${SYSLOG_PORT:-5514}:5514/udp"
-      - "${SYSLOG_PORT:-5514}:5514/tcp"
-      - "${HTTP_PORT:-8989}:8080"
+#    ports:
+#      - "${SYSLOG_PORT:-5514}:5514/udp"
+#      - "${SYSLOG_PORT:-5514}:5514/tcp"
+#      - "${HTTP_PORT:-8989}:8080"
     environment:
       VLOGS_URL: http://victorialogs:9428
       SYSLOG_PUBLIC_PORT: ${SYSLOG_PORT:-514}     # le port d'ecoute syslog par defaut (attention aux ports <1024)
@@ -60,12 +60,12 @@ services:
       - /run/log/journal:/host/run/log/journal:ro     # journal systemd volatile
       # audit Linux (auditd), source a activer dans Reglages > Sources ; le fichier doit etre
       # lisible par l'UID 10001 : log_group = adm dans /etc/audit/auditd.conf (voir le README)
-      # - /var/log/audit:/var/log/audit:ro
+      - /var/log/audit:/var/log/audit:ro
       # noms des utilisateurs et de l'hote pour les evenements d'audit (optionnel)
-      # - /etc/passwd:/host/etc/passwd:ro
-      # - /etc/hostname:/host/etc/hostname:ro
+      - /etc/passwd:/host/etc/passwd:ro
+      - /etc/hostname:/host/etc/hostname:ro
       # fichiers de logs d'autres programmes (Traefik, nginx...), a declarer dans Reglages > Sources > Fichiers de logs
-      # - /var/log/traefik:/logs/traefik:ro
+      - /home/traefik/log:/home/logs/traefik:ro
       # logo de la page de connexion, avec LOGIN_LOGO=/config/logo.png dans .env
       # - ./logo.png:/config/logo.png:ro
       # prereglages de tags personnalises, avec PRESETS_FILE=/config/presets.json dans .env
@@ -102,7 +102,12 @@ services:
       - /var/run/docker.sock:/var/run/docker.sock:ro
     labels:
       logstream.exclude: "true"      # pour exclure les logs propres de logstream
+    networks:
+      - traefik-net
 
+networks:
+  traefik-net:
+    external: true
 volumes:
   logstream-data:
   vlogs-data:

filesource.go

@@ -53,6 +53,7 @@ type fileSourceStatus struct {
 	Open     bool
 	Read     uint64
 	Detected string // format found when the source is in auto
+	Behind   int64  // bytes left to read (history being read)
 	Code     string
 	Error    string
 }
@@ -79,6 +80,7 @@ type fileTail struct {
 	known   bool  // readOff is set
 	pending *pendingEntry
 	auto    *fileParser // format detected from the file, in auto
+	behind  int64       // bytes left to read after the last poll
 }
 
 // Files follows log files of other programs (Traefik, nginx, applications…)
@@ -265,10 +267,11 @@ func (f *Files) Status() map[string]any {
 	for _, s := range f.cfg {
 		m := map[string]any{
 			"id": s.ID, "name": s.Name, "path": s.Path, "format": s.Format, "regex": s.Regex,
-			"host": s.Host, "enabled": s.Enabled, "open": false, "read": 0, "detected": "", "code": "", "error": "",
+			"host": s.Host, "enabled": s.Enabled, "open": false, "read": 0, "detected": "", "behind": 0, "code": "", "error": "",
 		}
 		if st := f.status[s.ID]; st != nil && s.Enabled {
 			m["open"], m["read"], m["detected"], m["code"], m["error"] = st.Open, st.Read, st.Detected, st.Code, st.Error
+			m["behind"] = st.Behind
 		}
 		list = append(list, m)
 	}
@@ -346,7 +349,8 @@ func (f *Files) poll(now time.Time) {
 }
 
 const (
-	maxFileRead       = 4 << 20         // per file and per poll
+	maxFileRead       = 4 << 20         // per read
+	maxFileReads      = 16              // reads per file and per poll (64 MB)
 	fileContinueDelay = 2 * time.Second // an entry waits this long for continuation lines
 )
 
@@ -370,6 +374,10 @@ func (f *Files) setStatus(id string, err error) {
 	}
 	changed := code != st.Code
 	st.Open, st.Detected, st.Code, st.Error = err == nil, detected, code, detail
+	st.Behind = 0
+	if tl := f.tails[id]; tl != nil && err == nil {
+		st.Behind = tl.behind
+	}
 	name := id
 	for _, s := range f.cfg {
 		if s.ID == id {
@@ -384,9 +392,12 @@ func (f *Files) setStatus(id string, err error) {
 
 func (f *Files) count(id string) {
 	f.mu.Lock()
-	if st := f.status[id]; st != nil {
-		st.Read++
+	st := f.status[id]
+	if st == nil { // first poll of the file: its status comes after
+		st = &fileSourceStatus{}
+		f.status[id] = st
 	}
+	st.Read++
 	f.mu.Unlock()
 }
 
@@ -430,11 +441,12 @@ func (f *Files) pollFile(s *fileSource, now time.Time) error {
 		pos = filePos{Ino: ino}
 		if first {
 			// Turned on: the last FILES_BACKFILL of what is already there
-			// (0: only new lines).
-			if f.backfill > 0 {
-				pos.Until = fi.Size()
-			} else {
+			// (0: only new lines), found by date rather than read from the
+			// start of a file that can hold months of logs.
 			pos.Off = fi.Size()
+			if p := f.parserOf(s, tl, path); f.backfill > 0 && p != nil {
+				pos.Off = backfillStart(path, p, fi.Size(), f.st.Started[s.ID].Add(-f.backfill), now)
+				pos.Until = fi.Size()
 			}
 		}
 		f.st.Pos[s.ID], f.dirty = pos, true
@@ -469,18 +481,78 @@ func (f *Files) pollFile(s *fileSource, now time.Time) error {
 		if p == nil {
 			return nil // nothing complete to detect the format from yet
 		}
-		next, _, err := f.readFrom(s, tl, p, path, tl.readOff, pos.Until, now)
+		for i := 0; i < maxFileReads; i++ {
+			next, more, err := f.readFrom(s, tl, p, path, tl.readOff, pos.Until, now)
 			if err != nil {
 				return err
 			}
 			tl.readOff = next
+			if !more {
+				break
 			}
+		}
+	}
+	tl.behind = max(fi.Size()-tl.readOff, 0)
 	if pe := tl.pending; pe != nil && now.Sub(pe.seen) >= fileContinueDelay {
 		f.flushPending(s.ID, now)
 	}
 	return nil
 }
 
+// backfillStart finds where the lines dated notBefore or later start, by
+// bisection on the dates of the lines (logs are written in order). Undated
+// lines are left out of the history anyway: then only new lines are read.
+func backfillStart(path string, p *fileParser, size int64, notBefore, now time.Time) int64 {
+	fh, err := os.Open(path)
+	if err != nil {
+		return size
+	}
+	defer fh.Close()
+	buf := make([]byte, 64<<10)
+	// lineAt reads the first line starting at or after off (off itself when
+	// it is 0): its start and its date.
+	lineAt := func(off int64) (int64, time.Time, bool) {
+		n, _ := fh.ReadAt(buf, off)
+		b := buf[:n]
+		start := off
+		if off > 0 {
+			nl := bytes.IndexByte(b, '\n')
+			if nl < 0 {
+				return size, time.Time{}, false
+			}
+			b, start = b[nl+1:], off+int64(nl)+1
+		}
+		end := bytes.IndexByte(b, '\n')
+		if end < 0 {
+			return size, time.Time{}, false
+		}
+		r := p.parse(string(b[:end]), now)
+		return start, r.t, r.dated
+	}
+	if _, t, ok := lineAt(0); !ok || !t.Before(notBefore) {
+		if !ok {
+			// No date on the first line: try the last ones.
+			if _, _, ok := lineAt(max(size-int64(len(buf)), 0)); !ok {
+				return size
+			}
+		} else {
+			return 0 // all of it is recent
+		}
+	}
+	lo, hi := int64(0), size // line at lo: before notBefore; at hi: not
+	for hi-lo > int64(len(buf)) {
+		mid := lo + (hi-lo)/2
+		_, t, ok := lineAt(mid)
+		if ok && t.Before(notBefore) {
+			lo = mid
+		} else {
+			hi = mid
+		}
+	}
+	start, _, _ := lineAt(lo)
+	return min(start, size)
+}
+
 // readFrom parses the complete lines written after off and returns the new
 // offset; more tells whether the read window was full.
 func (f *Files) readFrom(s *fileSource, tl *fileTail, p *fileParser, path string, off, until int64, now time.Time) (int64, bool, error) {

filesource_test.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"fmt"
 	"os"
 	"path/filepath"
 	"strings"
@@ -382,3 +383,45 @@ func TestFilesAutoAndErrors(t *testing.T) {
 		t.Fatalf("re-enabled: got %q", msgs(got))
 	}
 }
+
+// A large access log: the last hour is found by date, not read from the
+// start, and every entry is counted from the first poll.
+func TestFilesBackfillLargeFile(t *testing.T) {
+	dir, data := t.TempDir(), t.TempDir()
+	path := filepath.Join(dir, "access.log")
+	now := time.Now()
+	line := func(at time.Time, n int) string {
+		return fmt.Sprintf(`86.246.223.62 - - [%s] "GET /wp-content/x.png HTTP/2.0" 200 640365 "-" "-" %d "vblog-io-router@file" "http://wordpress" 3ms`+"\n",
+			at.UTC().Format("02/Jan/2006:15:04:05 -0700"), n)
+	}
+	var b strings.Builder
+	const n = 200000 // one line every 3 s for a week: ~32 MB
+	for i := 0; i < n; i++ {
+		b.WriteString(line(now.Add(-time.Duration(n-i)*3*time.Second), i))
+	}
+	appendFile(t, path, b.String())
+	sink := &fileSink{}
+	f := NewFiles(dir, data, time.Hour, sink.add)
+	f.Configure(filesConfig{Sources: []fileSourceDef{{ID: "a1", Name: "traefik-log", Path: path, Enabled: true}}})
+	f.poll(now)
+	if pos := f.st.Pos["a1"]; pos.Off < int64(b.Len())*99/100 {
+		t.Errorf("history read from offset %d of %d", pos.Off, b.Len())
+	}
+	appendFile(t, path, line(now, n))
+	f.poll(now.Add(time.Second))
+	got := sink.take()
+	// The last hour (1200 lines, the oldest one on the edge) and the new line.
+	if len(got) < 1200 || len(got) > 1201 || got[0].App != "traefik-log" || got[len(got)-1].Extra["http_status"] != "200" {
+		t.Fatalf("got %d entries", len(got))
+	}
+	if st := f.Status()["sources"].([]map[string]any)[0]; st["read"] != uint64(len(got)) || st["behind"] != int64(0) || st["detected"] != fmtCLF {
+		t.Errorf("status %+v", st)
+	}
+	// Undated plain text: only new lines.
+	txt := filepath.Join(dir, "app.log")
+	appendFile(t, txt, strings.Repeat("no date here\n", 1000))
+	p := mustParser(t, fmtText, "")
+	if off := backfillStart(txt, p, 13000, now.Add(-time.Hour), now); off != 13000 {
+		t.Errorf("undated: %d", off)
+	}
+}

logsapi.go

@@ -0,0 +1,281 @@
+package main
+
+import (
+	"context"
+	"crypto/sha256"
+	"encoding/base64"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"sort"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// GET /api/v1/logs: incremental reading of the stored logs for automation
+// tools (n8n, scripts). Each answer gives the rows that follow a cursor,
+// oldest first, and the cursor to send next time; following the cursors
+// returns every row once, without gaps or duplicates.
+//
+// The cursor is a position on _time, which is the reception time given by
+// LogStream (see Entry.Record), so new rows always come after it. Rows only
+// count once they are older than the lag (30 s by default): a message can
+// wait a little in the queue or for its DNS name before VictoriaLogs makes it
+// visible. Several rows can share the same _time to the nanosecond; inside
+// such a group the rows are ordered by a hash of their content, and the
+// cursor remembers how many of them were already returned.
+
+const (
+	logsAPIDefaultLimit = 1000
+	logsAPIMaxLimit     = 10000
+	logsAPIDefaultLag   = 30 * time.Second
+	logsAPIMaxLag       = time.Hour
+	logsAPIDefaultSince = time.Hour
+	logsAPIMaxGroup     = 100000 // rows with the very same _time
+)
+
+// logsCursor: every row with _time < T has been returned, plus the first K
+// rows (in hash order) of those with _time == T. Q is a hash of the query:
+// K only means something for the same query.
+type logsCursor struct {
+	T time.Time `json:"t"`
+	K int       `json:"k,omitempty"`
+	Q string    `json:"q,omitempty"`
+}
+
+func (c logsCursor) encode() string {
+	b, _ := json.Marshal(c)
+	return base64.RawURLEncoding.EncodeToString(b)
+}
+
+func decodeCursor(s string) (logsCursor, bool) {
+	var c logsCursor
+	b, err := base64.RawURLEncoding.DecodeString(s)
+	if err != nil || json.Unmarshal(b, &c) != nil || c.T.IsZero() || c.K < 0 {
+		return logsCursor{}, false
+	}
+	return c, true
+}
+
+// parseSince reads the since parameter: a cursor from a previous answer, an
+// RFC 3339 time, or a duration back from now (90s, 15m, 6h, 7d). Empty: the
+// last hour.
+func parseSince(s string, now time.Time) (logsCursor, error) {
+	s = strings.TrimSpace(s)
+	if s == "" {
+		return logsCursor{T: now.Add(-logsAPIDefaultSince)}, nil
+	}
+	if t, err := time.Parse(time.RFC3339Nano, s); err == nil {
+		return logsCursor{T: t}, nil
+	}
+	if d, ok := parseAge(s); ok {
+		return logsCursor{T: now.Add(-d)}, nil
+	}
+	if c, ok := decodeCursor(s); ok {
+		return c, nil
+	}
+	return logsCursor{}, &codedError{code: "bad_since", msg: "since: expected a next_cursor, an RFC 3339 time or a duration such as 15m, 6h or 7d"}
+}
+
+// parseAge reads a Go duration, plus "Nd" for days.
+func parseAge(s string) (time.Duration, bool) {
+	if n, ok := strings.CutSuffix(s, "d"); ok {
+		days, err := strconv.Atoi(n)
+		if err != nil || days < 0 {
+			return 0, false
+		}
+		return time.Duration(days) * 24 * time.Hour, true
+	}
+	d, err := time.ParseDuration(s)
+	return d, err == nil && d >= 0
+}
+
+type logsPage struct {
+	Rows       []map[string]any `json:"rows"`
+	NextCursor string           `json:"next_cursor"`
+	HasMore    bool             `json:"has_more"`          // call again right away: more rows are ready
+	Waiting    bool             `json:"waiting,omitempty"` // the disk buffer is still being sent to VictoriaLogs
+	Until      string           `json:"until"`             // rows are returned up to this time (now - lag)
+}
+
+// GET /api/v1/logs?query=<LogsQL filter>&since=<cursor|time|duration>&limit=<n>&lag=<duration>
+func (a *API) logsV1(w http.ResponseWriter, r *http.Request) {
+	q := r.URL.Query()
+	filter := strings.TrimSpace(q.Get("query"))
+	if f, pipes := splitPipes(filter); strings.TrimSpace(pipes) != "" {
+		writeErr(w, http.StatusBadRequest, &codedError{code: "api_pipes", msg: "query: LogsQL pipes (| ...) are not supported, keep only the filter part"})
+		return
+	} else {
+		filter = strings.TrimSpace(f)
+	}
+	if filter == "" {
+		filter = "*"
+	}
+	limit := logsAPIDefaultLimit
+	if v := q.Get("limit"); v != "" {
+		n, err := strconv.Atoi(v)
+		if err != nil || n <= 0 || n > logsAPIMaxLimit {
+			writeErr(w, http.StatusBadRequest, &codedError{code: "bad_limit", msg: fmt.Sprintf("limit: expected 1 to %d", logsAPIMaxLimit)})
+			return
+		}
+		limit = n
+	}
+	lag := logsAPIDefaultLag
+	if v := q.Get("lag"); v != "" {
+		d, ok := parseAge(v)
+		if !ok || d > logsAPIMaxLag {
+			writeErr(w, http.StatusBadRequest, &codedError{code: "bad_lag", msg: "lag: expected a duration from 0s to 1h"})
+			return
+		}
+		lag = d
+	}
+	now := time.Now()
+	cur, err := parseSince(q.Get("since"), now)
+	if err != nil {
+		writeErr(w, http.StatusBadRequest, err)
+		return
+	}
+	qh := queryHash(filter)
+	if cur.Q != qh {
+		cur.K = 0 // the query changed: the rank inside the group no longer applies
+	}
+	cur.Q = qh
+
+	// Logs waiting in the disk buffer were received earlier: returning newer
+	// rows now would move the cursor past them.
+	if a.store.PendingSpool() > 0 {
+		writeJSON(w, http.StatusOK, logsPage{Rows: []map[string]any{}, NextCursor: cur.encode(), Waiting: true, Until: logsqlTime(cur.T)})
+		return
+	}
+	until := now.Add(-lag)
+	page, err := readLogsPage(r.Context(), a.store.Query, filter, cur, until, limit)
+	if err != nil {
+		writeErr(w, queryStatus(err), err)
+		return
+	}
+	a.annotateHosts(page.Rows)
+	writeJSON(w, http.StatusOK, page)
+}
+
+func queryHash(filter string) string {
+	sum := sha256.Sum256([]byte(filter))
+	return hex.EncodeToString(sum[:4])
+}
+
+type queryFunc func(ctx context.Context, query string) ([]map[string]any, error)
+
+// readLogsPage returns at most limit rows after cur, up to until included.
+func readLogsPage(ctx context.Context, run queryFunc, filter string, cur logsCursor, until time.Time, limit int) (logsPage, error) {
+	page := logsPage{Rows: []map[string]any{}, Until: logsqlTime(until)}
+	next := cur
+	start := cur.T
+	if cur.K > 0 {
+		// Rest of the group the previous page stopped in.
+		group, err := readGroup(ctx, run, filter, cur.T)
+		if err != nil {
+			return page, err
+		}
+		rest := group[min(cur.K, len(group)):]
+		if len(rest) >= limit {
+			page.Rows = rest[:limit]
+			next.K = cur.K + limit
+			page.HasMore = true
+			page.NextCursor = next.encode()
+			return page, nil
+		}
+		page.Rows = append(page.Rows, rest...)
+		start = cur.T.Add(time.Nanosecond)
+	}
+
+	left := limit - len(page.Rows)
+	var rows []map[string]any
+	if !start.After(until) {
+		q := fmt.Sprintf("_time:[%s, %s] (%s) | sort by (_time) | limit %d", logsqlTime(start), logsqlTime(until), filter, left+1)
+		var err error
+		if rows, err = run(ctx, q); err != nil {
+			return page, err
+		}
+	}
+	if len(rows) <= left {
+		// Everything up to until has been read.
+		page.Rows = append(page.Rows, rows...)
+		next = logsCursor{T: until.Add(time.Nanosecond), Q: cur.Q}
+		if next.T.Before(start) {
+			next.T = start
+		}
+		page.NextCursor = next.encode()
+		return page, nil
+	}
+
+	// One row too many: the group of the last time may be cut. Whole groups
+	// before it are returned; that last group starts the next page.
+	page.HasMore = true
+	last, err := rowTime(rows[left])
+	if err != nil {
+		return page, err
+	}
+	n := 0
+	for n < left {
+		t, err := rowTime(rows[n])
+		if err != nil {
+			return page, err
+		}
+		if !t.Before(last) {
+			break
+		}
+		n++
+	}
+	if n > 0 || len(page.Rows) > 0 {
+		page.Rows = append(page.Rows, rows[:n]...)
+		page.NextCursor = logsCursor{T: last, Q: cur.Q}.encode()
+		return page, nil
+	}
+	// The whole page is a single group: read it all to cut it in a stable order.
+	group, err := readGroup(ctx, run, filter, last)
+	if err != nil {
+		return page, err
+	}
+	page.Rows = group[:min(left, len(group))]
+	page.NextCursor = logsCursor{T: last, K: len(page.Rows), Q: cur.Q}.encode()
+	return page, nil
+}
+
+// readGroup returns the rows whose _time is exactly t, in a stable order.
+func readGroup(ctx context.Context, run queryFunc, filter string, t time.Time) ([]map[string]any, error) {
+	ts := logsqlTime(t)
+	rows, err := run(ctx, fmt.Sprintf("_time:[%s, %s] (%s) | limit %d", ts, ts, filter, logsAPIMaxGroup+1))
+	if err != nil {
+		return nil, err
+	}
+	if len(rows) > logsAPIMaxGroup {
+		return nil, fmt.Errorf("more than %d logs share the time %s", logsAPIMaxGroup, ts)
+	}
+	keys := make([]string, len(rows))
+	for i, row := range rows {
+		b, _ := json.Marshal(row) // map keys are sorted: same row, same bytes
+		sum := sha256.Sum256(b)
+		keys[i] = string(sum[:])
+	}
+	idx := make([]int, len(rows))
+	for i := range idx {
+		idx[i] = i
+	}
+	sort.SliceStable(idx, func(i, j int) bool { return keys[idx[i]] < keys[idx[j]] })
+	out := make([]map[string]any, len(rows))
+	for i, k := range idx {
+		out[i] = rows[k]
+	}
+	return out, nil
+}
+
+func rowTime(row map[string]any) (time.Time, error) {
+	s, _ := row["_time"].(string)
+	t, err := time.Parse(time.RFC3339Nano, s)
+	if err != nil {
+		return time.Time{}, errors.New("VictoriaLogs returned a row without a valid _time")
+	}
+	return t, nil
+}

logsapi_test.go

@@ -0,0 +1,276 @@
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"math/rand"
+	"net/http"
+	"net/http/httptest"
+	"regexp"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+)
+
+// cursorVL plays VictoriaLogs for the two queries of /api/v1/logs: a time
+// range sorted by _time, and an exact time. Like VictoriaLogs, it gives no
+// order among rows of the same _time: they are shuffled on every query.
+type cursorVL struct {
+	mu      sync.Mutex
+	rows    []map[string]any
+	rnd     *rand.Rand
+	queries []string
+}
+
+var cursorQuery = regexp.MustCompile(`^_time:\[([^,]+), ([^\]]+)] \((.*?)\)( \| sort by \(_time\))? \| limit (\d+)$`)
+
+func (v *cursorVL) add(t time.Time, msg string) {
+	v.mu.Lock()
+	defer v.mu.Unlock()
+	v.rows = append(v.rows, map[string]any{"_time": t.UTC().Format(time.RFC3339Nano), "_msg": msg, "host": "h1"})
+}
+
+func (v *cursorVL) RoundTrip(r *http.Request) (*http.Response, error) {
+	_ = r.ParseForm()
+	q := r.PostForm.Get("query")
+	m := cursorQuery.FindStringSubmatch(q)
+	if m == nil {
+		return &http.Response{StatusCode: http.StatusBadRequest, Body: io.NopCloser(strings.NewReader("unexpected query " + q)), Header: http.Header{}}, nil
+	}
+	from, _ := time.Parse(time.RFC3339Nano, m[1])
+	to, _ := time.Parse(time.RFC3339Nano, m[2])
+	limit, _ := strconv.Atoi(m[5])
+
+	v.mu.Lock()
+	defer v.mu.Unlock()
+	v.queries = append(v.queries, q)
+	var out []map[string]any
+	for _, row := range v.rows {
+		t, _ := time.Parse(time.RFC3339Nano, row["_time"].(string))
+		if !t.Before(from) && !t.After(to) {
+			out = append(out, row)
+		}
+	}
+	v.rnd.Shuffle(len(out), func(i, j int) { out[i], out[j] = out[j], out[i] })
+	if m[4] != "" {
+		sort.SliceStable(out, func(i, j int) bool {
+			a, _ := time.Parse(time.RFC3339Nano, out[i]["_time"].(string))
+			b, _ := time.Parse(time.RFC3339Nano, out[j]["_time"].(string))
+			return a.Before(b)
+		})
+	}
+	if len(out) > limit {
+		out = out[:limit]
+	}
+	var b strings.Builder
+	for _, row := range out {
+		line, _ := json.Marshal(row)
+		b.Write(line)
+		b.WriteByte('\n')
+	}
+	return &http.Response{StatusCode: http.StatusOK, Body: io.NopCloser(strings.NewReader(b.String())), Header: http.Header{}}, nil
+}
+
+func newCursorVL(seed int64) (*cursorVL, *Store) {
+	vl := &cursorVL{rnd: rand.New(rand.NewSource(seed))}
+	store := NewStore("http://vl", 10, 10, time.Second, nil)
+	store.client.Transport = vl
+	store.streamClient.Transport = vl
+	return vl, store
+}
+
+// fill adds rows by groups of the same _time, some groups larger than a
+// page, some with identical rows, and returns the multiset of the rows.
+func fill(vl *cursorVL, start time.Time, groups []int, tag string) map[string]int {
+	want := map[string]int{}
+	t := start
+	for g, size := range groups {
+		for i := 0; i < size; i++ {
+			msg := fmt.Sprintf("%s g%d r%d", tag, g, i%4) // r0..r3 repeat: identical rows
+			vl.add(t, msg)
+			want[rowKey(t, msg)]++
+		}
+		t = t.Add(time.Duration(1+g%3) * time.Microsecond)
+	}
+	return want
+}
+
+func rowKey(t time.Time, msg string) string { return t.UTC().Format(time.RFC3339Nano) + " " + msg }
+
+func TestLogsCursorNoLossNoDuplicate(t *testing.T) {
+	base := time.Date(2026, 11, 1, 10, 0, 0, 0, time.UTC)
+	groups := []int{1, 3, 1, 1, 7, 2, 1, 12, 1, 4, 4, 1, 25, 1, 2}
+	for _, limit := range []int{1, 2, 3, 5, 8, 1000} {
+		for seed := int64(1); seed <= 5; seed++ {
+			vl, store := newCursorVL(seed)
+			want := fill(vl, base, groups, "a")
+			got := map[string]int{}
+			cur := logsCursor{T: base.Add(-time.Minute)}
+			until := base.Add(time.Second)
+			var last time.Time
+			for calls := 0; ; calls++ {
+				if calls > 500 {
+					t.Fatalf("limit %d: no end", limit)
+				}
+				page, err := readLogsPage(context.Background(), store.Query, "*", cur, until, limit)
+				if err != nil {
+					t.Fatal(err)
+				}
+				if len(page.Rows) > limit {
+					t.Fatalf("limit %d: page of %d rows", limit, len(page.Rows))
+				}
+				for _, row := range page.Rows {
+					rt, _ := rowTime(row)
+					if rt.Before(last) {
+						t.Fatalf("limit %d: rows out of order (%s after %s)", limit, rt, last)
+					}
+					last = rt
+					got[rowKey(rt, row["_msg"].(string))]++
+				}
+				var ok bool
+				if cur, ok = decodeCursor(page.NextCursor); !ok {
+					t.Fatalf("bad cursor %q", page.NextCursor)
+				}
+				if !page.HasMore {
+					break
+				}
+				if len(page.Rows) == 0 {
+					t.Fatalf("limit %d: has_more with an empty page", limit)
+				}
+			}
+			if fmt.Sprint(got) != fmt.Sprint(want) {
+				t.Fatalf("limit %d seed %d:\n got %v\nwant %v", limit, seed, got, want)
+			}
+		}
+	}
+}
+
+// New logs arrive between the calls, as with a scheduled n8n workflow.
+func TestLogsCursorFollowsNewLogs(t *testing.T) {
+	vl, store := newCursorVL(7)
+	base := time.Date(2026, 11, 1, 10, 0, 0, 0, time.UTC)
+	want := map[string]int{}
+	got := map[string]int{}
+	cur := logsCursor{T: base}
+	for round := 0; round < 6; round++ {
+		start := base.Add(time.Duration(round) * time.Second)
+		for k, n := range fill(vl, start, []int{2, 5, 1, 3}, fmt.Sprintf("round%d", round)) {
+			want[k] += n
+		}
+		until := start.Add(500 * time.Millisecond) // after this round's rows
+		for {
+			page, err := readLogsPage(context.Background(), store.Query, "*", cur, until, 4)
+			if err != nil {
+				t.Fatal(err)
+			}
+			for _, row := range page.Rows {
+				rt, _ := rowTime(row)
+				got[rowKey(rt, row["_msg"].(string))]++
+			}
+			cur, _ = decodeCursor(page.NextCursor)
+			if !page.HasMore {
+				break
+			}
+		}
+		// Polling again with nothing new returns nothing.
+		page, err := readLogsPage(context.Background(), store.Query, "*", cur, until, 4)
+		if err != nil || len(page.Rows) != 0 || page.HasMore {
+			t.Fatalf("round %d: second poll returned %d rows (has_more %v, err %v)", round, len(page.Rows), page.HasMore, err)
+		}
+	}
+	if fmt.Sprint(got) != fmt.Sprint(want) {
+		t.Fatalf("got %v\nwant %v", got, want)
+	}
+}
+
+func TestParseSince(t *testing.T) {
+	now := time.Date(2026, 11, 1, 12, 0, 0, 0, time.UTC)
+	c := logsCursor{T: now.Add(-time.Minute), K: 3, Q: "abcd"}
+	for _, tc := range []struct {
+		in   string
+		want logsCursor
+	}{
+		{"", logsCursor{T: now.Add(-time.Hour)}},
+		{"15m", logsCursor{T: now.Add(-15 * time.Minute)}},
+		{"7d", logsCursor{T: now.Add(-7 * 24 * time.Hour)}},
+		{"2026-11-01T08:30:00.5Z", logsCursor{T: time.Date(2026, 11, 1, 8, 30, 0, 5e8, time.UTC)}},
+		{c.encode(), c},
+	} {
+		got, err := parseSince(tc.in, now)
+		if err != nil || !got.T.Equal(tc.want.T) || got.K != tc.want.K || got.Q != tc.want.Q {
+			t.Errorf("parseSince(%q) = %+v, %v; want %+v", tc.in, got, err, tc.want)
+		}
+	}
+	for _, bad := range []string{"yesterday", "-5m", "e30"} {
+		if _, err := parseSince(bad, now); err == nil {
+			t.Errorf("parseSince(%q): expected an error", bad)
+		}
+	}
+}
+
+func TestLogsV1Handler(t *testing.T) {
+	vl, store := newCursorVL(3)
+	a := &API{store: store, rdns: NewReverseDNS(false, "")}
+	old := time.Now().Add(-10 * time.Minute)
+	fill(vl, old, []int{2, 1}, "x")
+	vl.add(time.Now().Add(time.Hour), "in the future: after until")
+
+	get := func(url string) (int, logsPage, map[string]string) {
+		rec := httptest.NewRecorder()
+		a.logsV1(rec, httptest.NewRequest("GET", url, nil))
+		var page logsPage
+		var e map[string]string
+		_ = json.Unmarshal(rec.Body.Bytes(), &page)
+		_ = json.Unmarshal(rec.Body.Bytes(), &e)
+		return rec.Code, page, e
+	}
+	code, page, _ := get("/api/v1/logs?query=host:h1&since=1h")
+	if code != http.StatusOK || len(page.Rows) != 3 || page.HasMore || page.NextCursor == "" {
+		t.Fatalf("status %d, %d rows, has_more %v", code, len(page.Rows), page.HasMore)
+	}
+	if q := vl.queries[len(vl.queries)-1]; !strings.Contains(q, "(host:h1)") {
+		t.Fatalf("query filter not applied: %s", q)
+	}
+	code, page, _ = get("/api/v1/logs?query=host:h1&since=" + page.NextCursor)
+	if code != http.StatusOK || len(page.Rows) != 0 {
+		t.Fatalf("second call: status %d, %d rows", code, len(page.Rows))
+	}
+	for url, wantCode := range map[string]string{
+		"/api/v1/logs?query=" + "*%20|%20stats%20count()": "api_pipes",
+		"/api/v1/logs?limit=0":                            "bad_limit",
+		"/api/v1/logs?limit=20000":                        "bad_limit",
+		"/api/v1/logs?lag=2h":                             "bad_lag",
+		"/api/v1/logs?since=nope":                         "bad_since",
+	} {
+		if code, _, e := get(url); code != http.StatusBadRequest || e["code"] != wantCode {
+			t.Errorf("%s: status %d, %v; want 400 %s", url, code, e, wantCode)
+		}
+	}
+}
+
+func TestLogsV1WaitsForSpool(t *testing.T) {
+	spool, err := OpenSpool(t.TempDir(), 1<<20)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := spool.Write([]byte(`{"_msg":"x"}`+"\n"), 1); err != nil {
+		t.Fatal(err)
+	}
+	vl, _ := newCursorVL(1)
+	store := NewStore("http://vl", 10, 10, time.Second, spool)
+	store.streamClient.Transport = vl
+	a := &API{store: store}
+	rec := httptest.NewRecorder()
+	a.logsV1(rec, httptest.NewRequest("GET", "/api/v1/logs?since=2026-11-01T10:00:00Z", nil))
+	var page logsPage
+	_ = json.Unmarshal(rec.Body.Bytes(), &page)
+	c, _ := decodeCursor(page.NextCursor)
+	if rec.Code != http.StatusOK || !page.Waiting || len(page.Rows) != 0 || !c.T.Equal(time.Date(2026, 11, 1, 10, 0, 0, 0, time.UTC)) || len(vl.queries) != 0 {
+		t.Fatalf("status %d, waiting %v, %d rows, cursor %v, %d queries", rec.Code, page.Waiting, len(page.Rows), c.T, len(vl.queries))
+	}
+}

main.go

@@ -142,6 +142,10 @@ func main() {
 	if err != nil {
 		log.Fatalf("tags: %v", err)
 	}
+	tokens, err := LoadTokenStore(filepath.Join(cfg.dataDir, "api_tokens.json"))
+	if err != nil {
+		log.Fatalf("api tokens: %v", err)
+	}
 
 	var spool *Spool
 	if cfg.spoolMax > 0 {
@@ -200,7 +204,7 @@ func main() {
 	}
 	mux := http.NewServeMux()
 	presets := getenv("PRESETS_FILE", filepath.Join(cfg.dataDir, "presets.json"))
-	api := &API{store: store, hub: hub, tags: tags, presets: presets, rdns: rdns, allowPurge: cfg.allowPurge, exportMax: cfg.exportMax, syslog: syslogSrv}
+	api := &API{store: store, hub: hub, tags: tags, presets: presets, rdns: rdns, allowPurge: cfg.allowPurge, exportMax: cfg.exportMax, syslog: syslogSrv, tokens: tokens}
 	if cfg.dockerLogs {
 		dm, err := NewDockerManager(cfg.dockerHost, cfg.dataDir, cfg.backfill, sink)
 		if err != nil {
@@ -226,9 +230,10 @@ func main() {
 	go api.files.Run(ctx)
 	api.selfLog = selfLog
 	api.Routes(mux)
-	mux.Handle("GET /", http.FileServer(http.FS(static)))
+	mux.Handle("GET /", staticFiles(static))
 
-	handler, err := newAuth(cfg.auth, readOnly(mux))
+	app := readOnly(mux)
+	handler, err := newAuth(cfg.auth, app)
 	if err != nil {
 		log.Fatalf("auth: %v", err)
 	}
@@ -241,7 +246,8 @@ func main() {
 	if !authOn {
 		log.Printf("warning: no authentication (AUTH_USER is empty): anyone who can reach %s can read the logs and change the settings", cfg.httpAddr)
 	}
-	handler = secure(handler, contentSecurityPolicy(static), authOn)
+	// API tokens (Settings > API) are checked before the login, in every auth mode.
+	handler = secure(tokenAuth(tokens, handler, app), contentSecurityPolicy(static), authOn)
 	srv := &http.Server{
 		Addr:              cfg.httpAddr,
 		Handler:           handler,

static.go

@@ -0,0 +1,44 @@
+package main
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"io/fs"
+	"net/http"
+	"strings"
+)
+
+// staticFiles serves the embedded UI with an ETag per file (a hash of its
+// content, computed at startup). Pages, scripts and styles are revalidated on
+// every load ("no-cache"): an unchanged file costs a 304 without body, and the
+// files of a new version are always taken after an update. Fonts are kept for
+// 30 days.
+func staticFiles(static fs.FS) http.Handler {
+	etags := map[string]string{}
+	_ = fs.WalkDir(static, ".", func(path string, d fs.DirEntry, err error) error {
+		if err != nil || d.IsDir() {
+			return err
+		}
+		b, err := fs.ReadFile(static, path)
+		if err != nil {
+			return err
+		}
+		sum := sha256.Sum256(b)
+		etags["/"+path] = `"` + hex.EncodeToString(sum[:8]) + `"`
+		return nil
+	})
+	etags["/"] = etags["/index.html"]
+	files := http.FileServer(http.FS(static))
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if tag, ok := etags[r.URL.Path]; ok {
+			// http.FileServer answers If-None-Match with 304 from this header.
+			w.Header().Set("ETag", tag)
+			if strings.HasPrefix(r.URL.Path, "/fonts/") {
+				w.Header().Set("Cache-Control", "public, max-age=2592000")
+			} else {
+				w.Header().Set("Cache-Control", "no-cache")
+			}
+		}
+		files.ServeHTTP(w, r)
+	})
+}

static_test.go

@@ -0,0 +1,42 @@
+package main
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"testing"
+	"testing/fstest"
+)
+
+func TestStaticFilesCache(t *testing.T) {
+	h := staticFiles(fstest.MapFS{
+		"index.html":       {Data: []byte("<p>v1</p>")},
+		"app.js":           {Data: []byte("v1")},
+		"fonts/mono.woff2": {Data: []byte("font")},
+	})
+	get := func(path, etag string) *httptest.ResponseRecorder {
+		req := httptest.NewRequest("GET", path, nil)
+		if etag != "" {
+			req.Header.Set("If-None-Match", etag)
+		}
+		rec := httptest.NewRecorder()
+		h.ServeHTTP(rec, req)
+		return rec
+	}
+	rec := get("/app.js", "")
+	tag := rec.Header().Get("ETag")
+	if rec.Code != http.StatusOK || tag == "" || rec.Header().Get("Cache-Control") != "no-cache" {
+		t.Fatalf("app.js: %d, ETag %q, Cache-Control %q", rec.Code, tag, rec.Header().Get("Cache-Control"))
+	}
+	if rec := get("/app.js", tag); rec.Code != http.StatusNotModified || rec.Body.Len() != 0 {
+		t.Fatalf("revalidation: %d with %d bytes, want 304 without body", rec.Code, rec.Body.Len())
+	}
+	if rec := get("/app.js", `"old"`); rec.Code != http.StatusOK || rec.Body.String() != "v1" {
+		t.Fatalf("stale ETag: %d %q", rec.Code, rec.Body)
+	}
+	if rec := get("/", ""); rec.Code != http.StatusOK || rec.Header().Get("ETag") == "" || rec.Header().Get("Cache-Control") != "no-cache" {
+		t.Fatalf("index: %d %v", rec.Code, rec.Header())
+	}
+	if rec := get("/fonts/mono.woff2", ""); rec.Header().Get("Cache-Control") != "public, max-age=2592000" {
+		t.Fatalf("font Cache-Control %q", rec.Header().Get("Cache-Control"))
+	}
+}

store.go

@@ -423,6 +423,15 @@ func queryStatus(err error) int {
 	return http.StatusBadGateway
 }
 
+// PendingSpool is the number of logs waiting in the disk spool.
+func (s *Store) PendingSpool() int64 {
+	if s.spool == nil {
+		return 0
+	}
+	lines, _ := s.spool.Pending()
+	return lines
+}
+
 func (s *Store) Stats() map[string]any {
 	st := map[string]any{
 		"received":  s.received.Load(),

web/app.js

@@ -101,6 +101,7 @@ const I18N = {
     fileDraft: 'Fill in the name and the path to save this file.',
     fileOk: (r) => `Followed: ${r} entries since startup.`,
     fileDetected: (f) => ` Format detected: ${f}.`,
+    fileBehind: (mb) => ` Reading the history: ${mb} MB left.`,
     probeFound: (f) => `File found, format detected: ${f}.`,
     probeSample: 'Last line: ',
     file_empty: 'The file is empty for now: its format will be detected from its first lines.',
@@ -108,7 +109,7 @@ const I18N = {
     file_not_found: 'No file at this path: mount it in the container (see docker-compose.yml) and check the path.',
     file_permission: 'Permission denied: the file must be readable by UID 10001 or the adm group (see the README). ',
     file_read: 'Cannot read the file: ',
-    filesHelp: 'Follows the log file of another program, mounted read-only in the container (e.g. /var/log/traefik:/logs/traefik:ro, then /logs/traefik/access.log here). Once the path is filled in, the file is checked and its format detected from its last lines (JSON lines, logfmt, access logs of nginx, Apache or Traefik, else plain text); choose a format to force it, or a regular expression whose named groups time, level, msg, host and app fill these fields, the other groups giving extra fields. Each line gives an entry (badge FILE, app = the name); indented lines are added to the previous entry; for access logs 5xx are errors and 4xx warnings. When turned on, the last hour of dated lines is read first (FILES_BACKFILL); rotations are followed and reading resumes where it stopped.',
+    filesHelp: 'Follows the log file of another program, mounted read-only in the container (e.g. /var/log/traefik:/logs/traefik:ro, then /logs/traefik/access.log here). Once the path is filled in, the file is checked and its format detected from its last lines (JSON lines, logfmt, access logs of nginx, Apache or Traefik, else plain text); choose a format to force it, or a regular expression whose named groups time, level, msg, host and app fill these fields, the other groups giving extra fields. Each line gives an entry (badge FILE at the start of the line, app = the name); indented lines are added to the previous entry; for access logs 5xx are errors and 4xx warnings. When turned on, the last hour of dated lines is read first (FILES_BACKFILL); rotations are followed and reading resumes where it stopped.',
     auditTitle: 'Linux audit (auditd)',
     auditEnabled: 'Collect the audit events of this machine',
     auditPath: 'File',
@@ -173,6 +174,27 @@ const I18N = {
     exportNote: (n) => `Up to ${n} logs matching the current filters, newest first. Dates use the time zone chosen in Settings.`,
     exporting: 'Exporting…', exportDone: (f) => `Downloaded: ${f}`, exportError: 'Export failed: ',
     err_export_pipes: 'Export does not support LogsQL pipes (| …): keep only the filter part',
+    tabApi: 'API',
+    tokensTitle: 'API tokens',
+    tokensHelp: 'Tokens give scripts and automation tools (n8n…) read-only access to /api/v1/logs, whatever the authentication mode. Additions and revocations apply with Save. Only a hash is kept: a new token is shown once, right after Save.',
+    tokenCount: (n) => `${n} token${n > 1 ? 's' : ''}`,
+    tokenNone: 'No API token yet.',
+    tokenNamePh: 'Token name (n8n, backup script…)',
+    tokenCreate: 'Add a token',
+    tokenOnce: 'Copy this token now: it will not be shown again.',
+    tokenCopy: 'Copy',
+    tokenRevoke: 'Revoke',
+    tokenUndo: 'Undo',
+    tokenCreatePending: 'created on Save',
+    tokenRevokePending: 'revoked on Save',
+    tokenExampleCopy: 'Copy the command',
+    tokenCreated: (d) => `created ${d}`,
+    tokenUsed: (p) => `last used ${p.d}${p.ip ? ' from ' + p.ip : ''}`,
+    tokenNever: 'never used',
+    tokensUse: 'Usage',
+    tokensUseHelp: 'Send the token in an Authorization header (highlighted below). Each answer has next_cursor: pass it as since on the next call to get the following logs, without gaps or duplicates (see the README for an n8n workflow).',
+    err_token_name: 'Give the token a name of 1 to 80 characters',
+    err_token_max: 'Too many API tokens',
     tabLocale: 'Localization', tabFilters: 'Patterns', tabInterface: 'Appearance', tabData: 'Database',
     themeTitle: 'Theme', themeSystem: 'System', themeLight: 'Light', themeDark: 'Dark',
     themeHelp: 'System follows the light/dark preference of your computer or phone.',
@@ -227,7 +249,7 @@ const I18N = {
     hCappedTitle: 'The chosen division would exceed 300 intervals over this range: it has been enlarged.',
     hUnzoom: '× Back to stream', unitDay: 'd',
     toTop: 'Back to top',
-    colRcv: 'Received', colMt: 'Message time', colSev: 'Severity', colHost: 'Host', colApp: 'App', colCodes: 'Pattern', colMsg: 'Message',
+    colSrc: 'Source', colRcv: 'Received', colMt: 'Message time', colSev: 'Severity', colHost: 'Host', colApp: 'App', colCodes: 'Pattern', colMsg: 'Message',
     codesTitle: 'Codes of the color tags found in the message',
     tagCodeTitle: 'Code shown on the log lines this tag matches',
     colGrip: 'Drag to resize, double-click for the automatic width',
@@ -332,6 +354,7 @@ const I18N = {
     fileDraft: 'Renseignez le nom et le chemin pour enregistrer ce fichier.',
     fileOk: (r) => `Suivi : ${r} entrées depuis le démarrage.`,
     fileDetected: (f) => ` Format détecté : ${f}.`,
+    fileBehind: (mb) => ` Lecture de l\'historique : ${mb} Mo restants.`,
     probeFound: (f) => `Fichier trouvé, format détecté : ${f}.`,
     probeSample: 'Dernière ligne : ',
     file_empty: 'Le fichier est vide pour l\'instant : son format sera détecté à partir de ses premières lignes.',
@@ -339,7 +362,7 @@ const I18N = {
     file_not_found: 'Aucun fichier à ce chemin : montez-le dans le conteneur (voir docker-compose.yml) et vérifiez le chemin.',
     file_permission: 'Accès refusé : le fichier doit être lisible par l\'UID 10001 ou le groupe adm (voir le README). ',
     file_read: 'Lecture du fichier impossible : ',
-    filesHelp: 'Suit le fichier de logs d\'un autre programme, monté en lecture seule dans le conteneur (par ex. /var/log/traefik:/logs/traefik:ro, puis /logs/traefik/access.log ici). Une fois le chemin saisi, le fichier est vérifié et son format détecté à partir de ses dernières lignes (JSON lines, logfmt, journaux d\'accès de nginx, Apache ou Traefik, sinon texte brut) ; choisissez un format pour le forcer, ou une expression régulière dont les groupes nommés time, level, msg, host et app remplissent ces champs, les autres groupes donnant des champs supplémentaires. Chaque ligne donne une entrée (badge FILE, app = le nom) ; les lignes indentées sont rattachées à l\'entrée précédente ; pour les journaux d\'accès, 5xx en erreur et 4xx en warning. À l\'activation, la dernière heure de lignes datées est lue d\'abord (FILES_BACKFILL) ; les rotations sont suivies et la lecture reprend où elle s\'était arrêtée.',
+    filesHelp: 'Suit le fichier de logs d\'un autre programme, monté en lecture seule dans le conteneur (par ex. /var/log/traefik:/logs/traefik:ro, puis /logs/traefik/access.log ici). Une fois le chemin saisi, le fichier est vérifié et son format détecté à partir de ses dernières lignes (JSON lines, logfmt, journaux d\'accès de nginx, Apache ou Traefik, sinon texte brut) ; choisissez un format pour le forcer, ou une expression régulière dont les groupes nommés time, level, msg, host et app remplissent ces champs, les autres groupes donnant des champs supplémentaires. Chaque ligne donne une entrée (badge FILE en début de ligne, app = le nom) ; les lignes indentées sont rattachées à l\'entrée précédente ; pour les journaux d\'accès, 5xx en erreur et 4xx en warning. À l\'activation, la dernière heure de lignes datées est lue d\'abord (FILES_BACKFILL) ; les rotations sont suivies et la lecture reprend où elle s\'était arrêtée.',
     auditTitle: 'Audit Linux (auditd)',
     auditEnabled: 'Collecter les événements d\'audit de cette machine',
     auditPath: 'Fichier',
@@ -404,6 +427,27 @@ const I18N = {
     exportNote: (n) => `Jusqu'à ${n} logs correspondant aux filtres, du plus récent au plus ancien. Dates dans le fuseau choisi dans Paramètres.`,
     exporting: 'Export…', exportDone: (f) => `Téléchargé : ${f}`, exportError: 'Échec de l\'export : ',
     err_export_pipes: 'L\'export ne gère pas les pipes LogsQL (| …) : gardez seulement la partie filtre',
+    tabApi: 'API',
+    tokensTitle: "Jetons d'API",
+    tokensHelp: "Les jetons donnent aux scripts et outils d'automatisation (n8n…) un accès en lecture seule à /api/v1/logs, quel que soit le mode d'authentification. Ajouts et révocations s'appliquent avec Valider. Seule une empreinte est conservée : un nouveau jeton n'est affiché qu'une fois, juste après Valider.",
+    tokenCount: (n) => `${n} jeton${n > 1 ? 's' : ''}`,
+    tokenNone: "Aucun jeton d'API pour l'instant.",
+    tokenNamePh: 'Nom du jeton (n8n, script de sauvegarde…)',
+    tokenCreate: 'Ajouter un jeton',
+    tokenOnce: "Copiez ce jeton maintenant : il ne sera plus affiché.",
+    tokenCopy: 'Copier',
+    tokenRevoke: 'Révoquer',
+    tokenUndo: 'Annuler',
+    tokenCreatePending: 'créé en validant',
+    tokenRevokePending: 'révoqué en validant',
+    tokenExampleCopy: 'Copier la commande',
+    tokenCreated: (d) => `créé le ${d}`,
+    tokenUsed: (p) => `dernière utilisation ${p.d}${p.ip ? ' depuis ' + p.ip : ''}`,
+    tokenNever: 'jamais utilisé',
+    tokensUse: 'Utilisation',
+    tokensUseHelp: "Envoyez le jeton dans un en-tête Authorization (surligné ci-dessous). Chaque réponse contient next_cursor : passez-le dans since à l'appel suivant pour obtenir les logs qui suivent, sans trou ni doublon (voir le README pour un workflow n8n).",
+    err_token_name: 'Donnez au jeton un nom de 1 à 80 caractères',
+    err_token_max: "Trop de jetons d'API",
     tabLocale: 'Localisation et langue', tabFilters: 'Motifs de recherche', tabInterface: 'Affichage', tabData: 'Base de données',
     themeTitle: 'Thème', themeSystem: 'Système', themeLight: 'Clair', themeDark: 'Sombre',
     themeHelp: 'Système suit la préférence clair/sombre de votre ordinateur ou de votre téléphone.',
@@ -458,7 +502,7 @@ const I18N = {
     hCappedTitle: 'La division choisie dépasserait 300 intervalles sur cette plage : elle a été élargie.',
     hUnzoom: '× Revenir au flux', unitDay: 'j',
     toTop: 'Revenir en haut',
-    colRcv: 'Réception', colMt: 'Heure message', colSev: 'Sévérité', colHost: 'Hôte', colApp: 'App', colCodes: 'Motif de recherche', colMsg: 'Message',
+    colSrc: 'Source', colRcv: 'Réception', colMt: 'Heure message', colSev: 'Sévérité', colHost: 'Hôte', colApp: 'App', colCodes: 'Motif de recherche', colMsg: 'Message',
     codesTitle: 'Codes des tags de couleur trouvés dans le message',
     tagCodeTitle: 'Code affiché sur les lignes de log où ce tag est trouvé',
     colGrip: 'Glisser pour redimensionner, double-clic pour la largeur automatique',
@@ -983,15 +1027,18 @@ function fmtTime(s) {
   return `${day} ${p.h}:${p.m}:${p.s}<span class="ms">.${p.ms}</span>`;
 }
 
-// Sources with their own look in the list: a colored edge and a badge
-// before the app name (the severity badge stays the same).
-const SRC_BADGE = { audit: 'AUDIT', logstream: 'LS', file: 'FILE' };
+// Every row starts with a badge naming its source, one color per source
+// (--src-<type> in style.css). Audit events, log files and LogStream's own
+// logs also get a colored edge and a light tint (the severity badge stays the same).
+const SRC_BADGE = { syslog: 'SYSLOG', docker: 'DOCKER', kernel: 'KERNEL', host: 'HOST', audit: 'AUDIT', file: 'FILE', logstream: 'LS' };
+const srcOf = (r) => (SRC_BADGE[r.source_type] ? r.source_type : 'syslog'); // older entries have no source_type
 
 function rowHTML(r, isNew) {
   const sev = r.severity || '';
   const cls = SEV_CLASS[sev] || 'info';
-  const badge = SRC_BADGE[r.source_type];
-  return `<article class="row sev-${cls}${badge ? ' src-' + r.source_type : ''}${isNew ? ' new' : ''}" tabindex="0">`
+  const src = srcOf(r);
+  return `<article class="row sev-${cls} src-${src}${isNew ? ' new' : ''}" tabindex="0">`
+    + `<b class="src-badge">${SRC_BADGE[src]}</b>`
     + `<time class="rcv" title="${esc(t('rcvTitle'))}">${r.received ? esc(fmtFull(r.received)) : '—'}</time>`
     + `<time class="mt" title="${esc(t('msgTimeTitle'))}">${fmtTime(r.msg_time || r._time)}</time>`
     + `<span class="sev">${esc(sev || '—')}</span>`
@@ -999,7 +1046,7 @@ function rowHTML(r, isNew) {
       ? `<span class="host" data-act="host" title="${esc(hostTitle(r) + ' · ' + t('clickHost'))}">${esc(r.host_name || r.host)}</span>`
       : '<span class="host"></span>')
     + (r.app
-      ? `<span class="app${r.source_type === 'docker' ? ' proj' : ''}" data-act="app"${r.source_type === 'docker' ? ` style="--h:${hueOf(r.compose_project || r.container || r.app)}"` : ''} title="${esc((r.compose_project ? r.compose_project + '/' : '') + (r.container ? r.container + ' · ' : '') + r.app + ' · ' + t('clickApp'))}">${r.source_type === 'docker' ? DOCKER_ICON : ''}${badge ? `<b class="src-badge">${badge}</b>` : ''}${esc(r.app)}</span>`
+      ? `<span class="app${r.source_type === 'docker' ? ' proj' : ''}" data-act="app"${r.source_type === 'docker' ? ` style="--h:${hueOf(r.compose_project || r.container || r.app)}"` : ''} title="${esc((r.compose_project ? r.compose_project + '/' : '') + (r.container ? r.container + ' · ' : '') + r.app + ' · ' + t('clickApp'))}">${r.source_type === 'docker' ? DOCKER_ICON : ''}${esc(r.app)}</span>`
       : '<span class="app"></span>')
     + `<span class="codes">${codesHTML(r._msg)}</span>`
     + `<div class="msg">${msgHTML(r)}</div>`
@@ -2279,7 +2326,10 @@ function fileStatus(s) {
   if (s.draft) return [t('fileDraft'), 'muted'];
   if (!s.enabled) return [t('fileOff'), 'muted'];
   if (s.code) return [t('file_' + s.code) + (s.code === 'not_found' ? '' : s.error || ''), 'bad'];
-  if (s.open) return [t('fileOk', fmtNum(s.read)) + (s.format === 'auto' && s.detected ? t('fileDetected', t('fmt_' + s.detected)) : ''), 'good'];
+  if (s.open) {
+    return [t('fileOk', fmtNum(s.read)) + (s.format === 'auto' && s.detected ? t('fileDetected', t('fmt_' + s.detected)) : '')
+      + (s.behind > 1 << 20 ? t('fileBehind', fmtNum(Math.round(s.behind / (1 << 20)))) : ''), 'good'];
+  }
   return [t('hostWaiting'), 'muted'];
 }
 
@@ -2656,6 +2706,129 @@ $('#purgeBtn').addEventListener('click', async () => {
   }
 });
 
+/* ================= API tokens ================= */
+
+// Creations and revocations wait for Save like the other settings
+// (settings.pending.tokens = { create: [names], revoke: [ids] }); a new token
+// is shown once, after Save, with a curl example using it.
+const tokens = { list: null, error: null, fresh: [] };
+
+const pendingTokens = () => settings.pending.tokens || { create: [], revoke: [] };
+function pendTokens(p) {
+  if (p.create.length || p.revoke.length) settings.pending.tokens = p;
+  else delete settings.pending.tokens;
+  renderTokens();
+  renderSaveState();
+}
+
+function curlExample(token) {
+  const url = `${location.origin}/api/v1/logs?query=severity:err&since=15m&limit=1000`;
+  return {
+    text: `curl -H "Authorization: Bearer ${token}" \\\n  "${url}"`,
+    html: `curl -H "Authorization: Bearer <mark>${esc(token)}</mark>" \\\n  "${esc(url)}"`,
+  };
+}
+
+function renderTokens() {
+  const el = $('#tokenList');
+  const day = (x) => fmtFull(x).replace(/[.,]\d{3}(?=\D*$)/, '');
+  const p = pendingTokens();
+  $('#tokenCount').textContent = tokens.list ? t('tokenCount', tokens.list.length) : '';
+
+  // Tokens shown once after Save, and the curl example with the last one.
+  const fresh = tokens.fresh;
+  $('#tokenNew').hidden = !fresh.length;
+  $('#tokenNew').innerHTML = fresh.length ? `<p class="small">${esc(t('tokenOnce'))}</p>` + fresh.map((f, i) =>
+    `<div class="token-secret"><span class="token-label">${esc(f.name)}</span><code>${esc(f.token)}</code>`
+    + `<button class="btn" type="button" data-copy-token="${i}">${esc(t('tokenCopy'))}</button></div>`).join('') : '';
+  $('#tokenExample').innerHTML = curlExample(fresh.length ? fresh[fresh.length - 1].token : 'lst_…').html;
+
+  if (!tokens.list) {
+    el.innerHTML = tokens.error ? `<div class="token-empty bad">${esc(tokens.error)}</div>` : '';
+    return;
+  }
+  let html = tokens.list.map((k) => {
+    const gone = p.revoke.includes(k.id);
+    const used = k.last_used ? t('tokenUsed', { d: day(k.last_used), ip: k.last_ip }) : t('tokenNever');
+    const info = gone ? t('tokenRevokePending') : `${t('tokenCreated', day(k.created))} · ${used}`;
+    return `<div class="token-row${gone ? ' revoked' : ''}"><div><b>${esc(k.name)}</b> <code>${esc(k.hint)}…</code></div>`
+      + (gone
+        ? `<button class="btn" type="button" data-unrevoke="${esc(k.id)}">${esc(t('tokenUndo'))}</button>`
+        : `<button class="btn danger" type="button" data-revoke="${esc(k.id)}">${esc(t('tokenRevoke'))}</button>`)
+      + `<div class="muted">${esc(info)}</div></div>`;
+  }).join('');
+  html += p.create.map((name, i) => `<div class="token-row pending"><div><b>${esc(name)}</b></div>`
+    + `<button class="btn" type="button" data-uncreate="${i}">${esc(t('tokenUndo'))}</button>`
+    + `<div class="muted">${esc(t('tokenCreatePending'))}</div></div>`).join('');
+  el.innerHTML = html || `<div class="token-empty">${esc(t('tokenNone'))}</div>`;
+}
+
+async function loadTokens() {
+  try {
+    tokens.list = await api('/api/tokens');
+    tokens.error = null;
+  } catch (e) {
+    tokens.list = null;
+    tokens.error = e.data?.code === 'read_only' ? '' : e.message;
+  }
+  renderTokens();
+}
+
+// Save: revocations first, then creations; the new tokens are shown once.
+// What failed stays pending, so Save can try it again.
+async function saveTokens() {
+  const p = pendingTokens();
+  const errors = [];
+  const left = { create: [], revoke: [] };
+  for (const id of p.revoke) {
+    try {
+      await api('/api/tokens/' + encodeURIComponent(id), { method: 'DELETE' });
+    } catch (e) {
+      if (!/not found/.test(e.message)) { errors.push(e.message); left.revoke.push(id); }
+    }
+  }
+  const fresh = [];
+  for (const name of p.create) {
+    try {
+      const res = await api('/api/tokens', { method: 'POST', body: { name } });
+      fresh.push({ name: res.name, token: res.token });
+    } catch (e) { errors.push(e.message); left.create.push(name); }
+  }
+  tokens.fresh = fresh;
+  if (errors.length) settings.pending.tokens = left;
+  else delete settings.pending.tokens;
+  await loadTokens();
+  if (errors.length) throw new Error(errors.join(' · '));
+}
+
+$('#tokenForm').addEventListener('submit', (ev) => {
+  ev.preventDefault();
+  const name = $('#tokenName').value.trim();
+  if (!name) { $('#tokenName').focus(); return; }
+  const p = pendingTokens();
+  pendTokens({ create: [...p.create, name], revoke: p.revoke });
+  $('#tokenName').value = '';
+});
+
+$('#tokenNew').addEventListener('click', (ev) => {
+  const b = ev.target.closest('[data-copy-token]');
+  if (b) copy(tokens.fresh[b.dataset.copyToken].token);
+});
+
+$('#tokenExampleCopy').addEventListener('click', () => {
+  const fresh = tokens.fresh;
+  copy(curlExample(fresh.length ? fresh[fresh.length - 1].token : 'lst_…').text);
+});
+
+$('#tokenList').addEventListener('click', (ev) => {
+  const p = pendingTokens();
+  const b = ev.target.closest('[data-revoke], [data-unrevoke], [data-uncreate]');
+  if (!b) return;
+  if (b.dataset.revoke) pendTokens({ create: p.create, revoke: [...p.revoke, b.dataset.revoke] });
+  else if (b.dataset.unrevoke) pendTokens({ create: p.create, revoke: p.revoke.filter((id) => id !== b.dataset.unrevoke) });
+  else pendTokens({ create: p.create.filter((_, i) => i !== Number(b.dataset.uncreate)), revoke: p.revoke });
+});
+
 /* ================= Tag management ================= */
 
 const PALETTE = ['#6366f1', '#0ea5e9', '#14b8a6', '#a855f7', '#ec4899', '#eab308', '#64748b', '#f97316'];
@@ -2820,6 +2993,8 @@ $('#settingsBtn').addEventListener('click', () => {
   loadDocker();
   renderDbStats();
   loadDbStats();
+  tokens.fresh = [];
+  loadTokens();
   showSettingsTab(store.get('settingsTab', 'locale'));
   $('#settingsDlg').showModal();
 });
@@ -2869,6 +3044,7 @@ const SOURCE_SAVE = {
   selflog: async () => { selfState = await api('/api/selflog', { method: 'PUT', body: { enabled: selfState.enabled, min_level: selfState.min_level || 'warning' } }); },
   docker: async () => { dockerState = await api('/api/docker', { method: 'PUT', body: settings.pending.docker }); },
   files: saveFiles,
+  tokens: saveTokens,
 };
 
 async function saveSettings() {
@@ -2947,6 +3123,7 @@ function cancelSettings() {
   settings.orig = {};
   fileDrafts = [];
   reloadSources();
+  renderTokens();
   renderSaveState();
 }
 
@@ -3021,7 +3198,7 @@ fetch('/auth/me').then((res) => (res.ok ? res.json() : null)).then((me) => {
   }
   if (me.role === 'viewer') {
     document.body.classList.add('read-only');
-    for (const p of document.querySelectorAll('[data-panel="filters"], [data-panel="sources"], [data-panel="data"]')) {
+    for (const p of document.querySelectorAll('[data-panel="filters"], [data-panel="sources"], [data-panel="data"], [data-panel="api"]')) {
       for (const s of p.querySelectorAll('.set-section')) s.inert = true;
       const note = document.createElement('p');
       note.className = 'ro-note';

web/index.html

@@ -140,6 +140,7 @@
 
   <div id="table" class="table" hidden>
     <div id="listHead" class="list-head" role="presentation">
+      <span data-col="src"><span class="lbl" data-i18n="colSrc">Source</span></span>
       <span data-col="rcv"><span class="lbl" data-i18n="colRcv">Received</span><i class="grip" data-i18n-title="colGrip"></i></span>
       <span data-col="mt"><span class="lbl" data-i18n="colMt">Message time</span><i class="grip" data-i18n-title="colGrip"></i></span>
       <span data-col="sev"><span class="lbl" data-i18n="colSev">Severity</span><i class="grip" data-i18n-title="colGrip"></i></span>
@@ -196,6 +197,10 @@
             <svg viewBox="0 0 24 24"><ellipse cx="12" cy="5" rx="8" ry="3"/><path d="M4 5v14c0 1.7 3.6 3 8 3s8-1.3 8-3V5M4 12c0 1.7 3.6 3 8 3s8-1.3 8-3"/></svg>
             <span data-i18n="tabData">Database</span>
           </button>
+          <button type="button" role="tab" data-tab="api">
+            <svg viewBox="0 0 24 24"><circle cx="7.5" cy="15.5" r="4.5"/><path d="m10.7 12.3 9.3-9.3M16 7l3 3M18 5l2 2"/></svg>
+            <span data-i18n="tabApi">API</span>
+          </button>
         </nav>
         <div id="setActions" class="set-actions">
           <button id="setSave" class="btn save" type="button" disabled>
@@ -445,6 +450,31 @@
               </div>
             </section>
           </div>
+
+          <!-- API tokens -->
+          <div class="set-panel wide" role="tabpanel" data-panel="api" hidden>
+            <section class="set-section">
+              <div class="sec-head">
+                <h3 data-i18n="tokensTitle">API tokens</h3>
+                <span id="tokenCount" class="muted small"></span>
+              </div>
+              <p class="muted small" data-i18n="tokensHelp"></p>
+              <div id="tokenList" class="token-list"></div>
+              <form id="tokenForm" class="token-form">
+                <input id="tokenName" type="text" class="dk-filter" maxlength="80" autocomplete="off" spellcheck="false" data-i18n-ph="tokenNamePh" data-i18n-aria="tokenNamePh">
+                <button id="tokenCreate" class="btn" type="submit" data-i18n="tokenCreate">Add a token</button>
+              </form>
+              <div id="tokenNew" class="token-new" hidden></div>
+            </section>
+            <section class="set-section">
+              <h3 data-i18n="tokensUse">Usage</h3>
+              <p class="muted small" data-i18n="tokensUseHelp"></p>
+              <div class="token-example-box">
+                <pre id="tokenExample" class="token-example"></pre>
+                <button id="tokenExampleCopy" class="btn" type="button" data-i18n="tokenExampleCopy">Copy the command</button>
+              </div>
+            </section>
+          </div>
         </div>
       </div>
     </div>

web/style.css

@@ -22,6 +22,10 @@
   --hit: rgba(99, 102, 241, .18);
   --shadow: 0 10px 30px rgba(15, 23, 42, .12);
   --chip-l: 38%;            /* lightness of project-colored text (labels, Docker apps) */
+  --src-syslog: #64748b;    /* source badges: syslog received over the network */
+  --src-docker: #1d6fd6;    /* Docker containers */
+  --src-kernel: #b4540a;    /* kernel ring buffer (dmesg) */
+  --src-host: #b0407a;      /* /var/log files of the host system */
   --src-audit: #7c4dbd;     /* audit events (auditd) */
   --src-ls: #1f9aa3;        /* LogStream internal logs */
   --src-file: #5f8a1f;      /* log files (Settings > Sources > Log files) */
@@ -74,6 +78,10 @@
   --hit: rgba(129, 140, 248, .28);
   --shadow: 0 10px 30px rgba(0, 0, 0, .5);
   --chip-l: 70%;
+  --src-syslog: #94a3b8;
+  --src-docker: #5aa2f5;
+  --src-kernel: #e7a25a;
+  --src-host: #e58bbb;
   --src-audit: #a983e6;
   --src-ls: #3cc3cc;
   --src-file: #9ccc54;
@@ -110,10 +118,13 @@
     --hit: rgba(129, 140, 248, .28);
     --shadow: 0 10px 30px rgba(0, 0, 0, .5);
   --chip-l: 70%;
+    --src-syslog: #94a3b8;
+    --src-docker: #5aa2f5;
+    --src-kernel: #e7a25a;
+    --src-host: #e58bbb;
     --src-audit: #a983e6;
     --src-ls: #3cc3cc;
     --src-file: #9ccc54;
-  --src-file: #9ccc54;
     --src-tint: 9%;
     --sev-crit: #fb7185;
     --sev-err: #f87171;
@@ -344,12 +355,12 @@ body.busy .progress::after {
 .list { margin: 6px 20px 0; background: var(--panel); border: 1px solid var(--border); border-radius: var(--radius); overflow: hidden; }
 .list:empty { display: none; }
 
-/* Columns: received, message time, severity, host, app, tag codes, message. Widths come from
+/* Columns: source, received, message time, severity, host, app, tag codes, message. Widths come from
    --col-* (set on #table when a column has been resized, see app.js), shared by
    the header and every row through subgrid so that the columns line up. */
 .table {
   display: grid;
-  grid-template-columns: var(--col-rcv, max-content) var(--col-mt, max-content) var(--col-sev, 4.6rem)
+  grid-template-columns: minmax(3.9rem, max-content) var(--col-rcv, max-content) var(--col-mt, max-content) var(--col-sev, 4.6rem)
     var(--col-host, minmax(5rem, 9rem)) var(--col-app, minmax(4rem, 8rem)) var(--codes-w) minmax(0, 1fr);
   column-gap: 12px;
   margin: 6px 20px 0; background: var(--panel); border: 1px solid var(--border); border-radius: var(--radius);
@@ -379,20 +390,19 @@ body.col-resizing, body.col-resizing * { cursor: col-resize !important; user-sel
 
 .row {
   display: grid;
-  grid-template-columns: max-content max-content 4.6rem minmax(5rem, 9rem) minmax(4rem, 8rem) var(--codes-w) 1fr;
+  grid-template-columns: max-content max-content max-content 4.6rem minmax(5rem, 9rem) minmax(4rem, 8rem) var(--codes-w) 1fr;
   gap: 0 12px; align-items: baseline;
-  padding: 5px 14px 5px 11px;
+  padding: 3px 14px 3px 11px;
   border-left: 3px solid transparent;
   border-bottom: 1px solid var(--border);
-  font-family: var(--log-font, var(--mono)); font-size: var(--log-size);
+  font-family: var(--log-font, var(--mono)); font-size: var(--log-size); line-height: 1.35;
   font-variant-ligatures: none; /* keep -> != etc. as typed */
   cursor: pointer;
 }
 .row:last-child { border-bottom: 0; }
 /* Settings > Interface > Density: compact fits about 50% more lines on screen */
-:root[data-density="compact"] .row { padding-top: 1px; padding-bottom: 1px; line-height: 1.25; }
-:root[data-density="compact"] .row .sev { padding-top: 0; padding-bottom: 0; line-height: 1.3; }
-:root[data-density="compact"] .row .codes b { padding-top: 0; padding-bottom: 0; line-height: 1.2; }
+:root[data-density="compact"] .row { padding-top: 1px; padding-bottom: 1px; line-height: 1.2; }
+:root[data-density="compact"] .row :is(.src-badge, .sev, .codes b) { line-height: 1.15; }
 .row:hover { background: var(--row-hover); }
 .row:focus-visible { outline: 2px solid var(--accent); outline-offset: -2px; }
 .row.new { animation: flash 1.2s ease-out; }
@@ -410,7 +420,7 @@ body.col-resizing, body.col-resizing * { cursor: col-resize !important; user-sel
 .sev {
   justify-self: start; max-width: 100%; overflow: hidden; text-overflow: ellipsis;
   font-size: 10.5px; font-weight: 700; text-transform: uppercase; letter-spacing: .03em;
-  padding: 1px 6px; border-radius: 5px;
+  padding: 0 6px; border-radius: 5px; line-height: 1.3;
   color: var(--sev); background: color-mix(in srgb, var(--sev) 14%, transparent);
 }
 .sev-crit    { --sev: var(--tag-error); --sev-text: var(--tag-error-text); }
@@ -424,16 +434,21 @@ body.col-resizing, body.col-resizing * { cursor: col-resize !important; user-sel
 .row.sev-crit, .row.sev-err, .row.sev-warning { border-left-color: var(--sev); }
 .row.sev-crit, .row.sev-err { background: color-mix(in srgb, var(--sev) 7%, transparent); }
 .row .host[data-act], .row .app[data-act] { cursor: pointer; }
-/* Audit events and LogStream internal logs: colored edge (over the severity
-   one), light tint, badge before the app name. */
+/* Source badge at the start of every row, fixed width, one color per source.
+   Audit events, log files and LogStream internal logs also get a colored edge
+   (over the severity one) and a light tint. */
+.row.src-syslog { --src: var(--src-syslog); }
+.row.src-docker { --src: var(--src-docker); }
+.row.src-kernel { --src: var(--src-kernel); }
+.row.src-host { --src: var(--src-host); }
 .row.src-audit { --src: var(--src-audit); }
 .row.src-logstream { --src: var(--src-ls); }
 .row.src-file { --src: var(--src-file); }
 .row.src-audit, .row.src-logstream, .row.src-file { border-left-color: var(--src); background: color-mix(in srgb, var(--src) var(--src-tint), transparent); }
 .row.src-audit:hover, .row.src-logstream:hover, .row.src-file:hover { background: color-mix(in srgb, var(--src) 13%, var(--row-hover)); }
 .src-badge {
-  display: inline-block; margin-right: 5px; padding: 0 4px; border-radius: 4px; vertical-align: 1px;
-  font: 700 9.5px/1.45 var(--sans); letter-spacing: .04em;
+  justify-self: start; box-sizing: content-box; width: 6ch; padding: 0 4px; border-radius: 4px; text-align: center;
+  font-family: var(--log-font, var(--mono)); font-size: 10px; font-weight: 700; letter-spacing: .02em; line-height: 1.3;
   color: var(--src); background: color-mix(in srgb, var(--src) 16%, transparent);
   box-shadow: inset 0 0 0 1px color-mix(in srgb, var(--src) 40%, transparent);
 }
@@ -448,10 +463,10 @@ body.col-resizing, body.col-resizing * { cursor: col-resize !important; user-sel
 .row .codes { white-space: nowrap; overflow: hidden; }
 .row .codes > span { display: inline-flex; gap: 3px; vertical-align: 1px; }
 .row .codes b, .tag-code {
-  min-width: 2.2ch; padding: 1px 4px; border-radius: 5px; text-align: center;
+  min-width: 2.2ch; padding: 0 4px; border-radius: 5px; text-align: center;
   /* same font as the severity badges (the log font) */
   font-family: var(--log-font, var(--mono)); font-size: 10.5px; font-weight: 700; letter-spacing: .03em;
-  line-height: 1.35; font-variant-numeric: tabular-nums;
+  line-height: 1.3; font-variant-numeric: tabular-nums;
   background: var(--code-bg); color: #0b0f17;
 }
 .row .codes b.more { background: none; box-shadow: inset 0 0 0 1px var(--code-bg); color: var(--muted); }
@@ -589,10 +604,11 @@ dialog.settings .dlg > header { padding: 14px 16px 14px 22px; border-bottom: 1px
 .set-panel .preview-list { margin: 14px 0 10px; pointer-events: none; }
 .preview-list .row {
   cursor: default;
-  grid-template-columns: max-content max-content minmax(0, max-content) minmax(0, 1fr);
-  grid-template-areas: "rcv sev host app" "msg msg msg msg";
+  grid-template-columns: max-content max-content max-content minmax(0, max-content) minmax(0, 1fr);
+  grid-template-areas: "src rcv sev host app" "msg msg msg msg msg";
   gap: 3px 10px;
 }
+.preview-list .row .src-badge { grid-area: src; }
 .preview-list .row time.rcv { grid-area: rcv; }
 .preview-list .row time.mt { display: none; }
 .preview-list .row .sev { grid-area: sev; }
@@ -827,10 +843,11 @@ input.switch:focus-visible { outline: 2px solid var(--accent); outline-offset: 2
   .table .list { display: block; margin: 0; }
   .list-head { display: none; }
   .row, .table .row {
-    grid-template-columns: auto auto 1fr auto;
-    grid-template-areas: "rcv sev host codes" "msg msg msg msg";
+    grid-template-columns: auto auto auto 1fr auto;
+    grid-template-areas: "src rcv sev host codes" "msg msg msg msg msg";
     gap: 3px 8px; padding: 8px 12px 8px 10px;
   }
+  .row .src-badge { grid-area: src; }
   .row time.rcv { grid-area: rcv; }
   .row time.mt { display: none; } /* still shown in the details panel */
   .row .sev { grid-area: sev; }
@@ -895,3 +912,32 @@ body.login-page {
 .login-card input:focus { outline: 0; border-color: var(--accent); box-shadow: 0 0 0 3px var(--accent-soft); }
 .login-card .btn.primary { height: 38px; justify-content: center; margin-top: 6px; font-size: 14px; }
 .login-err { margin: 0; color: var(--sev-err); font-size: 13px; }
+
+/* API tokens (Settings > API) */
+.token-list { display: flex; flex-direction: column; gap: 4px; margin: 12px 0 10px; }
+.token-row {
+  display: grid; grid-template-columns: minmax(0, 1fr) auto; gap: 2px 12px; align-items: center;
+  padding: 8px 10px; border-radius: 8px; background: var(--panel-2); font-size: 12.5px;
+}
+.token-row b { font-weight: 600; overflow-wrap: anywhere; }
+.token-row code { font-family: var(--mono); font-size: 11.5px; color: var(--muted); }
+.token-row .muted { grid-column: 1; font-size: 11.5px; }
+.token-row .btn { grid-row: 1 / span 2; grid-column: 2; height: 28px; font-size: 12px; }
+.token-empty { color: var(--muted); font-size: 12.5px; }
+.token-form { display: flex; gap: 10px; align-items: center; }
+.token-form .btn { height: 28px; font-size: 12px; }
+.token-new { margin-top: 12px; padding: 10px 12px; border-radius: 8px; border: 1px solid var(--accent); background: var(--accent-soft); }
+.token-secret { display: flex; gap: 10px; align-items: center; margin-top: 8px; }
+.token-label { font-size: 12.5px; font-weight: 600; white-space: nowrap; }
+.token-row.revoked b, .token-row.revoked code { text-decoration: line-through; opacity: .6; }
+.token-row.pending { border: 1px dashed var(--accent); }
+.token-example-box { display: flex; gap: 10px; align-items: flex-start; margin-top: 8px; }
+.token-example-box .token-example { flex: 1; min-width: 0; margin: 0; }
+.token-example-box .btn { height: 28px; font-size: 12px; }
+.token-example mark { background: var(--accent-soft); color: var(--accent); border-radius: 3px; padding: 0 2px; font-weight: 600; }
+.token-secret code { flex: 1; min-width: 0; font: 12px var(--mono); overflow-wrap: anywhere; user-select: all; }
+.token-secret .btn { height: 28px; font-size: 12px; }
+.token-example {
+  margin: 8px 0 0; padding: 8px 12px; border-radius: 8px; background: var(--panel-2);
+  font: 12px var(--mono); white-space: pre-wrap; overflow-wrap: anywhere;
+}