Compare commits

..
Author SHA1 Message Date
claude BotandClaude Opus 5.5 78e363652b Header: lowercase oidc badge
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 17:27:30 +02:00
claude BotandClaude Opus 5.5 a57531c95f Header: show the logged-in user with an OIDC / local badge, or (no auth)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 17:25:22 +02:00
claude BotandClaude Opus 5.5 7f053a0fe9 Settings: clearer tab names
EN: Localization, Patterns, Log sources, Appearance, Database.
FR: Localisation et langue, Motifs de recherche, Sources de logs,
Affichage, Base de données. Internal tab ids are unchanged, so the
remembered tab still opens. READMEs and preset docs follow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 17:16:19 +02:00
claude BotandClaude Opus 5.5 5b28bff48b Settings: size the side menu to its longest label
"Motif de recherche" overflowed the fixed 176px menu column. The column
now grows to its longest label (176px minimum), and on narrow screens the
tab labels wrap on spaces instead of being cut.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 17:08:05 +02:00
claude BotandClaude Opus 5.5 c2c3926135 Rename the color tags "Filters" tab and column to "Pattern" (FR: "Motif de recherche")
Only the color highlight rules are renamed: the Settings tab, the codes
column of the log list and the docs. Real filters (source, severity,
range, search) keep their name. The internal "filters" tab id is kept so
the saved settings tab still opens.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 17:01:35 +02:00
cedricandClaude Opus 5.5 076c01aac4 Linux audit (auditd) source and LogStream internal logs
Audit source (source_type=audit), off until enabled in Settings > Sources:
- follows /var/log/audit/audit.log (path configurable, AUDIT_LOG), with
  rotation (end of audit.log.1 read first) and an offset checkpoint in
  /data/audit-state.json; last AUDIT_BACKFILL (1h) read on first enable
- groups the records of an event by msg=audit(time:serial), closed on EOE,
  after 2 s without a record, or once the log has moved 2 s past it
- fields audit_type, audit_serial, auid, user (enriched name, host
  /etc/passwd or uid), exe, res, key, audit_raw; app = audit_type, host =
  node= / AUDIT_HOST / /host/etc/hostname; summary "who: what [key] →
  result"; warning when res=failed / success=no, info otherwise
- hexadecimal values (proctitle, cmd, exec arguments…) decoded

Internal logs (source_type=logstream, host=logstream, app=component):
- logEvent() for VictoriaLogs unreachable / back, lost batches, spooled
  batches refused, syslog listener errors, Docker API and container
  streams lost, failing sources, start and stop
- on by default from warning; switch and minimum level in Settings >
  Sources (/data/selflog.json)
- bounded queue, never waits for the Store, same event at most once per
  30 s with a count of the repeats: a failing Store cannot loop

UI: violet (audit) and turquoise (LogStream) left edge and tint, AUDIT /
LS badge before the app, raw audit record under the summary (folded in
compact density until the row is opened), Audit and LogStream in the
source filter, FR/EN translations. docker-compose: optional
/var/log/audit, /etc/passwd and /etc/hostname mounts. README (EN/FR):
host prerequisites (log_group, ACL for UID 10001).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:01:58 +02:00
claude Bot f3373e1f66 Merge pull request 'Source « Messages du noyau (dmesg) »' (#20) from feat/dmesg into main 2026-10-04 09:59:37 +02:00
cedricandClaude Opus 5.5 0beaa24f82 Kernel messages (dmesg) source read from /dev/kmsg
New source in Settings > Sources, off by default: reads the kernel ring
buffer of the host through /dev/kmsg, dates records from the boot time and
resumes after the last sequence number of the same boot (no duplicates on
restart). Missing device or denied access is shown in the UI, retried every
10 s. docker-compose.yml and .env.example document the device and the
kernel.dmesg_restrict / CAP_SYSLOG options.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 09:52:17 +02:00
cedric b626b3b8db Merge pull request 'Modes Flux et Période avec dates de début et de fin' (#19) from feat/mode-periode into main
Reviewed-on: #19
2026-10-03 22:45:55 +02:00
claude BotandClaude Opus 5.5 aff7cf8839 Stream and Time range display modes with start/end fields
A segmented control at the start of the filter bar switches between
Stream (sliding duration, live view) and Time range (start and end
dates in the chosen time zone, previous/next and zoom-out buttons).
Timeline clicks and drags switch to Time range mode; the mode and
range are kept across reloads.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 20:37:05 +02:00
23 changed files with 3149 additions and 109 deletions

No files matched your search

+14
View File
@@ -53,3 +53,17 @@ DOCKER_BACKFILL=1h
HOST_LOGS_GID=4 HOST_LOGS_GID=4
# History read when the source is turned on (e.g. 30m, 1h, 24h; 0 = only new entries) # History read when the source is turned on (e.g. 30m, 1h, 24h; 0 = only new entries)
HOST_LOGS_BACKFILL=1h HOST_LOGS_BACKFILL=1h
# Kernel messages of the host (dmesg, enable them in Settings > Sources). The container needs
# /dev/kmsg: uncomment "devices" in docker-compose.yml. When the host has
# kernel.dmesg_restrict=1 (default on Debian/Ubuntu), either run on the host
# sysctl -w kernel.dmesg_restrict=0 (persist it in /etc/sysctl.d/)
# or run the container as root with the SYSLOG capability ("user" and "cap_add" in docker-compose.yml).
# History read when the source is turned on (e.g. 30m, 1h, 24h; 0 = only new messages)
DMESG_BACKFILL=1h
# Linux audit log of the host (auditd, enable it in Settings > Sources). Uncomment the
# /var/log/audit volume in docker-compose.yml and let the container read the file:
# log_group = adm in /etc/audit/auditd.conf, and chgrp adm + chmod 750 on /var/log/audit.
# History read when the source is turned on (e.g. 30m, 1h, 24h; 0 = only new events)
AUDIT_BACKFILL=1h
# Host name of the audit events (empty: node= of the records, else /host/etc/hostname, else localhost)
AUDIT_HOST=
+136 -17
View File
@@ -55,7 +55,7 @@ Ouvrez ensuite <http://localhost:8080>.
- **Test manuel** : `logger -n 127.0.0.1 -P 514 -d "hello error"` (util-linux) ou - **Test manuel** : `logger -n 127.0.0.1 -P 514 -d "hello error"` (util-linux) ou
`echo "<14>test ok" | nc -u -w1 127.0.0.1 514`. `echo "<14>test ok" | nc -u -w1 127.0.0.1 514`.
**Paramètres > Sources > Syslog** active ou désactive la réception syslog et choisit les **Paramètres > Sources de logs > Syslog** active ou désactive la réception syslog et choisit les
protocoles (UDP, TCP) sans redémarrage ; le choix est enregistré dans `/data/syslog.json`. Cet protocoles (UDP, TCP) sans redémarrage ; le choix est enregistré dans `/data/syslog.json`. Cet
onglet affiche aussi l'état d'écoute (et l'erreur si le port est déjà utilisé). Le port onglet affiche aussi l'état d'écoute (et l'erreur si le port est déjà utilisé). Le port
lui-même est publié par docker-compose : modifiez `SYSLOG_PORT` dans `.env`, puis lancez lui-même est publié par docker-compose : modifiez `SYSLOG_PORT` dans `.env`, puis lancez
@@ -98,10 +98,20 @@ affiche tous ses champs.
Les en-têtes de colonnes restent visibles pendant le défilement. Faites glisser le bord d'un Les en-têtes de colonnes restent visibles pendant le défilement. Faites glisser le bord d'un
en-tête (réception, heure message, sévérité, hôte, app) pour redimensionner la colonne, et en-tête (réception, heure message, sévérité, hôte, app) pour redimensionner la colonne, et
double-cliquez dessus pour revenir à la largeur automatique ; le message occupe l'espace double-cliquez dessus pour revenir à la largeur automatique ; le message occupe l'espace
restant. Les largeurs sont mémorisées par le navigateur (**Paramètres > Interface > restant. Les largeurs sont mémorisées par le navigateur (**Paramètres > Affichage >
Réinitialiser les colonnes** les rétablit toutes). Sur téléphone, la liste garde sa Réinitialiser les colonnes** les rétablit toutes). Sur téléphone, la liste garde sa
présentation sur deux lignes, sans colonnes. présentation sur deux lignes, sans colonnes.
## Modes Flux et Période
Le premier contrôle de la barre de filtres bascule entre deux modes d'affichage :
- **Flux** : les derniers logs sur une durée glissante (5 min à 30 jours, ou tout), avec le direct.
- **Période** : les logs entre une date de début et une date de fin, saisies dans le fuseau choisi
dans les Paramètres. ◀ et ▶ passent à la période précédente ou suivante de même durée, la loupe
la double autour de son milieu. Le direct se met en pause ; le mode et la période sont conservés
au rechargement.
## Frise ## Frise
La frise au-dessus de la liste montre le volume de logs par intervalle, compté selon l'**heure La frise au-dessus de la liste montre le volume de logs par intervalle, compté selon l'**heure
@@ -109,15 +119,14 @@ de réception** sur l'horloge du serveur (elle correspond donc aux heures affich
lignes). lignes).
- Au survol d'un intervalle : ses bornes, son total et le détail par sévérité. - Au survol d'un intervalle : ses bornes, son total et le détail par sévérité.
- Un clic sur une barre zoome sur cet intervalle ; un glisser sur plusieurs barres zoome sur la - Un clic sur une barre affiche cet intervalle en mode Période ; un glisser sur plusieurs barres
sélection. La plage de temps affiche alors la période zoomée (« × Annuler le zoom » ou le affiche la sélection. « × Revenir au flux » repasse en mode Flux. La liste, les compteurs et
choix d'une autre plage en sort) ; la liste, les compteurs et l'export CSV suivent le zoom, l'export CSV suivent la période.
et le direct se met en pause.
- En direct, le dernier intervalle grandit à l'arrivée des messages, et la frise se recharge à - En direct, le dernier intervalle grandit à l'arrivée des messages, et la frise se recharge à
chaque nouvel intervalle. Rien n'est rafraîchi tant que l'onglet du navigateur est masqué ; chaque nouvel intervalle. Rien n'est rafraîchi tant que l'onglet du navigateur est masqué ;
la frise se met à jour dès qu'il redevient visible. la frise se met à jour dès qu'il redevient visible.
**Paramètres > Interface > Frise** (mémorisé par navigateur) : échelle (linéaire, √ par défaut, **Paramètres > Affichage > Frise** (mémorisé par navigateur) : échelle (linéaire, √ par défaut,
log), hauteur (S/M/L : 40/80/120 px), couleur (empilement par sévérité, intensité comparée à log), hauteur (S/M/L : 40/80/120 px), couleur (empilement par sévérité, intensité comparée à
la médiane de la fenêtre : calme, rafale au-delà de 3×, anomalie au-delà de 10×, ou aucune), la médiane de la fenêtre : calme, rafale au-delà de 3×, anomalie au-delà de 10×, ou aucune),
barres ou aire, division (automatique, environ 100 intervalles, ou fixe : 1 s, 10 s, 1 min, barres ou aire, division (automatique, environ 100 intervalles, ou fixe : 1 s, 10 s, 1 min,
@@ -146,7 +155,7 @@ colorés et exportés comme les messages syslog :
(`"level":"error"`), logfmt (`level=warn`), `[ERROR]`, ou un niveau en majuscules au début de (`"level":"error"`), logfmt (`level=warn`), `[ERROR]`, ou un niveau en majuscules au début de
la ligne (`ERROR`, `WARN`…). Sinon, elle vaut `info`. Les codes de couleur du terminal sont la ligne (`ERROR`, `WARN`…). Sinon, elle vaut `info`. Les codes de couleur du terminal sont
supprimés. supprimés.
- **Paramètres > Sources** affiche une étiquette par conteneur (`project/service`) dans un seul - **Paramètres > Sources de logs** affiche une étiquette par conteneur (`project/service`) dans un seul
champ : les conteneurs suivis en couleur, puis les non suivis en gris ; un clic bascule une champ : les conteneurs suivis en couleur, puis les non suivis en gris ; un clic bascule une
étiquette. La couleur identifie le projet compose, et les noms d'applications Docker de la étiquette. La couleur identifie le projet compose, et les noms d'applications Docker de la
liste utilisent la même couleur. Les conteneurs arrêtés sont masqués par défaut (« Afficher liste utilisent la même couleur. Les conteneurs arrêtés sont masqués par défaut (« Afficher
@@ -172,7 +181,7 @@ moteur (`GET` uniquement).
Logstream peut aussi collecter les logs système de la machine qui héberge la stack, sans rien Logstream peut aussi collecter les logs système de la machine qui héberge la stack, sans rien
configurer sur l'hôte. La source est **désactivée par défaut** : activez-la dans configurer sur l'hôte. La source est **désactivée par défaut** : activez-la dans
**Paramètres > Sources > Logs système de l'hôte** (enregistré dans `/data/hostlogs.json`). **Paramètres > Sources de logs > Logs système de l'hôte** (enregistré dans `/data/hostlogs.json`).
- `docker-compose.yml` monte `/var/log` et `/run/log/journal` en lecture seule sous `/host`. - `docker-compose.yml` monte `/var/log` et `/run/log/journal` en lecture seule sous `/host`.
- Si l'hôte utilise systemd, Logstream lit directement les fichiers du **journal systemd** - Si l'hôte utilise systemd, Logstream lit directement les fichiers du **journal systemd**
@@ -187,13 +196,115 @@ configurer sur l'hôte. La source est **désactivée par défaut** : activez-la
- **Droits** : le conteneur tourne avec un utilisateur sans privilège et reçoit le groupe `adm` - **Droits** : le conteneur tourne avec un utilisateur sans privilège et reçoit le groupe `adm`
(gid 4), qui peut lire le journal et `/var/log` sur Debian et Ubuntu. Sur d'autres systèmes, (gid 4), qui peut lire le journal et `/var/log` sur Debian et Ubuntu. Sur d'autres systèmes,
réglez `HOST_LOGS_GID` dans `.env` sur le gid de `systemd-journal` réglez `HOST_LOGS_GID` dans `.env` sur le gid de `systemd-journal`
(`getent group systemd-journal | cut -d: -f3`). Paramètres > Sources affiche un message clair (`getent group systemd-journal | cut -d: -f3`). Paramètres > Sources de logs affiche un message clair
si l'accès est refusé. si l'accès est refusé.
- Limites : les champs du journal compressés par journald (messages de plus de 512 octets, - Limites : les champs du journal compressés par journald (messages de plus de 512 octets,
compressés en zstd/lz4/xz) ne peuvent pas être décodés sans bibliothèque supplémentaire ; ils compressés en zstd/lz4/xz) ne peuvent pas être décodés sans bibliothèque supplémentaire ; ils
sont comptés dans les Paramètres et affichés comme « (compressed journal entry) ». Les fichiers sont comptés dans les Paramètres et affichés comme « (compressed journal entry) ». Les fichiers
texte tournés ou compressés (`*.1`, `*.gz`) ne sont pas lus. texte tournés ou compressés (`*.1`, `*.gz`) ne sont pas lus.
## Messages du noyau (dmesg)
Logstream peut collecter le buffer circulaire du noyau de l'hôte (ce qu'affiche `dmesg` :
matériel, pilotes, OOM killer, pare-feu…) en lisant `/dev/kmsg`. La source est **désactivée par
défaut** : activez-la dans **Paramètres > Sources de logs > Messages du noyau (dmesg)** (enregistré dans
`/data/dmesg.json`).
- **Périphérique** : le conteneur ne voit pas `/dev/kmsg` par défaut. Décommentez la section
`devices` de `docker-compose.yml` (`/dev/kmsg:/dev/kmsg:r`, en lecture seule), puis
`docker compose up -d`.
- **Droits** : si l'hôte a `kernel.dmesg_restrict=1` (le défaut sur Debian et Ubuntu), seuls
les processus qui ont la capacité `CAP_SYSLOG` peuvent lire le log du noyau, et le conteneur
tourne avec un utilisateur sans privilège. Au choix :
- l'autoriser sur l'hôte : `sysctl -w kernel.dmesg_restrict=0`, conservé au redémarrage avec
`echo kernel.dmesg_restrict=0 | sudo tee /etc/sysctl.d/50-dmesg.conf` (tous les
utilisateurs de l'hôte peuvent alors lancer `dmesg`) ; ce réglage ne peut pas être changé
depuis un conteneur ;
- ou lancer Logstream en root avec la capacité `SYSLOG` : décommentez `user: "0:0"` et
`cap_add: [SYSLOG]` dans `docker-compose.yml`.
Paramètres > Sources de logs indique si le périphérique est absent ou l'accès refusé ; le reste de
Logstream continue de fonctionner et le périphérique est réessayé toutes les 10 secondes.
- **Champs** : **app** vaut `kernel` (ou le nom du programme pour les messages écrits par des
programmes, comme `systemd[1]: …`), la sévérité et la facility viennent de l'enregistrement,
**host** vaut `localhost`, `subsystem` et `device` sont conservés quand le noyau les fournit.
Ces logs ont la source `kernel` (`proto` `kmsg`) : le filtre **Source** permet de les afficher
seuls.
- **Heure** : les enregistrements portent le temps écoulé depuis le démarrage ; Logstream le
convertit en date grâce à l'heure de démarrage (`/proc/uptime`). Comme avec `dmesg -T`,
l'heure peut être trop tôt après une mise en veille de l'hôte.
- À l'activation, la dernière heure est lue d'abord (`DMESG_BACKFILL`, `0` = seulement les
nouveaux messages). Le numéro de séquence du dernier message et l'identifiant de démarrage
sont enregistrés dans `/data/dmesg-state.json` : après un redémarrage de Logstream, la lecture
reprend après le dernier message (sans doublon) ; après un redémarrage de l'hôte, le nouveau
buffer est lu depuis `DMESG_BACKFILL`. Les messages écrasés dans le buffer du noyau avant
d'être lus sont comptés dans Paramètres.
## Audit Linux (auditd)
Logstream peut suivre le journal du démon d'audit Linux (`/var/log/audit/audit.log`) :
connexions, `sudo`, fichiers surveillés, appels système visés par vos règles `auditctl`… La
source est **désactivée par défaut** : activez-la dans **Réglages > Sources > Audit Linux
(auditd)**, où le chemin du fichier peut aussi être changé (enregistré dans `/data/audit.json` ;
`AUDIT_LOG` fixe le chemin par défaut).
- **Montage** : décommentez `/var/log/audit:/var/log/audit:ro` dans `docker-compose.yml`, puis
`docker compose up -d`. (`/var/log` est déjà monté sous `/host` pour les logs système de
l'hôte : `/host/var/log/audit/audit.log` fonctionne donc aussi comme chemin, sans nouveau
volume.)
- **Droits** (côté hôte) : auditd écrit son journal en `root`, mode `0600`, dans un répertoire
`0700`, et Logstream tourne avec l'UID `10001`. Le plus fiable est le groupe que le conteneur
reçoit déjà pour les logs de l'hôte (`HOST_LOGS_GID`, `adm` par défaut) :
```bash
sudo sed -i 's/^log_group.*/log_group = adm/' /etc/audit/auditd.conf
sudo chgrp adm /var/log/audit && sudo chmod 750 /var/log/audit
sudo service auditd restart # systemctl refuse de redémarrer auditd sur certaines distributions
```
auditd crée alors chaque fichier, rotations comprises, en mode `0640` avec le groupe `adm`.
Une ACL fonctionne aussi
(`sudo setfacl -m u:10001:rx /var/log/audit && sudo setfacl -m u:10001:r /var/log/audit/audit.log`)
mais doit être réappliquée à chaque nouveau fichier après une rotation. Réglages > Sources
indique si le fichier est absent ou l'accès refusé ; il est réessayé chaque seconde.
- **Événements** : les enregistrements d'un même événement (`SYSCALL`, `EXECVE`, `CWD`, `PATH`,
`PROCTITLE`…) partagent le même `msg=audit(heure:numéro)` et deviennent **un seul** log, dès
qu'aucun enregistrement n'est arrivé depuis 2 secondes. Le message est un résumé — qui, quoi,
résultat — comme `cedric (as root): exec apt install nginx [exec] → success` ou
`bob: PAM:authentication via /usr/sbin/sshd from 203.0.113.9 on ssh → failed` ; les
enregistrements bruts sont gardés dans `audit_raw` et affichés sous le résumé (repliés en
densité compacte tant que la ligne n'est pas ouverte). Les valeurs hexadécimales
(`proctitle`, `cmd`, arguments avec espaces) sont décodées.
- **Champs** : `audit_type` (type du premier enregistrement, aussi utilisé comme **app**),
`audit_serial`, `auid` (identifiant de l'utilisateur connecté), `user` (son nom, ou l'uid s'il
ne peut pas être résolu), `exe`, `res` (`success` / `failed`, d'après `res=` ou `success=`) et
`key` (clé de la règle). La sévérité est `warning` en cas d'échec, `info` sinon ; facility
`authpriv` ; source `audit`. Dans la liste, les lignes d'audit ont un liseré gauche violet,
un fond légèrement teinté et un badge **AUDIT** devant l'app ; le filtre **Source** permet de
les afficher seules.
- **Noms** : `user` vient du format enrichi (`log_format = ENRICHED`, le défaut des versions
récentes d'auditd), sinon du `/etc/passwd` de l'hôte s'il est monté en `/host/etc/passwd`.
**host** est le `node=` des enregistrements (`name_format = hostname` dans `auditd.conf`),
sinon `AUDIT_HOST`, sinon `/host/etc/hostname`, sinon `localhost`.
- À l'activation, la dernière heure est lue d'abord (`AUDIT_BACKFILL`, `0` = seulement les
nouveaux événements). L'inode et une position avant laquelle tout événement a été envoyé sont
enregistrés dans `/data/audit-state.json` : après un redémarrage, la lecture reprend là ;
après une rotation par auditd, la fin de `audit.log.1` est lue avant le nouveau fichier.
## Logs internes de LogStream
Logstream enregistre ses propres événements comme des logs, avec l'**hôte** `logstream`,
l'**app** du composant (`store`, `syslog`, `docker`, `host`, `dmesg`, `audit`, `main`) et la
source `logstream` : VictoriaLogs injoignable puis de retour, lots perdus, lots refusés depuis
le tampon disque, erreurs du listener syslog, API Docker ou flux d'un conteneur décrochés,
sources qui ne peuvent pas lire, démarrage et arrêt. Ils sont actifs par défaut à partir du
niveau **warning** ; **Réglages > Sources > Logs internes de LogStream** permet de les couper
ou de changer le niveau minimal (enregistré dans `/data/selflog.json`). Un même événement
(composant et message) est enregistré au plus une fois toutes les 30 secondes, avec le nombre
de répétitions écartées ; les événements passent par une file bornée et n'attendent jamais le
stockage : un VictoriaLogs injoignable ne peut ni les accumuler ni les faire boucler. Tout
continue d'aller dans `docker compose logs logstream` comme avant. Dans la liste, ces lignes
ont un liseré gauche turquoise et un badge **LS** devant l'app ; le filtre **Source** permet de
les afficher seules.
## Export CSV ## Export CSV
Le bouton **Exporter** (à côté du nombre de logs) télécharge tous les logs stockés qui Le bouton **Exporter** (à côté du nombre de logs) télécharge tous les logs stockés qui
@@ -219,13 +330,13 @@ En ligne de commande : `curl -o logs.csv 'localhost:8080/api/export.csv?q=error&
L'icône en forme d'engrenage ouvre les paramètres, organisés en onglets. Tout, sauf les tags de L'icône en forme d'engrenage ouvre les paramètres, organisés en onglets. Tout, sauf les tags de
couleur, est mémorisé par navigateur. couleur, est mémorisé par navigateur.
- **Localisation** - **Localisation et langue**
- *Langue* : anglais ou français. - *Langue* : anglais ou français.
- *Date et heure* : fuseau horaire (celui du navigateur, UTC ou environ 80 fuseaux courants) - *Date et heure* : fuseau horaire (celui du navigateur, UTC ou environ 80 fuseaux courants)
et format d'affichage de l'heure de réception : `DD/MM/YYYY HH:MM:SS` (par défaut, et format d'affichage de l'heure de réception : `DD/MM/YYYY HH:MM:SS` (par défaut,
l'affichage français habituel), avec millisecondes, `YYYY-MM-DD`, horloge sur 12 heures, l'affichage français habituel), avec millisecondes, `YYYY-MM-DD`, horloge sur 12 heures,
ISO 8601 ou epoch Unix. Le fuseau horaire s'applique à toutes les dates affichées. ISO 8601 ou epoch Unix. Le fuseau horaire s'applique à toutes les dates affichées.
- **Filtres** : tags de couleur. Chaque tag a un mot-clé, une couleur de fond (le texte passe - **Motifs de recherche** : tags de couleur. Chaque tag a un mot-clé, une couleur de fond (le texte passe
automatiquement en noir ou en blanc pour rester lisible) et des options : mot entier, respect automatiquement en noir ou en blanc pour rester lisible) et des options : mot entier, respect
de la casse, expression régulière, actif. Les tags sont stockés sur le serveur dans de la casse, expression régulière, actif. Les tags sont stockés sur le serveur dans
`/data/tags.json` (volume `logstream-data`) : ils sont donc partagés par tous les navigateurs. `/data/tags.json` (volume `logstream-data`) : ils sont donc partagés par tous les navigateurs.
@@ -244,10 +355,10 @@ couleur, est mémorisé par navigateur.
les préréglages s'en servent pour colorer le code de statut ou la méthode, pas le texte autour. les préréglages s'en servent pour colorer le code de statut ou la méthode, pas le texte autour.
Chaque tag reçoit un code à deux chiffres (`01`, `02`…) attribué par le serveur : il reste Chaque tag reçoit un code à deux chiffres (`01`, `02`…) attribué par le serveur : il reste
attaché au tag jusqu'à sa suppression (les tags créés par les versions précédentes en reçoivent attaché au tag jusqu'à sa suppression (les tags créés par les versions précédentes en reçoivent
un aussi). La colonne *Filtres* de la liste affiche, en badges gris, les codes des tags actifs un aussi). La colonne *Motif de recherche* de la liste affiche, en badges gris, les codes des tags actifs
trouvés dans chaque message ; elle a la place pour 3, au-delà elle en affiche 2 et `+N`, et trouvés dans chaque message ; elle a la place pour 3, au-delà elle en affiche 2 et `+N`, et
l'infobulle les liste tous. l'infobulle les liste tous.
- **Interface** - **Affichage**
- *Thème* : Système (suit la préférence de l'ordinateur ou du téléphone), Clair ou Sombre. Le - *Thème* : Système (suit la préférence de l'ordinateur ou du téléphone), Clair ou Sombre. Le
bouton soleil/lune de l'en-tête bascule entre clair et sombre. bouton soleil/lune de l'en-tête bascule entre clair et sombre.
- *Affichage des logs* : taille du texte (très petite, petite, moyenne, grande), densité - *Affichage des logs* : taille du texte (très petite, petite, moyenne, grande), densité
@@ -261,7 +372,7 @@ couleur, est mémorisé par navigateur.
européen de polices respectueux de la vie privée ; sans accès à internet, la police du européen de polices respectueux de la vie privée ; sans accès à internet, la police du
système est utilisée. Les ligatures sont désactivées pour que `->` ou `!=` s'affichent tels système est utilisée. Les ligatures sont désactivées pour que `->` ou `!=` s'affichent tels
quels. Le réglage le plus dense est Très petite + Compacte + Inconsolata Condensed. quels. Le réglage le plus dense est Très petite + Compacte + Inconsolata Condensed.
- **Données** : chiffres de la base lus dans VictoriaLogs (actualisés au plus toutes les 30 s) : - **Base de données** : chiffres de la base lus dans VictoriaLogs (actualisés au plus toutes les 30 s) :
lignes stockées, taille sur disque (index compris), taille brute et taux de compression, lignes stockées, taille sur disque (index compris), taille brute et taux de compression,
période couverte avec la rétention, lignes des dernières 24 h et de la dernière heure, hôtes période couverte avec la rétention, lignes des dernières 24 h et de la dernière heure, hôtes
et applications distincts, espace disque libre. Les tailles sont en Ko/Mo/Go (KB/MB/GB en et applications distincts, espace disque libre. Les tailles sont en Ko/Mo/Go (KB/MB/GB en
@@ -387,6 +498,11 @@ résolutions.
| `HOST_LOGS_GID` | `4` (adm) dans compose | groupe donné au conteneur pour lire les logs de l'hôte | | `HOST_LOGS_GID` | `4` (adm) dans compose | groupe donné au conteneur pour lire les logs de l'hôte |
| `HOST_LOGS_BACKFILL` | `1h` | historique lu à l'activation de la source « logs système de l'hôte » | | `HOST_LOGS_BACKFILL` | `1h` | historique lu à l'activation de la source « logs système de l'hôte » |
| `HOST_LOGS_ROOT` | `/host` | emplacement de montage des répertoires de l'hôte | | `HOST_LOGS_ROOT` | `/host` | emplacement de montage des répertoires de l'hôte |
| `DMESG_BACKFILL` | `1h` | historique lu à l'activation de la source « messages du noyau » |
| `DMESG_DEVICE` | `/dev/kmsg` | périphérique du log noyau lu par la source « messages du noyau » |
| `AUDIT_LOG` | `/var/log/audit/audit.log` | chemin par défaut du journal d'audit Linux (modifiable dans Réglages > Sources) |
| `AUDIT_BACKFILL` | `1h` | historique lu à l'activation de la source « audit Linux » |
| `AUDIT_HOST` | vide | nom d'hôte des événements d'audit quand les enregistrements n'ont pas de `node=` |
| `TZ` | `Europe/Paris` | fuseau horaire des horodatages RFC 3164 (qui n'en portent pas) | | `TZ` | `Europe/Paris` | fuseau horaire des horodatages RFC 3164 (qui n'en portent pas) |
| `BATCH_SIZE`, `FLUSH_MS`, `QUEUE_SIZE` | `1000`, `1000`, `100000` | réglage de l'ingestion | | `BATCH_SIZE`, `FLUSH_MS`, `QUEUE_SIZE` | `1000`, `1000`, `100000` | réglage de l'ingestion |
| `SPOOL_MAX_MB` | `1024` | taille du tampon disque des lots refusés par VictoriaLogs (`0` = pas de tampon : 15 s de tentatives, puis perte) | | `SPOOL_MAX_MB` | `1024` | taille du tampon disque des lots refusés par VictoriaLogs (`0` = pas de tampon : 15 s de tentatives, puis perte) |
@@ -427,7 +543,7 @@ Pour mettre à jour l'une d'elles :
VictoriaLogs conserve son format de stockage entre versions mineures ; lisez le changelog VictoriaLogs conserve son format de stockage entre versions mineures ; lisez le changelog
avant un changement de version majeure. avant un changement de version majeure.
2. Modifiez la version dans le fichier indiqué ci-dessus, puis lancez `docker compose up -d --build`. 2. Modifiez la version dans le fichier indiqué ci-dessus, puis lancez `docker compose up -d --build`.
3. Vérifiez la barre du bas (reçus / stockés / perdus) et **Paramètres > Sources**. Pour revenir 3. Vérifiez la barre du bas (reçus / stockés / perdus) et **Paramètres > Sources de logs**. Pour revenir
en arrière, remettez la version précédente et lancez la même commande. en arrière, remettez la version précédente et lancez la même commande.
## Organisation du code ## Organisation du code
@@ -445,8 +561,11 @@ Pour mettre à jour l'une d'elles :
| `rdns.go` | résolutions DNS inverses avec cache | | `rdns.go` | résolutions DNS inverses avec cache |
| `export.go` | export CSV en flux | | `export.go` | export CSV en flux |
| `docker.go` | logs des conteneurs Docker (API, lecteurs, positions, détection du niveau) | | `docker.go` | logs des conteneurs Docker (API, lecteurs, positions, détection du niveau) |
| `syslogserver.go` | écoutes syslog ouvertes et fermées depuis Paramètres > Sources | | `syslogserver.go` | écoutes syslog ouvertes et fermées depuis Paramètres > Sources de logs |
| `hostlogs.go`, `journal.go` | logs système de l'hôte : lecteur du journal systemd (sans `journalctl`) et suivi de `/var/log` | | `hostlogs.go`, `journal.go` | logs système de l'hôte : lecteur du journal systemd (sans `journalctl`) et suivi de `/var/log` |
| `dmesg.go` | messages du noyau : lecture de `/dev/kmsg`, analyse des enregistrements, position par identifiant de démarrage et numéro de séquence |
| `audit.go` | audit Linux : suivi de `audit.log` (rotation, position enregistrée), analyse des enregistrements, regroupement par événement, résumé |
| `selflog.go` | logs internes de LogStream : `logEvent`, niveau minimal, limitation des répétitions, file bornée |
| `tags.go` | stockage des tags de couleur | | `tags.go` | stockage des tags de couleur |
| `presets.go`, `presets.json` | préréglages de tags (`/api/presets`), liste intégrée | | `presets.go`, `presets.json` | préréglages de tags (`/api/presets`), liste intégrée |
| `api.go` | routes HTTP `/api/*` | | `api.go` | routes HTTP `/api/*` |
+125 -15
View File
@@ -50,7 +50,7 @@ Then open <http://localhost:8080>.
- **Manual test**: `logger -n 127.0.0.1 -P 514 -d "hello error"` (util-linux) or - **Manual test**: `logger -n 127.0.0.1 -P 514 -d "hello error"` (util-linux) or
`echo "<14>test ok" | nc -u -w1 127.0.0.1 514`. `echo "<14>test ok" | nc -u -w1 127.0.0.1 514`.
**Settings > Sources > Syslog** turns syslog reception on or off and chooses the protocols **Settings > Log sources > Syslog** turns syslog reception on or off and chooses the protocols
(UDP, TCP) without restarting; the choice is saved in `/data/syslog.json`. It also shows the (UDP, TCP) without restarting; the choice is saved in `/data/syslog.json`. It also shows the
listening state (and the error if the port is already in use). The port itself is published listening state (and the error if the port is already in use). The port itself is published
by docker-compose: change `SYSLOG_PORT` in `.env`, then run `docker compose up -d`. by docker-compose: change `SYSLOG_PORT` in `.env`, then run `docker compose up -d`.
@@ -89,22 +89,32 @@ Shortcuts: `/` focuses the search box, `Esc` clears it. Clicking a row shows all
The column headers stay visible while scrolling. Drag the edge of a header (received, message The column headers stay visible while scrolling. Drag the edge of a header (received, message
time, severity, host, app) to resize the column, double-click it to go back to the automatic time, severity, host, app) to resize the column, double-click it to go back to the automatic
width; the message takes the remaining space. Widths are remembered by the browser width; the message takes the remaining space. Widths are remembered by the browser
(**Settings > Interface > Reset column widths** restores them all). On phones the list keeps its (**Settings > Appearance > Reset column widths** restores them all). On phones the list keeps its
two-line layout without columns. two-line layout without columns.
## Stream and Time range modes
The first control of the filter bar switches between two display modes:
- **Stream**: the latest logs over a sliding duration (5 min to 30 days, or all), with the live
view.
- **Time range**: the logs between a start and an end date, typed in the time zone chosen in
Settings. ◀ and ▶ move to the previous or next range of the same length, the magnifier doubles
it around its middle. The live view pauses; the mode and the range are kept across reloads.
## Timeline ## Timeline
The timeline above the list shows the volume of logs per interval, counted by **reception The timeline above the list shows the volume of logs per interval, counted by **reception
time** on the server clock (so it matches the times shown in the rows). time** on the server clock (so it matches the times shown in the rows).
- Hover an interval: its bounds, total and detail per severity. - Hover an interval: its bounds, total and detail per severity.
- Click a bar to zoom on that interval, or drag across several bars to zoom on the selection. - Click a bar to show that interval in Time range mode, or drag across several bars to show the
The time range then shows the zoomed period ("× Reset zoom" or any other range leaves it); selection; "× Back to stream" returns to Stream mode. The list, the counters and the CSV export
the list, the counters and the CSV export follow the zoom, and the live view pauses. follow the range.
- In live mode the last interval grows as messages arrive, and the timeline reloads at each new - In live mode the last interval grows as messages arrive, and the timeline reloads at each new
interval. Nothing is refreshed while the browser tab is hidden; it catches up when shown again. interval. Nothing is refreshed while the browser tab is hidden; it catches up when shown again.
**Settings > Interface > Timeline** (remembered per browser): scale (linear, √ by default, log), **Settings > Appearance > Timeline** (remembered per browser): scale (linear, √ by default, log),
height (S/M/L: 40/80/120 px), color (stacked by severity, intensity compared with the median of height (S/M/L: 40/80/120 px), color (stacked by severity, intensity compared with the median of
the window: calm, burst above 3×, anomaly above 10×, or none), bars or area, division the window: calm, burst above 3×, anomaly above 10×, or none), bars or area, division
(automatic, about 100 intervals, or fixed: 1 s, 10 s, 1 min, 5 min, 1 h, 1 day) and refresh (automatic, about 100 intervals, or fixed: 1 s, 10 s, 1 min, 5 min, 1 h, 1 day) and refresh
@@ -131,7 +141,7 @@ colored and exported like syslog messages:
- The severity comes from the line itself when the application writes it: JSON - The severity comes from the line itself when the application writes it: JSON
(`"level":"error"`), logfmt (`level=warn`), `[ERROR]`, or an upper-case level word at the (`"level":"error"`), logfmt (`level=warn`), `[ERROR]`, or an upper-case level word at the
start of the line (`ERROR`, `WARN`…). Otherwise it is `info`. Terminal color codes are removed. start of the line (`ERROR`, `WARN`…). Otherwise it is `info`. Terminal color codes are removed.
- **Settings > Sources** shows one label per container (`project/service`) in a single field: - **Settings > Log sources** shows one label per container (`project/service`) in a single field:
followed containers in color, then not followed ones in grey; a click switches a label. followed containers in color, then not followed ones in grey; a click switches a label.
The color identifies the compose project, and Docker app names in the log list use the The color identifies the compose project, and Docker app names in the log list use the
same color. Stopped containers are hidden by default ("Show stopped containers" displays same color. Stopped containers are hidden by default ("Show stopped containers" displays
@@ -155,7 +165,7 @@ which only lets through listing containers, reading logs, events and engine info
Logstream can also collect the system logs of the machine hosting the stack, without Logstream can also collect the system logs of the machine hosting the stack, without
configuring anything on the host. The source is **off by default**: turn it on in configuring anything on the host. The source is **off by default**: turn it on in
**Settings > Sources > Host system logs** (saved in `/data/hostlogs.json`). **Settings > Log sources > Host system logs** (saved in `/data/hostlogs.json`).
- `docker-compose.yml` mounts `/var/log` and `/run/log/journal` read-only under `/host`. - `docker-compose.yml` mounts `/var/log` and `/run/log/journal` read-only under `/host`.
- When the host runs systemd, Logstream reads the **systemd journal** files directly (no - When the host runs systemd, Logstream reads the **systemd journal** files directly (no
@@ -170,13 +180,105 @@ configuring anything on the host. The source is **off by default**: turn it on i
- **Permissions**: the container runs as an unprivileged user and gets the `adm` group - **Permissions**: the container runs as an unprivileged user and gets the `adm` group
(gid 4), which can read the journal and `/var/log` on Debian and Ubuntu. On other systems, (gid 4), which can read the journal and `/var/log` on Debian and Ubuntu. On other systems,
set `HOST_LOGS_GID` in `.env` to the gid of `systemd-journal` set `HOST_LOGS_GID` in `.env` to the gid of `systemd-journal`
(`getent group systemd-journal | cut -d: -f3`). Settings > Sources shows a clear message when (`getent group systemd-journal | cut -d: -f3`). Settings > Log sources shows a clear message when
access is denied. access is denied.
- Limits: journal fields compressed by journald (messages longer than 512 bytes, compressed - Limits: journal fields compressed by journald (messages longer than 512 bytes, compressed
with zstd/lz4/xz) cannot be decoded without extra libraries; they are counted in Settings and with zstd/lz4/xz) cannot be decoded without extra libraries; they are counted in Settings and
shown as "(compressed journal entry)". Rotated or compressed text files (`*.1`, `*.gz`) are shown as "(compressed journal entry)". Rotated or compressed text files (`*.1`, `*.gz`) are
not read. not read.
## Kernel messages (dmesg)
Logstream can collect the kernel ring buffer of the host (what `dmesg` shows: hardware,
drivers, OOM killer, firewall…) by reading `/dev/kmsg`. The source is **off by default**: turn it
on in **Settings > Log sources > Kernel messages (dmesg)** (saved in `/data/dmesg.json`).
- **Device**: the container does not see `/dev/kmsg` by default. Uncomment the `devices`
section of `docker-compose.yml` (`/dev/kmsg:/dev/kmsg:r`, read-only), then
`docker compose up -d`.
- **Permissions**: when the host has `kernel.dmesg_restrict=1` (the default on Debian and
Ubuntu), only processes with the `CAP_SYSLOG` capability can read the kernel log, and the
container runs as an unprivileged user. Either:
- allow it on the host: `sysctl -w kernel.dmesg_restrict=0`, kept across reboots with
`echo kernel.dmesg_restrict=0 | sudo tee /etc/sysctl.d/50-dmesg.conf` (every user of the
host can then run `dmesg`); this setting cannot be changed from a container;
- or run Logstream as root with the `SYSLOG` capability: uncomment `user: "0:0"` and
`cap_add: [SYSLOG]` in `docker-compose.yml`.
Settings > Log sources says when the device is missing or access is denied; the rest of
Logstream keeps working and the device is tried again every 10 seconds.
- **Fields**: **app** is `kernel` (or the program name for messages written by programs such
as `systemd[1]: …`), severity and facility come from the record, **host** is `localhost`,
`subsystem` and `device` are kept when the kernel gives them. These logs have the `kernel`
source (`proto` `kmsg`): the **Source** filter shows them alone.
- **Time**: records carry the time since boot; Logstream turns it into a date using the boot
time (`/proc/uptime`). As with `dmesg -T`, the time can be early after the host was suspended.
- When the source is turned on, the last hour is read first (`DMESG_BACKFILL`, `0` = only new
messages). The sequence number of the last message and the boot ID are saved in
`/data/dmesg-state.json`: after a restart of Logstream, reading resumes after the last
message (no duplicates); after a reboot of the host, the new buffer is read from
`DMESG_BACKFILL` ago. Messages overwritten in the kernel buffer before being read are counted
in Settings.
## Linux audit (auditd)
Logstream can follow the log of the Linux audit daemon (`/var/log/audit/audit.log`): logins,
`sudo`, watched files, syscalls matched by your `auditctl` rules… The source is **off by
default**: turn it on in **Settings > Log sources > Linux audit (auditd)**, where the file path can
also be changed (saved in `/data/audit.json`; `AUDIT_LOG` sets the default path).
- **Mount**: uncomment `/var/log/audit:/var/log/audit:ro` in `docker-compose.yml`, then
`docker compose up -d`. (`/var/log` is already mounted under `/host` for the host system
logs, so `/host/var/log/audit/audit.log` also works as the path, without a new volume.)
- **Permissions** (host side): auditd writes its log as `root`, mode `0600`, in a `0700`
directory, and Logstream runs as UID `10001`. The reliable way is the group the container
already gets for the host logs (`HOST_LOGS_GID`, `adm` by default):
```bash
sudo sed -i 's/^log_group.*/log_group = adm/' /etc/audit/auditd.conf
sudo chgrp adm /var/log/audit && sudo chmod 750 /var/log/audit
sudo service auditd restart # systemctl refuses to restart auditd on some distributions
```
auditd then creates every rotated file with mode `0640` and group `adm`. An ACL also works
(`sudo setfacl -m u:10001:rx /var/log/audit && sudo setfacl -m u:10001:r /var/log/audit/audit.log`)
but must be applied again to each new file after a rotation. Settings > Log sources says when the
file is missing or access is denied; it is tried again every second.
- **Events**: the records of one event (`SYSCALL`, `EXECVE`, `CWD`, `PATH`, `PROCTITLE`…) share
the same `msg=audit(time:serial)` and become **one** log, once no record came for 2 seconds.
The message is a summary — who, what, result — such as
`cedric (as root): exec apt install nginx [exec] → success` or
`bob: PAM:authentication via /usr/sbin/sshd from 203.0.113.9 on ssh → failed`; the raw
records are kept in `audit_raw` and shown under the summary (folded in compact density until
the line is opened). Hexadecimal values (`proctitle`, `cmd`, arguments with spaces) are
decoded.
- **Fields**: `audit_type` (type of the first record, also used as **app**), `audit_serial`,
`auid` (login user ID), `user` (its name, or the uid when it cannot be resolved), `exe`, `res`
(`success` / `failed`, from `res=` or `success=`) and `key` (rule key). Severity is `warning`
when the result is a failure, `info` otherwise; facility `authpriv`; source `audit`. In the
list, audit lines have a violet left edge, a light tint and an **AUDIT** badge before the app;
the **Source** filter shows them alone.
- **Names**: `user` comes from the enriched format (`log_format = ENRICHED`, the default of
recent auditd versions), else from the host's `/etc/passwd` when it is mounted as
`/host/etc/passwd`. **host** is the `node=` of the records (`name_format = hostname` in
`auditd.conf`), else `AUDIT_HOST`, else `/host/etc/hostname`, else `localhost`.
- When the source is turned on, the last hour is read first (`AUDIT_BACKFILL`, `0` = only new
events). The inode and an offset before which every event was sent are saved in
`/data/audit-state.json`: after a restart, reading resumes there; after a rotation by auditd,
the end of `audit.log.1` is read before the new file.
## LogStream internal logs
Logstream records its own events as logs, with **host** `logstream`, **app** the component
(`store`, `syslog`, `docker`, `host`, `dmesg`, `audit`, `main`) and the `logstream` source:
VictoriaLogs unreachable and back, lost batches, batches refused from the disk buffer, syslog
listener errors, Docker API or container streams lost, sources that cannot read, start and
stop. They are on by default from level **warning**; **Settings > Log sources > LogStream internal
logs** turns them off or changes the minimum level (saved in `/data/selflog.json`). The same
event (component and message) is recorded at most once every 30 seconds, with the number of
repeats left out; events go through a bounded queue and never wait for the storage, so an
unreachable VictoriaLogs cannot make them pile up or loop. Everything still goes to
`docker compose logs logstream` as before. In the list, these lines have a turquoise left edge
and an **LS** badge before the app; the **Source** filter shows them alone.
## CSV export ## CSV export
The **Export** button (next to the log count) downloads every stored log matching the The **Export** button (next to the log count) downloads every stored log matching the
@@ -205,7 +307,7 @@ remembered per browser.
format of the reception time: `DD/MM/YYYY HH:MM:SS` (default, the usual French display), format of the reception time: `DD/MM/YYYY HH:MM:SS` (default, the usual French display),
with milliseconds, `YYYY-MM-DD`, 12-hour clock, ISO 8601 or Unix epoch. The time zone with milliseconds, `YYYY-MM-DD`, 12-hour clock, ISO 8601 or Unix epoch. The time zone
applies to every date shown. applies to every date shown.
- **Filters**: color tags. Each tag has a keyword, a background color (the text automatically - **Patterns**: color tags. Each tag has a keyword, a background color (the text automatically
switches to black or white to stay readable) and options: whole word, match case, switches to black or white to stay readable) and options: whole word, match case,
regular expression, active. Tags are stored on the server in `/data/tags.json` regular expression, active. Tags are stored on the server in `/data/tags.json`
(`logstream-data` volume), so they are shared by every browser. Default tags (pastel): (`logstream-data` volume), so they are shared by every browser. Default tags (pastel):
@@ -222,10 +324,10 @@ remembered per browser.
or the method, not the text around it. or the method, not the text around it.
Each tag gets a two-digit code (`01`, `02`…) assigned by the server: it stays with the tag Each tag gets a two-digit code (`01`, `02`…) assigned by the server: it stays with the tag
until the tag is deleted (codes are also given to tags created by earlier versions). The until the tag is deleted (codes are also given to tags created by earlier versions). The
*Filters* column of the log list shows, as grey badges, the codes of the active tags found in *Pattern* column of the log list shows, as grey badges, the codes of the active tags found in
each message; it has room for 3, beyond that it shows 2 and `+N`, and the tooltip lists them each message; it has room for 3, beyond that it shows 2 and `+N`, and the tooltip lists them
all. all.
- **Interface** - **Appearance**
- *Theme*: System (follows the computer/phone preference), Light or Dark. The sun/moon - *Theme*: System (follows the computer/phone preference), Light or Dark. The sun/moon
button in the header switches between light and dark. button in the header switches between light and dark.
- *Log display*: font size (tiny, small, medium, large), density (normal, or compact to - *Log display*: font size (tiny, small, medium, large), density (normal, or compact to
@@ -237,7 +339,7 @@ remembered per browser.
[Bunny Fonts](https://fonts.bunny.net), a privacy-friendly European font service; without [Bunny Fonts](https://fonts.bunny.net), a privacy-friendly European font service; without
internet access, the system font is used. Ligatures are disabled so `->` or `!=` show as internet access, the system font is used. Ligatures are disabled so `->` or `!=` show as
typed. The densest setting is Tiny + Compact + Inconsolata Condensed. typed. The densest setting is Tiny + Compact + Inconsolata Condensed.
- **Data**: database figures read from VictoriaLogs (refreshed at most every 30 s): stored - **Database**: database figures read from VictoriaLogs (refreshed at most every 30 s): stored
lines, size on disk (index included), raw size and compression ratio, period covered with lines, size on disk (index included), raw size and compression ratio, period covered with
the retention, lines of the last 24 h and last hour, distinct hosts and apps, free disk the retention, lines of the last 24 h and last hour, distinct hosts and apps, free disk
space. Sizes use KB/MB/GB (Ko/Mo/Go in French). "Delete all logs" permanently erases every stored log (you must type space. Sizes use KB/MB/GB (Ko/Mo/Go in French). "Delete all logs" permanently erases every stored log (you must type
@@ -353,6 +455,11 @@ are only known by your router or a local DNS (Pi-hole, AdGuard, Unbound…), set
| `HOST_LOGS_GID` | `4` (adm) in compose | group given to the container to read the host logs | | `HOST_LOGS_GID` | `4` (adm) in compose | group given to the container to read the host logs |
| `HOST_LOGS_BACKFILL` | `1h` | history read when the host system logs source is turned on | | `HOST_LOGS_BACKFILL` | `1h` | history read when the host system logs source is turned on |
| `HOST_LOGS_ROOT` | `/host` | where the host directories are mounted | | `HOST_LOGS_ROOT` | `/host` | where the host directories are mounted |
| `DMESG_BACKFILL` | `1h` | history read when the kernel messages source is turned on |
| `DMESG_DEVICE` | `/dev/kmsg` | kernel log device read by the kernel messages source |
| `AUDIT_LOG` | `/var/log/audit/audit.log` | default path of the Linux audit log (can be changed in Settings > Log sources) |
| `AUDIT_BACKFILL` | `1h` | history read when the Linux audit source is turned on |
| `AUDIT_HOST` | empty | host name of the audit events when the records carry no `node=` |
| `TZ` | `Europe/Paris` | time zone for RFC 3164 timestamps (which carry none) | | `TZ` | `Europe/Paris` | time zone for RFC 3164 timestamps (which carry none) |
| `BATCH_SIZE`, `FLUSH_MS`, `QUEUE_SIZE` | `1000`, `1000`, `100000` | ingestion tuning | | `BATCH_SIZE`, `FLUSH_MS`, `QUEUE_SIZE` | `1000`, `1000`, `100000` | ingestion tuning |
| `SPOOL_MAX_MB` | `1024` | disk buffer size for batches VictoriaLogs could not take (`0` = no buffer: retried for 15 s, then dropped) | | `SPOOL_MAX_MB` | `1024` | disk buffer size for batches VictoriaLogs could not take (`0` = no buffer: retried for 15 s, then dropped) |
@@ -391,7 +498,7 @@ To update one of them:
VictoriaLogs keeps its storage format across minor versions; read the changelog before a VictoriaLogs keeps its storage format across minor versions; read the changelog before a
major version change. major version change.
2. Change the version in the file above, then run `docker compose up -d --build`. 2. Change the version in the file above, then run `docker compose up -d --build`.
3. Check the bottom bar (received / stored / dropped) and **Settings > Sources**. To go back, 3. Check the bottom bar (received / stored / dropped) and **Settings > Log sources**. To go back,
restore the previous version and run the same command. restore the previous version and run the same command.
## Code layout ## Code layout
@@ -409,8 +516,11 @@ To update one of them:
| `rdns.go` | cached reverse DNS lookups | | `rdns.go` | cached reverse DNS lookups |
| `export.go` | streamed CSV export | | `export.go` | streamed CSV export |
| `docker.go` | Docker container logs (API, followers, positions, level detection) | | `docker.go` | Docker container logs (API, followers, positions, level detection) |
| `syslogserver.go` | syslog listeners opened and closed from Settings > Sources | | `syslogserver.go` | syslog listeners opened and closed from Settings > Log sources |
| `hostlogs.go`, `journal.go` | host system logs: systemd journal reader (no `journalctl`) and `/var/log` follower | | `hostlogs.go`, `journal.go` | host system logs: systemd journal reader (no `journalctl`) and `/var/log` follower |
| `dmesg.go` | kernel messages: `/dev/kmsg` reader, record parsing, position by boot ID and sequence number |
| `audit.go` | Linux audit: `audit.log` follower (rotation, saved offset), record parsing, grouping by event, summary |
| `selflog.go` | LogStream internal logs: `logEvent`, minimum level, throttling, bounded queue |
| `tags.go` | color tag storage | | `tags.go` | color tag storage |
| `presets.go`, `presets.json` | color tag presets (`/api/presets`), built-in list | | `presets.go`, `presets.json` | color tag presets (`/api/presets`), built-in list |
| `api.go` | `/api/*` HTTP routes | | `api.go` | `/api/*` HTTP routes |
+69
View File
@@ -23,6 +23,9 @@ type API struct {
docker *DockerManager // nil when DOCKER_LOGS is off docker *DockerManager // nil when DOCKER_LOGS is off
syslog *SyslogServer syslog *SyslogServer
host *HostLogs host *HostLogs
dmesg *Dmesg
audit *Audit
selfLog *SelfLog
dbstats dbStatsCache dbstats dbStatsCache
} }
@@ -49,6 +52,72 @@ func (a *API) Routes(mux *http.ServeMux) {
mux.HandleFunc("PUT /api/docker", a.dockerConfigure) mux.HandleFunc("PUT /api/docker", a.dockerConfigure)
mux.HandleFunc("GET /api/hostlogs", a.hostLogsStatus) mux.HandleFunc("GET /api/hostlogs", a.hostLogsStatus)
mux.HandleFunc("PUT /api/hostlogs", a.hostLogsConfigure) mux.HandleFunc("PUT /api/hostlogs", a.hostLogsConfigure)
mux.HandleFunc("GET /api/dmesg", a.dmesgStatus)
mux.HandleFunc("PUT /api/dmesg", a.dmesgConfigure)
mux.HandleFunc("GET /api/audit", a.auditStatus)
mux.HandleFunc("PUT /api/audit", a.auditConfigure)
mux.HandleFunc("GET /api/selflog", a.selfLogStatus)
mux.HandleFunc("PUT /api/selflog", a.selfLogConfigure)
}
// GET /api/audit: state of the Linux audit source (Settings > Sources).
func (a *API) auditStatus(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, a.audit.Status())
}
// PUT /api/audit {"enabled": bool, "path": "/var/log/audit/audit.log"}
func (a *API) auditConfigure(w http.ResponseWriter, r *http.Request) {
var cfg auditConfig
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&cfg); err != nil {
writeErr(w, http.StatusBadRequest, err)
return
}
if err := a.audit.Configure(cfg); err != nil {
writeErr(w, http.StatusInternalServerError, err)
return
}
log.Printf("audit source changed from %s: %+v", r.RemoteAddr, cfg)
writeJSON(w, http.StatusOK, a.audit.Status())
}
// GET /api/selflog: state of the LogStream internal logs (Settings > Sources).
func (a *API) selfLogStatus(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, a.selfLog.Status())
}
// PUT /api/selflog {"enabled": bool, "min_level": "warning"}
func (a *API) selfLogConfigure(w http.ResponseWriter, r *http.Request) {
var cfg selfLogConfig
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&cfg); err != nil {
writeErr(w, http.StatusBadRequest, err)
return
}
if err := a.selfLog.Configure(cfg); err != nil {
writeErr(w, http.StatusInternalServerError, err)
return
}
log.Printf("internal logs changed from %s: %+v", r.RemoteAddr, cfg)
writeJSON(w, http.StatusOK, a.selfLog.Status())
}
// GET /api/dmesg: state of the kernel messages source (Settings > Sources).
func (a *API) dmesgStatus(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, a.dmesg.Status())
}
// PUT /api/dmesg {"enabled": bool}
func (a *API) dmesgConfigure(w http.ResponseWriter, r *http.Request) {
var cfg dmesgConfig
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&cfg); err != nil {
writeErr(w, http.StatusBadRequest, err)
return
}
if err := a.dmesg.Configure(cfg); err != nil {
writeErr(w, http.StatusInternalServerError, err)
return
}
log.Printf("kernel messages source changed from %s: %+v", r.RemoteAddr, cfg)
writeJSON(w, http.StatusOK, a.dmesg.Status())
} }
// GET /api/hostlogs: state of the host system logs source (Settings > Sources). // GET /api/hostlogs: state of the host system logs source (Settings > Sources).
+830
View File
@@ -0,0 +1,830 @@
package main
import (
"bufio"
"bytes"
"context"
"encoding/hex"
"encoding/json"
"errors"
"io"
"io/fs"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"sync"
"syscall"
"time"
"unicode/utf8"
)
// auditConfig is saved in /data/audit.json and edited in Settings > Sources.
type auditConfig struct {
Enabled bool `json:"enabled"`
Path string `json:"path"` // empty: AUDIT_LOG (/var/log/audit/audit.log)
}
// auditPos is where reading resumes: the inode of the file and an offset
// before which every event was sent.
type auditPos struct {
Ino uint64 `json:"ino"`
Off int64 `json:"off"`
}
// Audit follows the log of the Linux audit daemon (auditd) and sends one
// entry per audit event: the records of an event (SYSCALL, EXECVE, PATH…)
// share the same msg=audit(time:serial) and are grouped together.
type Audit struct {
defPath string
hostRoot string // to read the host's /etc/passwd and /etc/hostname when mounted
hostName string // AUDIT_HOST: host name of the entries when records have no node=
sink func(*Entry)
backfill time.Duration
cfgPath string
statePath string
wake chan struct{}
// Owned by the Run goroutine.
pos auditPos // checkpoint saved in /data
readOff int64 // where the next read starts (after the lines of open events)
known bool
dirty bool
notBefore time.Time
asm *auditAssembler
users *userNames
mu sync.Mutex
cfg auditConfig
reset bool
open bool
read uint64
errCode string
errDetail string
}
func NewAudit(defPath, hostRoot, hostName, dataDir string, backfill time.Duration, sink func(*Entry)) *Audit {
a := &Audit{
defPath: defPath,
hostRoot: hostRoot,
hostName: hostName,
sink: sink,
backfill: backfill,
cfgPath: filepath.Join(dataDir, "audit.json"),
statePath: filepath.Join(dataDir, "audit-state.json"),
wake: make(chan struct{}, 1),
asm: newAuditAssembler(),
users: &userNames{path: filepath.Join(hostRoot, "etc/passwd")},
}
if b, err := os.ReadFile(a.cfgPath); err == nil {
_ = json.Unmarshal(b, &a.cfg)
}
if b, err := os.ReadFile(a.statePath); err == nil && json.Unmarshal(b, &a.pos) == nil {
a.known, a.readOff = true, a.pos.Off
}
if a.hostName == "" {
a.hostName = readTrim(filepath.Join(hostRoot, "etc/hostname"))
}
return a
}
func (a *Audit) path() string {
a.mu.Lock()
defer a.mu.Unlock()
if a.cfg.Path != "" {
return a.cfg.Path
}
return a.defPath
}
// Run polls the file every second while the source is enabled.
func (a *Audit) Run(ctx context.Context) {
tick := time.NewTicker(time.Second)
defer tick.Stop()
n := 0
for {
select {
case <-ctx.Done():
a.emit(a.asm.flushAll())
a.checkpoint()
a.saveState()
return
case <-tick.C:
case <-a.wake:
}
a.mu.Lock()
enabled, reset := a.cfg.Enabled, a.reset
a.reset = false
a.mu.Unlock()
if reset {
// Disabled then enabled again (or another file): start over from
// AUDIT_BACKFILL ago rather than reading everything in between.
a.asm = newAuditAssembler()
a.pos, a.readOff, a.known, a.dirty = auditPos{}, 0, false, true
}
if enabled {
a.poll(time.Now())
}
if n++; n%5 == 0 || reset {
a.saveState()
}
}
}
func (a *Audit) saveState() {
if !a.dirty {
return
}
a.dirty = false
if err := writeJSONFile(a.statePath, a.pos); err != nil {
logEvent("audit", 4, "saving position: %v", err)
}
}
func (a *Audit) fail(err error) {
code := "read"
switch {
case errors.Is(err, fs.ErrNotExist):
code = "not_found"
case errors.Is(err, fs.ErrPermission):
code = "permission"
}
a.mu.Lock()
changed := code != a.errCode
a.open, a.errCode, a.errDetail = false, code, err.Error()
a.mu.Unlock()
if changed {
logEvent("audit", 4, "cannot read the audit log: %v", err)
}
}
const maxAuditRead = 4 << 20 // per poll
// poll reads the lines added since the previous one, follows rotations and
// sends the events that are complete.
func (a *Audit) poll(now time.Time) {
path := a.path()
fi, err := os.Stat(path)
if err != nil {
a.fail(err)
return
}
ino := inodeOf(fi)
if !a.known {
// First read: the last AUDIT_BACKFILL (0: only new events).
a.pos, a.readOff, a.known, a.dirty = auditPos{Ino: ino}, 0, true, true
a.notBefore = time.Time{}
if a.backfill <= 0 {
a.readOff = fi.Size()
} else {
a.notBefore = now.Add(-a.backfill)
}
}
if a.pos.Ino != ino {
// Rotated by auditd (audit.log -> audit.log.1): finish the old file,
// then read the new one from its start.
if old, err := os.Stat(path + ".1"); err == nil && inodeOf(old) == a.pos.Ino {
for {
off, more, err := a.readFrom(path+".1", a.readOff, now)
if err != nil || !more {
break
}
a.readOff = off
}
}
a.emit(a.asm.flushAll())
a.pos, a.readOff, a.dirty = auditPos{Ino: ino}, 0, true
} else if fi.Size() < a.readOff {
a.emit(a.asm.flushAll()) // truncated
a.readOff = 0
}
if fi.Size() > a.readOff {
off, _, err := a.readFrom(path, a.readOff, now)
if err != nil {
a.fail(err)
return
}
a.readOff = off
}
a.mu.Lock()
if !a.open {
a.open, a.errCode, a.errDetail = true, "", ""
}
a.mu.Unlock()
a.emit(a.asm.flush(now, auditCloseDelay))
a.checkpoint()
}
// checkpoint saves an offset before which every event is sent: the start of
// the oldest event still open, or the read offset.
func (a *Audit) checkpoint() {
off := a.readOff
if o, ok := a.asm.minOff(); ok && o < off {
off = o
}
if off != a.pos.Off {
a.pos.Off, a.dirty = off, true
}
}
// readFrom feeds the complete lines written after off to the assembler and
// returns the new offset; more tells whether the read window was full.
func (a *Audit) readFrom(path string, off int64, now time.Time) (int64, bool, error) {
f, err := os.Open(path)
if err != nil {
return off, false, err
}
defer f.Close()
buf, err := io.ReadAll(io.NewSectionReader(f, off, maxAuditRead))
if err != nil {
return off, false, err
}
full := len(buf) == maxAuditRead
end := bytes.LastIndexByte(buf, '\n')
if end < 0 {
if !full {
return off, false, nil // partial line: wait for its end
}
end = len(buf) - 1
}
start := off
for _, line := range bytes.Split(buf[:end], []byte{'\n'}) {
a.emit(a.asm.add(string(line), start, now))
start += int64(len(line)) + 1
}
return off + int64(end) + 1, full, nil
}
func (a *Audit) emit(evs []*auditEvent) {
for _, ev := range evs {
if !a.notBefore.IsZero() && ev.Time.Before(a.notBefore) {
continue
}
a.sink(auditEntry(ev, a.users, a.hostName, time.Now()))
a.mu.Lock()
a.read++
a.mu.Unlock()
}
}
func inodeOf(fi os.FileInfo) uint64 {
if st, ok := fi.Sys().(*syscall.Stat_t); ok {
return uint64(st.Ino)
}
return 0
}
// Configure saves and applies the configuration.
func (a *Audit) Configure(cfg auditConfig) error {
cfg.Path = strings.TrimSpace(cfg.Path)
a.mu.Lock()
if (a.cfg.Enabled && !cfg.Enabled) || cfg.Path != a.cfg.Path {
a.reset = true
a.open, a.errCode, a.errDetail = false, "", ""
}
a.cfg = cfg
a.mu.Unlock()
select {
case a.wake <- struct{}{}:
default:
}
return writeJSONFile(a.cfgPath, cfg)
}
// Status is the state shown in Settings > Sources.
func (a *Audit) Status() map[string]any {
a.mu.Lock()
defer a.mu.Unlock()
return map[string]any{
"enabled": a.cfg.Enabled,
"path": a.cfg.Path,
"defaultPath": a.defPath,
"open": a.open && a.cfg.Enabled,
"read": a.read,
"code": a.errCode,
"error": a.errDetail,
}
}
/* ---------- Records and events ---------- */
// auditCloseDelay: an event is complete when no line of it came for this
// long, or when the log has moved this far past its time (auditd does not
// write the EOE record that ends multi-record events).
const auditCloseDelay = 2 * time.Second
type auditField struct {
k, v string
bare bool // written without quotes: may be hexadecimal
}
type auditRecord struct {
Type string
Fields []auditField
Raw string
}
type auditEvent struct {
ID string // "1700000000.123:456"
Serial string
Time time.Time
Node string
Records []auditRecord
off int64 // offset of its first line in the file
seen time.Time // when its last line was read
order int
}
// parseAuditLine decodes "[node=N ]type=T msg=audit(SEC.MS:SERIAL): k=v …".
func parseAuditLine(line string) (rec auditRecord, id, node string, ok bool) {
line = strings.TrimRight(line, "\r")
rec.Raw = line
head, rest, found := strings.Cut(line, "): ")
if !found {
head, found = strings.CutSuffix(line, ")")
if !found {
return rec, "", "", false
}
}
pre, ts, found := strings.Cut(head, "msg=audit(")
if !found || ts == "" {
return rec, "", "", false
}
for _, f := range parseAuditKV(pre) {
switch f.k {
case "type":
rec.Type = f.v
case "node":
node = f.v
}
}
if rec.Type == "" {
return rec, "", "", false
}
// Enriched format (log_format=ENRICHED): resolved values after a GS byte.
main, enriched, _ := strings.Cut(rest, "\x1d")
rec.Fields = parseAuditKV(main)
for _, f := range parseAuditKV(enriched) {
rec.Fields = append(rec.Fields, auditField{k: "@" + f.k, v: f.v})
}
return rec, ts, node, true
}
// hexFields are the fields auditd writes in hexadecimal when they contain
// spaces, quotes or control characters (and then without quotes).
var hexFields = map[string]bool{
"proctitle": true, "cmd": true, "exe": true, "comm": true, "key": true, "acct": true,
"name": true, "cwd": true, "data": true, "path": true, "old": true, "new": true,
}
// parseAuditKV splits "k=v k2="v 2" msg='k3=v3 …'"; the fields of a quoted
// msg='…' (user space records) follow the outer ones.
func parseAuditKV(s string) []auditField {
var out []auditField
var inner []auditField
for i := 0; i < len(s); {
for i < len(s) && s[i] == ' ' {
i++
}
eq := strings.IndexByte(s[i:], '=')
if eq <= 0 {
break
}
k := s[i : i+eq]
if sp := strings.IndexByte(k, ' '); sp >= 0 { // word without value
i += sp + 1
continue
}
i += eq + 1
var v string
quoted := i < len(s) && (s[i] == '"' || s[i] == '\'')
if quoted {
q := s[i]
end := strings.IndexByte(s[i+1:], q)
if end < 0 {
end = len(s) - i - 1
}
v = s[i+1 : i+1+end]
i += end + 2
} else {
end := strings.IndexByte(s[i:], ' ')
if end < 0 {
end = len(s) - i
}
v = s[i : i+end]
i += end
}
if k == "msg" && quoted && strings.Contains(v, "=") {
inner = append(inner, parseAuditKV(v)...)
continue
}
if !quoted && hexFields[k] {
v = decodeHex(k, v)
}
if !quoted && (v == "(null)" || v == "?") {
v = ""
}
out = append(out, auditField{k: k, v: v, bare: !quoted})
}
return append(out, inner...)
}
// decodeHex decodes a hexadecimal value, or returns it unchanged.
func decodeHex(k, v string) string {
if len(v) < 2 || len(v)%2 != 0 {
return v
}
b, err := hex.DecodeString(v)
if err != nil {
return v
}
if k == "proctitle" {
b = bytes.ReplaceAll(bytes.TrimRight(b, "\x00"), []byte{0}, []byte{' '})
}
if !utf8.Valid(b) {
return v
}
return string(b)
}
func isDigits(s string) bool {
for i := 0; i < len(s); i++ {
if s[i] < '0' || s[i] > '9' {
return false
}
}
return s != ""
}
// parseAuditTime decodes "1700000000.123:456" into a time and a serial.
func parseAuditTime(id string) (time.Time, string) {
ts, serial, _ := strings.Cut(id, ":")
sec, frac, _ := strings.Cut(ts, ".")
s, err := strconv.ParseInt(sec, 10, 64)
if err != nil {
return time.Time{}, serial
}
ms, _ := strconv.ParseInt((frac + "000")[:3], 10, 64)
return time.Unix(s, ms*int64(time.Millisecond)), serial
}
// auditAssembler groups the records of each event.
type auditAssembler struct {
open map[string]*auditEvent
newest time.Time // most recent event time read
n int
}
func newAuditAssembler() *auditAssembler {
return &auditAssembler{open: map[string]*auditEvent{}}
}
const maxOpenAudit = 2000
// add takes one line read at offset off and returns the events it completes
// (EOE record, or too many events open).
func (a *auditAssembler) add(line string, off int64, now time.Time) []*auditEvent {
if strings.TrimSpace(line) == "" {
return nil
}
rec, id, node, ok := parseAuditLine(line)
if !ok {
return nil
}
ev := a.open[id]
if rec.Type == "EOE" {
if ev == nil {
return nil
}
delete(a.open, id)
return []*auditEvent{ev}
}
if ev == nil {
t, serial := parseAuditTime(id)
a.n++
ev = &auditEvent{ID: id, Serial: serial, Time: t, Node: node, off: off, order: a.n}
a.open[id] = ev
if t.After(a.newest) {
a.newest = t
}
}
ev.Records = append(ev.Records, rec)
ev.seen = now
if len(a.open) > maxOpenAudit {
return a.closeOldest(len(a.open) - maxOpenAudit/2)
}
return nil
}
// flush returns the events with no new line for delay, or that the log has
// left delay behind, in the order they started.
func (a *auditAssembler) flush(now time.Time, delay time.Duration) []*auditEvent {
var out []*auditEvent
for id, ev := range a.open {
if now.Sub(ev.seen) >= delay || a.newest.Sub(ev.Time) >= delay {
out = append(out, ev)
delete(a.open, id)
}
}
sortEvents(out)
return out
}
func (a *auditAssembler) flushAll() []*auditEvent {
return a.closeOldest(len(a.open))
}
func (a *auditAssembler) closeOldest(n int) []*auditEvent {
all := make([]*auditEvent, 0, len(a.open))
for _, ev := range a.open {
all = append(all, ev)
}
sortEvents(all)
all = all[:n]
for _, ev := range all {
delete(a.open, ev.ID)
}
return all
}
func sortEvents(evs []*auditEvent) {
sort.Slice(evs, func(i, j int) bool { return evs[i].order < evs[j].order })
}
func (a *auditAssembler) minOff() (int64, bool) {
var m int64
ok := false
for _, ev := range a.open {
if !ok || ev.off < m {
m, ok = ev.off, true
}
}
return m, ok
}
/* ---------- Entries ---------- */
// get returns the first non-empty value of a field across the records.
func (ev *auditEvent) get(k string) string {
for _, r := range ev.Records {
for _, f := range r.Fields {
if f.k == k && f.v != "" {
return f.v
}
}
}
return ""
}
func (ev *auditEvent) record(typ string) *auditRecord {
for i := range ev.Records {
if ev.Records[i].Type == typ {
return &ev.Records[i]
}
}
return nil
}
// result is "success", "failed" or "" from res= or success=.
func (ev *auditEvent) result() string {
v := strings.ToLower(firstNonEmpty(ev.get("res"), ev.get("success"), ev.get("result")))
switch v {
case "success", "yes", "1":
return "success"
case "failed", "fail", "failure", "no", "0":
return "failed"
}
return ""
}
const auditUnset = "4294967295"
// userOf resolves a uid: enriched name, then the host's /etc/passwd, then the number.
func (ev *auditEvent) userOf(field string, users *userNames) string {
uid := ev.get(field)
if uid == "" || uid == auditUnset || uid == "-1" {
return ""
}
if n := strings.Trim(ev.get("@"+strings.ToUpper(field)), `"`); n != "" && n != "unset" && n != uid {
return n
}
if n := users.name(uid); n != "" {
return n
}
return uid
}
// auditEntry turns an event into an Entry: app is the type of its first record.
func auditEntry(ev *auditEvent, users *userNames, hostName string, now time.Time) *Entry {
typ := "UNKNOWN"
if len(ev.Records) > 0 {
typ = ev.Records[0].Type
}
res := ev.result()
auid := ev.get("auid")
user := ev.userOf("auid", users)
if user == "" {
user = ev.userOf("uid", users)
}
sev := 6
if res == "failed" {
sev = 4
}
raw := make([]string, len(ev.Records))
for i, r := range ev.Records {
raw[i] = r.Raw
}
host := firstNonEmpty(ev.Node, hostName, "localhost")
e := &Entry{
Time: ev.Time,
Received: now,
Host: host,
App: typ,
ProcID: ev.get("pid"),
Facility: "authpriv",
Severity: severityNames[sev],
SevNum: sev,
Message: auditSummary(ev, typ, user, users, res),
Proto: "file",
SourceType: "audit",
Extra: map[string]string{
"audit_type": typ,
"audit_serial": ev.Serial,
"auid": auid,
"user": user,
"exe": ev.get("exe"),
"res": res,
"key": ev.get("key"),
"audit_raw": strings.Join(raw, "\n"),
},
Wait: true,
}
if e.Time.IsZero() {
e.Time = now
}
return e
}
// auditSummary writes "who: what [key] → result".
func auditSummary(ev *auditEvent, typ, user string, users *userNames, res string) string {
who := firstNonEmpty(user, "?")
// Syscalls: the login user and, when different, the user the process ran
// as (sudo, su). In user space records, uid is the daemon's (sshd: root).
if uid := ev.userOf("uid", users); ev.get("syscall") != "" && uid != "" && user != "" && uid != user {
who += " (as " + uid + ")"
}
var what string
exe := ev.get("exe")
switch {
case typ == "USER_CMD":
what = "sudo " + firstNonEmpty(ev.get("cmd"), "?")
if cwd := ev.get("cwd"); cwd != "" {
what += " (in " + cwd + ")"
}
case strings.HasPrefix(typ, "USER_") || strings.HasPrefix(typ, "CRED_") || typ == "LOGIN" ||
typ == "ADD_USER" || typ == "DEL_USER" || typ == "ADD_GROUP" || typ == "DEL_GROUP":
if acct := ev.get("acct"); acct != "" {
who = acct
}
what = firstNonEmpty(ev.get("op"), strings.ToLower(typ))
if exe != "" {
what += " via " + exe
}
if addr := firstNonEmpty(ev.get("hostname"), ev.get("addr")); addr != "" {
what += " from " + addr
}
if term := ev.get("terminal"); term != "" && term != "?" {
what += " on " + term
}
case ev.record("EXECVE") != nil:
what = "exec " + execLine(ev.record("EXECVE"))
case ev.get("syscall") != "":
what = "syscall " + firstNonEmpty(ev.get("@SYSCALL"), ev.get("syscall"))
if name := pathName(ev); name != "" {
what += " " + name
}
if prog := firstNonEmpty(exe, ev.get("comm")); prog != "" {
what += " by " + prog
}
default:
what = strings.ToLower(typ)
if op := ev.get("op"); op != "" {
what += " " + op
}
if exe != "" {
what += " by " + exe
}
}
if key := ev.get("key"); key != "" {
what += " [" + key + "]"
}
if res != "" {
what += " → " + res
}
return who + ": " + what
}
// execLine joins the arguments of an EXECVE record (a0, a1…).
func execLine(r *auditRecord) string {
args := map[int]string{}
argc := 0
for _, f := range r.Fields {
if f.k == "argc" {
argc, _ = strconv.Atoi(f.v)
} else if len(f.k) > 1 && f.k[0] == 'a' && isDigits(f.k[1:]) {
n, _ := strconv.Atoi(f.k[1:])
if f.bare {
args[n] = decodeHex(f.k, f.v) // arguments with spaces or quotes
} else {
args[n] = f.v
}
}
}
if argc == 0 {
argc = len(args)
}
parts := make([]string, 0, argc)
for i := 0; i < argc; i++ {
if v, ok := args[i]; ok {
parts = append(parts, v)
}
}
s := strings.Join(parts, " ")
if len(s) > 300 {
s = s[:300] + "…"
}
return s
}
// pathName is the file a syscall worked on: the first PATH record that is not
// its parent directory.
func pathName(ev *auditEvent) string {
var first string
for _, r := range ev.Records {
if r.Type != "PATH" {
continue
}
var name, nametype string
for _, f := range r.Fields {
switch f.k {
case "name":
name = f.v
case "nametype":
nametype = f.v
}
}
if name == "" {
continue
}
if nametype != "PARENT" {
return name
}
if first == "" {
first = name
}
}
return first
}
/* ---------- User names ---------- */
// userNames reads uid -> name from the host's /etc/passwd when it is mounted,
// at most once a minute.
type userNames struct {
path string
names map[string]string
loaded time.Time
modTime time.Time
}
func (u *userNames) name(uid string) string {
if u == nil {
return ""
}
if time.Since(u.loaded) > time.Minute {
u.loaded = time.Now()
if fi, err := os.Stat(u.path); err == nil && !fi.ModTime().Equal(u.modTime) {
u.modTime = fi.ModTime()
u.names = readPasswd(u.path)
}
}
return u.names[uid]
}
func readPasswd(path string) map[string]string {
f, err := os.Open(path)
if err != nil {
return nil
}
defer f.Close()
names := map[string]string{}
sc := bufio.NewScanner(f)
for sc.Scan() {
p := strings.Split(sc.Text(), ":")
if len(p) >= 3 && p[0] != "" && !strings.HasPrefix(p[0], "#") {
names[p[2]] = p[0]
}
}
return names
}
+356
View File
@@ -0,0 +1,356 @@
package main
import (
"context"
"encoding/hex"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
)
func hexOf(s string) string { return strings.ToUpper(hex.EncodeToString([]byte(s))) }
// Lines of real audit.log events (trimmed).
var (
auditExec = []string{
`type=SYSCALL msg=audit(1759572000.123:4242): arch=c000003e syscall=59 success=yes exit=0 a0=4142 a1=55d2 a2=55d3 a3=0 items=2 ppid=1000 pid=1234 auid=1000 uid=0 gid=0 euid=0 tty=pts0 ses=3 comm="apt" exe="/usr/bin/apt" subj=unconfined key="exec"`,
`type=EXECVE msg=audit(1759572000.123:4242): argc=3 a0="apt" a1="install" a2=` + hexOf(`my "pkg"`),
`type=CWD msg=audit(1759572000.123:4242): cwd="/root"`,
`type=PATH msg=audit(1759572000.123:4242): item=0 name="/usr/bin/apt" inode=1 dev=fd:00 mode=0100755 ouid=0 ogid=0 nametype=NORMAL`,
`type=PROCTITLE msg=audit(1759572000.123:4242): proctitle=` + hexOf("apt\x00install\x00my \"pkg\""),
}
auditSudo = `type=USER_CMD msg=audit(1759572001.500:4250): pid=2000 uid=1000 auid=1000 ses=3 subj=unconfined msg='cwd="/home/cedric" cmd=` + hexOf("systemctl restart nginx") + ` exe="/usr/bin/sudo" terminal=pts/0 res=success'`
auditSSH = `type=USER_AUTH msg=audit(1759572002.000:4260): pid=3000 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='op=PAM:authentication grantors=? acct="bob" exe="/usr/sbin/sshd" hostname=203.0.113.9 addr=203.0.113.9 terminal=ssh res=failed'`
auditOpen = []string{
"node=web1 type=SYSCALL msg=audit(1759572003.250:4270): arch=c000003e syscall=257 success=no exit=-13 items=1 pid=4000 auid=1001 uid=1001 comm=\"cat\" exe=\"/usr/bin/cat\" key=\"shadow\"\x1dARCH=x86_64 SYSCALL=openat AUID=\"alice\" UID=\"alice\"",
`node=web1 type=PATH msg=audit(1759572003.250:4270): item=0 name="/etc/shadow" inode=7 nametype=NORMAL`,
}
)
func TestParseAuditLine(t *testing.T) {
rec, id, node, ok := parseAuditLine(auditOpen[0])
if !ok || rec.Type != "SYSCALL" || id != "1759572003.250:4270" || node != "web1" {
t.Fatalf("got %+v %q %q %v", rec, id, node, ok)
}
ev := &auditEvent{Records: []auditRecord{rec}}
for k, want := range map[string]string{"syscall": "257", "exe": "/usr/bin/cat", "key": "shadow", "@SYSCALL": "openat", "@AUID": "alice"} {
if got := ev.get(k); got != want {
t.Errorf("%s = %q, want %q", k, got, want)
}
}
for _, bad := range []string{"", "garbage", "type=X no header", "msg=audit(1.2:3): no type"} {
if _, _, _, ok := parseAuditLine(bad); ok {
t.Errorf("%q parsed", bad)
}
}
}
func TestParseAuditKV(t *testing.T) {
tests := []struct {
in string
want map[string]string
}{
{`a=1 b="two words" c=`, map[string]string{"a": "1", "b": "two words", "c": ""}},
{`pid=1 msg='op=login acct="bob" res=success'`, map[string]string{"pid": "1", "op": "login", "acct": "bob", "res": "success"}},
{`comm=` + hexOf("my prog") + ` exe="/bin/x"`, map[string]string{"comm": "my prog", "exe": "/bin/x"}},
{`key=(null) hostname=? terminal=?`, map[string]string{"key": "", "hostname": "", "terminal": ""}},
{`a0=4142 name=ABC`, map[string]string{"a0": "4142", "name": "ABC"}}, // not hexadecimal fields, or odd length
{`avc: denied { read } for pid=12 comm="x"`, map[string]string{"pid": "12", "comm": "x"}},
}
for _, tt := range tests {
got := map[string]string{}
for _, f := range parseAuditKV(tt.in) {
got[f.k] = f.v
}
for k, v := range tt.want {
if got[k] != v {
t.Errorf("%q: %s = %q, want %q (all: %v)", tt.in, k, got[k], v, got)
}
}
}
}
func TestAuditAssembler(t *testing.T) {
now := time.Unix(1759572010, 0)
a := newAuditAssembler()
// Two interleaved events, then an EOE that closes the first one.
lines := []string{auditExec[0], auditSudo, auditExec[1], auditExec[2]}
off := int64(0)
for _, l := range lines {
if out := a.add(l, off, now); out != nil {
t.Fatalf("closed early: %v", out)
}
off += int64(len(l)) + 1
}
if len(a.open) != 2 {
t.Fatalf("%d open events", len(a.open))
}
if o, ok := a.minOff(); !ok || o != 0 {
t.Fatalf("minOff %d %v", o, ok)
}
out := a.add(`type=EOE msg=audit(1759572000.123:4242): `, off, now)
if len(out) != 1 || out[0].Serial != "4242" || len(out[0].Records) != 3 {
t.Fatalf("EOE: %+v", out)
}
if o, _ := a.minOff(); o != int64(len(lines[0])+1) {
t.Fatalf("minOff after EOE: %d", o)
}
// An EOE of an unknown event is ignored.
if out := a.add(`type=EOE msg=audit(1.0:1): `, 0, now); out != nil {
t.Fatal(out)
}
// The sudo event is closed by time.
if out := a.flush(now.Add(time.Second), 2*time.Second); len(out) != 0 {
t.Fatalf("closed too early: %v", out)
}
out = a.flush(now.Add(3*time.Second), 2*time.Second)
if len(out) != 1 || out[0].Records[0].Type != "USER_CMD" {
t.Fatalf("flush: %+v", out)
}
// Events the log has left behind are closed without waiting.
a.add(auditSSH, 0, now)
a.add(auditOpen[0], 0, now)
out = a.flush(now, 1*time.Second)
if len(out) != 1 || out[0].Serial != "4260" || len(a.open) != 1 {
t.Fatalf("newest: %+v, open %d", out, len(a.open))
}
if out = a.flushAll(); len(out) != 1 || out[0].Serial != "4270" || len(a.open) != 0 {
t.Fatalf("flushAll: %+v", out)
}
}
func auditEv(t *testing.T, lines ...string) *auditEvent {
t.Helper()
a := newAuditAssembler()
for _, l := range lines {
a.add(l, 0, time.Time{})
}
out := a.flushAll()
if len(out) != 1 {
t.Fatalf("%d events", len(out))
}
return out[0]
}
func TestAuditEntry(t *testing.T) {
dir := t.TempDir()
passwd := filepath.Join(dir, "passwd")
if err := os.WriteFile(passwd, []byte("root:x:0:0::/root:/bin/sh\ncedric:x:1000:1000::/home/cedric:/bin/bash\n"), 0o644); err != nil {
t.Fatal(err)
}
now := time.Unix(1759572100, 0)
tests := []struct {
name string
lines []string
host string
app string
sev string
msg string
fields map[string]string
}{
{"exec", auditExec, "srv", "SYSCALL", "info",
`cedric (as root): exec apt install my "pkg" [exec] → success`,
map[string]string{"audit_type": "SYSCALL", "audit_serial": "4242", "auid": "1000", "user": "cedric", "exe": "/usr/bin/apt", "res": "success", "key": "exec"}},
{"sudo", []string{auditSudo}, "srv", "USER_CMD", "info",
"cedric: sudo systemctl restart nginx (in /home/cedric) → success",
map[string]string{"user": "cedric", "exe": "/usr/bin/sudo", "res": "success", "key": ""}},
{"ssh failed", []string{auditSSH}, "srv", "USER_AUTH", "warning",
"bob: PAM:authentication via /usr/sbin/sshd from 203.0.113.9 on ssh → failed",
map[string]string{"auid": "4294967295", "user": "root", "res": "failed"}},
{"enriched, node", auditOpen, "web1", "SYSCALL", "warning",
"alice: syscall openat /etc/shadow by /usr/bin/cat [shadow] → failed",
map[string]string{"user": "alice", "auid": "1001", "res": "failed", "key": "shadow"}},
{"unknown uid", []string{`type=USER_LOGIN msg=audit(1759572004.000:4280): pid=1 uid=0 auid=4242 ses=9 msg='op=login id=4242 exe="/usr/sbin/sshd" hostname=? addr=10.0.0.5 terminal=/dev/pts/1 res=success'`},
"srv", "USER_LOGIN", "info", "4242: login via /usr/sbin/sshd from 10.0.0.5 on /dev/pts/1 → success",
map[string]string{"user": "4242"}},
{"no result", []string{`type=CONFIG_CHANGE msg=audit(1759572005.000:4290): auid=1000 ses=3 op=add_rule key="watch" list=4`},
"srv", "CONFIG_CHANGE", "info", "cedric: config_change add_rule [watch]",
map[string]string{"res": ""}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
ev := auditEv(t, tt.lines...)
e := auditEntry(ev, &userNames{path: passwd}, "srv", now)
if e.Host != tt.host || e.App != tt.app || e.Severity != tt.sev || e.SourceType != "audit" || !e.Wait {
t.Errorf("entry %+v", e)
}
if e.Message != tt.msg {
t.Errorf("message\n got %q\nwant %q", e.Message, tt.msg)
}
for k, v := range tt.fields {
if e.Extra[k] != v {
t.Errorf("%s = %q, want %q", k, e.Extra[k], v)
}
}
if e.Extra["audit_raw"] != strings.Join(tt.lines, "\n") {
t.Errorf("raw %q", e.Extra["audit_raw"])
}
if !e.Time.Equal(ev.Time) || e.Time.IsZero() {
t.Errorf("time %v", e.Time)
}
})
}
}
type entrySink struct {
mu sync.Mutex
es []*Entry
}
func (s *entrySink) add(e *Entry) {
s.mu.Lock()
s.es = append(s.es, e)
s.mu.Unlock()
}
func (s *entrySink) serials() []string {
s.mu.Lock()
defer s.mu.Unlock()
var out []string
for _, e := range s.es {
out = append(out, e.Extra["audit_serial"])
}
return out
}
func appendLines(t *testing.T, path string, lines ...string) {
t.Helper()
f, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644)
if err != nil {
t.Fatal(err)
}
defer f.Close()
for _, l := range lines {
if _, err := f.WriteString(l + "\n"); err != nil {
t.Fatal(err)
}
}
}
func TestAuditFollow(t *testing.T) {
dir := t.TempDir()
logPath := filepath.Join(dir, "audit.log")
data := filepath.Join(dir, "data")
if err := os.Mkdir(data, 0o755); err != nil {
t.Fatal(err)
}
sink := &entrySink{}
newAuditT := func() *Audit {
a := NewAudit(logPath, dir, "srv", data, 100*365*24*time.Hour, sink.add)
a.cfg.Enabled = true
return a
}
a := newAuditT()
now := time.Unix(1759572100, 0)
// Missing file: reported, then read once it exists.
a.poll(now)
if st := a.Status(); st["code"] != "not_found" || st["open"] != false {
t.Fatalf("status %v", st)
}
appendLines(t, logPath, auditExec[:3]...)
appendLines(t, logPath, auditSudo)
a.poll(now)
if got := sink.serials(); len(got) != 0 { // events may still get records
t.Fatalf("after first poll: %v", got)
}
// A partial line is not read.
f, _ := os.OpenFile(logPath, os.O_APPEND|os.O_WRONLY, 0)
f.WriteString(auditSSH[:20])
f.Close()
a.poll(now.Add(5 * time.Second))
if got := sink.serials(); len(got) != 2 || got[0] != "4242" || got[1] != "4250" {
t.Fatalf("after second poll: %v", got)
}
if a.Status()["open"] != true || a.Status()["read"] != uint64(2) {
t.Fatalf("status %v", a.Status())
}
a.saveState()
// Restart: nothing is sent twice, the partial line is read once complete.
a = newAuditT()
f, _ = os.OpenFile(logPath, os.O_APPEND|os.O_WRONLY, 0)
f.WriteString(auditSSH[20:] + "\n")
f.Close()
a.poll(now.Add(10 * time.Second))
a.poll(now.Add(20 * time.Second))
if got := sink.serials(); len(got) != 3 || got[2] != "4260" {
t.Fatalf("after restart: %v", got)
}
// Rotation: lines added to the old file before the switch are not lost.
appendLines(t, logPath, auditOpen[0])
a.poll(now.Add(21 * time.Second)) // read, still open
appendLines(t, logPath, auditOpen[1])
if err := os.Rename(logPath, logPath+".1"); err != nil {
t.Fatal(err)
}
appendLines(t, logPath, strings.Replace(auditSudo, "4250", "4300", 1))
a.poll(now.Add(22 * time.Second))
a.poll(now.Add(30 * time.Second))
got := sink.serials()
if len(got) != 5 || got[3] != "4270" || got[4] != "4300" {
t.Fatalf("after rotation: %v", got)
}
sink.mu.Lock()
if n := len(sink.es[3].Extra["audit_raw"]); n == 0 || !strings.Contains(sink.es[3].Extra["audit_raw"], "/etc/shadow") {
t.Errorf("rotated event missing its PATH record: %q", sink.es[3].Extra["audit_raw"])
}
sink.mu.Unlock()
}
func TestAuditBackfillAndConfigure(t *testing.T) {
dir := t.TempDir()
logPath := filepath.Join(dir, "audit.log")
appendLines(t, logPath, auditExec...)
appendLines(t, logPath, auditSudo)
sink := &entrySink{}
// Only events newer than the backfill window are sent on the first read.
a := NewAudit(logPath, dir, "", dir, time.Hour, sink.add)
a.cfg.Enabled = true
a.poll(time.Unix(1759572001+7200, 0))
a.emit(a.asm.flushAll())
if got := sink.serials(); len(got) != 0 {
t.Fatalf("too old: %v", got)
}
a = NewAudit(logPath, dir, "", filepath.Join(dir, "other"), time.Hour, sink.add)
if err := os.Mkdir(filepath.Join(dir, "other"), 0o755); err != nil {
t.Fatal(err)
}
a.cfg.Enabled = true
a.poll(time.Unix(1759572001+60, 0))
a.emit(a.asm.flushAll())
if got := sink.serials(); len(got) != 2 {
t.Fatalf("backfill: %v", got)
}
if e := sink.es[0]; e.Host != "localhost" {
t.Errorf("host %q", e.Host)
}
// Run picks the configuration up; turning it off resets the position.
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { a.Run(ctx); close(done) }()
if err := a.Configure(auditConfig{Enabled: false}); err != nil {
t.Fatal(err)
}
deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
if b, err := os.ReadFile(filepath.Join(dir, "other", "audit-state.json")); err == nil && strings.Contains(string(b), `"off": 0`) {
break
}
time.Sleep(10 * time.Millisecond)
}
cancel()
<-done
if st := a.Status(); st["enabled"] != false || st["defaultPath"] != logPath {
t.Fatalf("status %v", st)
}
b, _ := os.ReadFile(filepath.Join(dir, "other", "audit-state.json"))
if !strings.Contains(string(b), `"off": 0`) {
t.Fatalf("state %s", b)
}
}
+481
View File
@@ -0,0 +1,481 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"io"
"io/fs"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"syscall"
"time"
"unicode/utf8"
)
// dmesgConfig is saved in /data/dmesg.json and edited in Settings > Sources.
type dmesgConfig struct {
Enabled bool `json:"enabled"`
}
// dmesgPos is where reading resumes: the last record sent, valid for one boot
// only (the kernel numbers its records from 0 at each boot).
type dmesgPos struct {
BootID string `json:"boot_id"`
Seq uint64 `json:"seq"`
}
// Dmesg collects the kernel ring buffer of the machine hosting the stack by
// reading /dev/kmsg, the device dmesg itself reads.
type Dmesg struct {
device string // /dev/kmsg
bootIDPath string // identifies the current boot
uptimePath string // gives the boot time, to date the records
sink func(*Entry)
backfill time.Duration
cfgPath string
statePath string
retry time.Duration // delay before opening the device again after a failure
wake chan struct{}
mu sync.Mutex
cfg dmesgConfig
reset bool
pos dmesgPos
dirty bool
open bool
read uint64
lost uint64
errCode string
errDetail string
}
func NewDmesg(device, dataDir string, backfill time.Duration, sink func(*Entry)) *Dmesg {
d := &Dmesg{
device: device,
bootIDPath: "/proc/sys/kernel/random/boot_id",
uptimePath: "/proc/uptime",
sink: sink,
backfill: backfill,
cfgPath: filepath.Join(dataDir, "dmesg.json"),
statePath: filepath.Join(dataDir, "dmesg-state.json"),
retry: 10 * time.Second,
wake: make(chan struct{}, 1),
}
if b, err := os.ReadFile(d.cfgPath); err == nil {
_ = json.Unmarshal(b, &d.cfg)
}
if b, err := os.ReadFile(d.statePath); err == nil {
_ = json.Unmarshal(b, &d.pos)
}
return d
}
// kmsgSession is one opening of the device, read by its own goroutine.
type kmsgSession struct {
f *os.File
done chan struct{}
err error
stopped bool // closed on purpose: err is not a failure
}
func (s *kmsgSession) stop() {
s.stopped = true
s.f.Close() // ends a pending read (/dev/kmsg goes through the poller)
select {
case <-s.done:
case <-time.After(2 * time.Second):
}
}
// Run keeps the device open while the source is enabled.
func (d *Dmesg) Run(ctx context.Context) {
save := time.NewTicker(5 * time.Second)
defer save.Stop()
var cur *kmsgSession
var retryAt time.Time
for {
d.mu.Lock()
enabled, reset := d.cfg.Enabled, d.reset
d.reset = false
d.mu.Unlock()
if reset {
// Disabled then enabled again: start over from DMESG_BACKFILL ago
// rather than reading everything logged in between.
if cur != nil {
cur.stop()
cur = nil
}
d.mu.Lock()
d.pos, d.dirty = dmesgPos{}, true
d.mu.Unlock()
d.saveState()
retryAt = time.Time{}
}
if !enabled && cur != nil {
cur.stop()
cur = nil
d.mu.Lock()
d.open = false
d.mu.Unlock()
}
if enabled && cur == nil && !time.Now().Before(retryAt) {
var err error
if cur, err = d.start(); err != nil {
d.fail(err)
retryAt = time.Now().Add(d.retry)
}
}
var done chan struct{}
if cur != nil {
done = cur.done
}
wait := d.retry
if cur == nil && enabled {
wait = max(time.Until(retryAt), 10*time.Millisecond)
}
timer := time.NewTimer(wait)
select {
case <-ctx.Done():
timer.Stop()
if cur != nil {
cur.stop()
}
d.saveState()
return
case <-d.wake:
case <-save.C:
d.saveState()
case <-timer.C:
case <-done:
if !cur.stopped {
d.fail(cur.err)
}
cur = nil
retryAt = time.Now().Add(d.retry)
}
timer.Stop()
}
}
func (d *Dmesg) saveState() {
d.mu.Lock()
if !d.dirty {
d.mu.Unlock()
return
}
d.dirty = false
pos := d.pos
d.mu.Unlock()
if err := writeJSONFile(d.statePath, pos); err != nil {
logEvent("dmesg", 4, "saving position: %v", err)
}
}
// fail records why the device is not being read (nil error: end of a
// regular file, only seen in tests).
func (d *Dmesg) fail(err error) {
code, detail := "", ""
if err != nil && !errors.Is(err, io.EOF) {
detail = err.Error()
switch {
case errors.Is(err, fs.ErrNotExist):
code = "not_found"
case errors.Is(err, fs.ErrPermission):
code = "permission"
default:
code = "read"
}
}
d.mu.Lock()
changed := code != d.errCode
d.open, d.errCode, d.errDetail = false, code, detail
d.mu.Unlock()
if changed && code != "" {
logEvent("dmesg", 4, "cannot read the kernel messages: %v", err)
}
}
// start opens the device and starts reading it in the background.
func (d *Dmesg) start() (*kmsgSession, error) {
f, err := os.Open(d.device)
if err != nil {
return nil, err
}
bootID := readTrim(d.bootIDPath)
boot := d.bootTime(time.Now())
d.mu.Lock()
pos := d.pos
d.open, d.errCode, d.errDetail = true, "", ""
d.mu.Unlock()
// Same boot as the saved position: skip what was already sent. Otherwise
// read from DMESG_BACKFILL ago (0: only new records).
resume := pos.BootID != "" && pos.BootID == bootID
var notBefore time.Time
if !resume {
if d.backfill <= 0 {
if _, err := f.Seek(0, io.SeekEnd); err != nil {
f.Close()
return nil, err
}
} else if !boot.IsZero() {
notBefore = time.Now().Add(-d.backfill)
}
d.mu.Lock()
d.pos, d.dirty = dmesgPos{BootID: bootID}, true
d.mu.Unlock()
}
s := &kmsgSession{f: f, done: make(chan struct{})}
go func() {
defer close(s.done)
s.err = d.consume(f, bootID, boot, resume, pos.Seq, notBefore)
}()
return s, nil
}
func readTrim(path string) string {
b, err := os.ReadFile(path)
if err != nil {
return ""
}
return strings.TrimSpace(string(b))
}
// bootTime is now minus the uptime; zero when unknown (records are then dated
// when they are read).
func (d *Dmesg) bootTime(now time.Time) time.Time {
f := strings.Fields(readTrim(d.uptimePath))
if len(f) == 0 {
return time.Time{}
}
up, err := strconv.ParseFloat(f[0], 64)
if err != nil || up <= 0 {
return time.Time{}
}
return now.Add(-time.Duration(up * float64(time.Second)))
}
// consume reads records until an error. Each read of /dev/kmsg returns one
// record; several records in one read (a regular file, in tests) are split.
func (d *Dmesg) consume(r io.Reader, bootID string, boot time.Time, resume bool, after uint64, notBefore time.Time) error {
buf := make([]byte, 64<<10) // a record is at most 8 KiB
last, haveLast := after, resume
for {
n, err := r.Read(buf)
if n > 0 {
for _, raw := range splitKmsg(buf[:n]) {
rec, ok := parseKmsg(raw)
if !ok {
continue
}
if haveLast && rec.Seq <= last {
continue // already sent before a restart
}
var lost uint64
if haveLast && rec.Seq > last+1 {
lost = rec.Seq - last - 1 // overwritten in the ring buffer before being read
}
last, haveLast = rec.Seq, true
now := time.Now()
t := now
if !boot.IsZero() {
t = boot.Add(time.Duration(rec.Usec) * time.Microsecond)
}
if !notBefore.IsZero() && t.Before(notBefore) {
d.advance(bootID, rec.Seq, 0, 0)
continue
}
d.sink(kmsgEntry(rec, t, now))
d.advance(bootID, rec.Seq, 1, lost)
}
}
if err != nil {
if errors.Is(err, os.ErrClosed) {
return nil
}
if errors.Is(err, syscall.EPIPE) {
continue // records overwritten while reading: the next read goes on
}
return err
}
}
}
func (d *Dmesg) advance(bootID string, seq, read, lost uint64) {
d.mu.Lock()
d.pos, d.dirty = dmesgPos{BootID: bootID, Seq: seq}, true
d.read += read
d.lost += lost
d.mu.Unlock()
}
// kmsgRecord is one record of /dev/kmsg:
//
// prio,seq,usec,flags[,...];message
// KEY=value (optional dictionary lines)
type kmsgRecord struct {
Prio int
Seq uint64
Usec uint64
Msg string
Dict map[string]string
}
// splitKmsg cuts a buffer into records: a new record starts after a newline
// that is not followed by a space (dictionary lines start with one).
func splitKmsg(b []byte) [][]byte {
var out [][]byte
start := 0
for i := 0; i < len(b); i++ {
if b[i] == '\n' && (i+1 == len(b) || b[i+1] != ' ') {
out = append(out, b[start:i])
start = i + 1
}
}
if start < len(b) {
out = append(out, b[start:])
}
return out
}
func parseKmsg(raw []byte) (kmsgRecord, bool) {
var rec kmsgRecord
lines := bytes.Split(raw, []byte{'\n'})
head, msg, ok := bytes.Cut(lines[0], []byte{';'})
if !ok {
return rec, false
}
f := strings.Split(string(head), ",")
if len(f) < 3 {
return rec, false
}
prio, err1 := strconv.Atoi(f[0])
seq, err2 := strconv.ParseUint(f[1], 10, 64)
usec, err3 := strconv.ParseUint(f[2], 10, 64)
if err1 != nil || err2 != nil || err3 != nil || prio < 0 {
return rec, false
}
rec.Prio, rec.Seq, rec.Usec = prio, seq, usec
rec.Msg = unescapeKmsg(string(msg))
for _, l := range lines[1:] {
if k, v, ok := strings.Cut(strings.TrimPrefix(string(l), " "), "="); ok && k != "" {
if rec.Dict == nil {
rec.Dict = map[string]string{}
}
rec.Dict[k] = unescapeKmsg(v)
}
}
return rec, true
}
// unescapeKmsg decodes the \xNN escapes the kernel uses for non-printable bytes.
func unescapeKmsg(s string) string {
if !strings.Contains(s, `\x`) {
return s
}
var b strings.Builder
for i := 0; i < len(s); i++ {
if s[i] == '\\' && i+3 < len(s) && s[i+1] == 'x' {
if v, err := strconv.ParseUint(s[i+2:i+4], 16, 8); err == nil {
b.WriteByte(byte(v))
i += 3
continue
}
}
b.WriteByte(s[i])
}
return b.String()
}
// kmsgEntry turns a record into an Entry. Kernel records have the kern
// facility; programs writing to /dev/kmsg (systemd at boot…) use another
// one and prefix their name: "systemd[1]: …".
func kmsgEntry(rec kmsgRecord, t, now time.Time) *Entry {
sev := rec.Prio & 7
fac := rec.Prio >> 3
if fac >= len(facilityNames) {
fac = 1 // user
}
msg := strings.TrimRight(rec.Msg, "\n")
if !utf8.ValidString(msg) {
msg = strings.ToValidUTF8(msg, "�")
}
app, pid := "kernel", ""
if fac != 0 {
if a, p, rest, ok := splitTag(msg); ok {
app, pid, msg = a, p, rest
}
}
e := &Entry{
Time: t,
Received: now,
Host: "localhost",
App: app,
ProcID: pid,
Facility: facilityNames[fac],
Severity: severityNames[sev],
SevNum: sev,
Message: msg,
Proto: "kmsg",
SourceType: "kernel",
Wait: true,
}
if s, dev := rec.Dict["SUBSYSTEM"], rec.Dict["DEVICE"]; s != "" || dev != "" {
e.Extra = map[string]string{}
if s != "" {
e.Extra["subsystem"] = s
}
if dev != "" {
e.Extra["device"] = dev
}
}
return e
}
// splitTag cuts "name[pid]: message" or "name: message".
func splitTag(msg string) (app, pid, rest string, ok bool) {
tag, rest, found := strings.Cut(msg, ": ")
if !found || tag == "" || len(tag) > 48 || strings.ContainsAny(tag, " \t") {
return "", "", "", false
}
if i := strings.IndexByte(tag, '['); i > 0 && strings.HasSuffix(tag, "]") {
return tag[:i], tag[i+1 : len(tag)-1], rest, true
}
return tag, "", rest, true
}
// Configure saves and applies the configuration.
func (d *Dmesg) Configure(cfg dmesgConfig) error {
d.mu.Lock()
if d.cfg.Enabled && !cfg.Enabled {
d.reset = true
d.errCode, d.errDetail = "", ""
}
d.cfg = cfg
d.mu.Unlock()
select {
case d.wake <- struct{}{}:
default:
}
return writeJSONFile(d.cfgPath, cfg)
}
// Status is the state shown in Settings > Sources.
func (d *Dmesg) Status() map[string]any {
d.mu.Lock()
defer d.mu.Unlock()
return map[string]any{
"enabled": d.cfg.Enabled,
"open": d.open && d.cfg.Enabled,
"device": d.device,
"read": d.read,
"lost": d.lost,
"code": d.errCode,
"error": d.errDetail,
}
}
+265
View File
@@ -0,0 +1,265 @@
package main
import (
"context"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"testing"
"time"
)
func TestParseKmsg(t *testing.T) {
rec, ok := parseKmsg([]byte("6,339,5140900,-;NET: Registered protocol family 10\n SUBSYSTEM=net\n DEVICE=n2"))
if !ok || rec.Prio != 6 || rec.Seq != 339 || rec.Usec != 5140900 || rec.Msg != "NET: Registered protocol family 10" {
t.Fatalf("record %+v", rec)
}
if rec.Dict["SUBSYSTEM"] != "net" || rec.Dict["DEVICE"] != "n2" {
t.Fatalf("dict %+v", rec.Dict)
}
// Extra header fields (newer kernels: caller id) and a ';' in the message.
rec, ok = parseKmsg([]byte("4,12,99,c,caller=T1;a;b \\x1b[31mred\\x5c"))
if !ok || rec.Seq != 12 || rec.Msg != "a;b \x1b[31mred\\" {
t.Fatalf("record %+v", rec)
}
for _, bad := range []string{"", "no header", "6,1;short", "x,1,2,-;msg", "6,-1,2,-;msg", "-1,1,2,-;msg"} {
if _, ok := parseKmsg([]byte(bad)); ok {
t.Errorf("%q parsed", bad)
}
}
}
func TestUnescapeKmsg(t *testing.T) {
for in, want := range map[string]string{
`plain`: "plain",
`a\x0ab`: "a\nb",
`end\x4`: `end\x4`,
`bad\xzz`: `bad\xzz`,
`\x41\x42\x4`: `AB\x4`,
} {
if got := unescapeKmsg(in); got != want {
t.Errorf("%q: got %q, want %q", in, got, want)
}
}
}
func TestSplitKmsg(t *testing.T) {
got := splitKmsg([]byte("6,1,0,-;one\n SUBSYSTEM=pci\n6,2,0,-;two\n6,3,0,-;three"))
if len(got) != 3 || string(got[0]) != "6,1,0,-;one\n SUBSYSTEM=pci" || string(got[2]) != "6,3,0,-;three" {
t.Fatalf("got %q", got)
}
}
func TestKmsgEntry(t *testing.T) {
boot := time.Date(2026, 10, 4, 6, 0, 0, 0, time.UTC)
now := boot.Add(time.Hour)
rec, _ := parseKmsg([]byte("3,7,2500000,-;usb 1-1: device descriptor read/64, error -71\n SUBSYSTEM=usb\n DEVICE=c189:1"))
e := kmsgEntry(rec, boot.Add(2500*time.Millisecond), now)
if e.App != "kernel" || e.Facility != "kern" || e.Severity != "err" || e.SevNum != 3 || e.SourceType != "kernel" || e.Proto != "kmsg" || !e.Wait {
t.Fatalf("entry %+v", e)
}
if e.Message != "usb 1-1: device descriptor read/64, error -71" || e.Extra["subsystem"] != "usb" || e.Extra["device"] != "c189:1" {
t.Fatalf("entry %+v", e)
}
if !e.Time.Equal(boot.Add(2500*time.Millisecond)) || !e.Received.Equal(now) {
t.Fatalf("times %v %v", e.Time, e.Received)
}
// Written by a program (facility daemon = 3<<3 | info).
rec, _ = parseKmsg([]byte("30,8,3000000,-;systemd[1]: Started Journal Service."))
e = kmsgEntry(rec, now, now)
if e.App != "systemd" || e.ProcID != "1" || e.Facility != "daemon" || e.Severity != "info" || e.Message != "Started Journal Service." || e.Extra != nil {
t.Fatalf("entry %+v", e)
}
// Facility out of range.
rec, _ = parseKmsg([]byte("255,9,0,-;odd"))
if e = kmsgEntry(rec, now, now); e.Facility != "user" || e.Severity != "debug" {
t.Fatalf("entry %+v", e)
}
}
// fakeHost writes a kmsg file, a boot id and an uptime in a temp dir.
func fakeHost(t *testing.T, uptime float64, records ...string) (*Dmesg, *[]*Entry, *sync.Mutex) {
t.Helper()
dir := t.TempDir()
write := func(name, s string) string {
p := filepath.Join(dir, name)
if err := os.WriteFile(p, []byte(s), 0o644); err != nil {
t.Fatal(err)
}
return p
}
var mu sync.Mutex
var got []*Entry
d := NewDmesg(write("kmsg", strings.Join(records, "\n")+"\n"), filepath.Join(dir, "data"), time.Hour, func(e *Entry) {
mu.Lock()
got = append(got, e)
mu.Unlock()
})
d.bootIDPath = write("boot_id", "b1\n")
d.uptimePath = write("uptime", strconv.FormatFloat(uptime, 'f', 2, 64)+" 1234.00\n")
d.retry = 20 * time.Millisecond
return d, &got, &mu
}
func TestDmesgBackfillAndResume(t *testing.T) {
// Booted 3 h ago: the record at 1 s is outside the 1 h backfill window,
// the ones at 2.5 h and 2.9 h are inside.
d, got, mu := fakeHost(t, 3*3600,
"6,1,1000000,-;Linux version 6.12",
"4,2,9000000000,-;eth0: link down",
"6,3,10440000000,-;eth0: link up",
)
s, err := d.start()
if err != nil {
t.Fatal(err)
}
<-s.done
mu.Lock()
if len(*got) != 2 || (*got)[0].Message != "eth0: link down" || (*got)[1].Message != "eth0: link up" {
t.Fatalf("got %+v", *got)
}
if age := time.Since((*got)[1].Time); age < 5*time.Minute || age > 7*time.Minute {
t.Fatalf("record dated %v ago", age)
}
mu.Unlock()
d.saveState()
// Restart with the saved position: same boot, nothing is sent again,
// and a new record is.
f, _ := os.OpenFile(d.device, os.O_APPEND|os.O_WRONLY, 0)
_, _ = f.WriteString("6,6,10500000000,-;eth0: renamed\n")
f.Close()
d2 := NewDmesg(d.device, filepath.Dir(d.statePath), time.Hour, d.sink)
d2.bootIDPath, d2.uptimePath = d.bootIDPath, d.uptimePath
if d2.pos != (dmesgPos{BootID: "b1", Seq: 3}) {
t.Fatalf("saved position %+v", d2.pos)
}
s, err = d2.start()
if err != nil {
t.Fatal(err)
}
<-s.done
mu.Lock()
defer mu.Unlock()
if len(*got) != 3 || (*got)[2].Message != "eth0: renamed" || (*got)[2].App != "kernel" {
t.Fatalf("after restart %+v", (*got)[2:])
}
if st := d2.Status(); st["lost"] != uint64(2) || st["read"] != uint64(1) {
t.Fatalf("status %+v", st) // records 4 and 5 were overwritten
}
}
func TestDmesgNewBoot(t *testing.T) {
d, got, mu := fakeHost(t, 60, "6,1,1000000,-;booted")
d.pos = dmesgPos{BootID: "old", Seq: 500}
s, err := d.start()
if err != nil {
t.Fatal(err)
}
<-s.done
mu.Lock()
defer mu.Unlock()
if len(*got) != 1 || d.pos != (dmesgPos{BootID: "b1", Seq: 1}) {
t.Fatalf("got %d entries, position %+v", len(*got), d.pos)
}
}
func TestDmesgNoBackfill(t *testing.T) {
d, got, mu := fakeHost(t, 60, "6,1,1000000,-;old")
d.backfill = 0
s, err := d.start()
if err != nil {
t.Fatal(err)
}
<-s.done
mu.Lock()
defer mu.Unlock()
if len(*got) != 0 {
t.Fatalf("got %+v", *got)
}
}
func TestDmesgRun(t *testing.T) {
d, got, mu := fakeHost(t, 60, "6,1,1000000,-;first", "6,2,2000000,-;second")
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() {
d.Run(ctx)
close(done)
}()
if err := d.Configure(dmesgConfig{Enabled: true}); err != nil {
t.Fatal(err)
}
deadline := time.Now().Add(5 * time.Second)
for {
mu.Lock()
n := len(*got)
mu.Unlock()
if n >= 2 {
break
}
if time.Now().After(deadline) {
t.Fatalf("got %d entries", n)
}
time.Sleep(10 * time.Millisecond)
}
// The file is re-opened after each end of file: still no duplicate.
time.Sleep(100 * time.Millisecond)
cancel()
<-done
mu.Lock()
defer mu.Unlock()
if len(*got) != 2 {
t.Fatalf("got %d entries", len(*got))
}
b, err := os.ReadFile(d.statePath)
if err != nil || !strings.Contains(string(b), `"seq": 2`) {
t.Fatalf("state %s %v", b, err)
}
b, _ = os.ReadFile(d.cfgPath)
if !strings.Contains(string(b), `"enabled": true`) {
t.Fatalf("config %s", b)
}
}
func TestDmesgMissingDevice(t *testing.T) {
d := NewDmesg(filepath.Join(t.TempDir(), "kmsg"), t.TempDir(), time.Hour, func(*Entry) {})
d.retry = 20 * time.Millisecond
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() {
d.Run(ctx)
close(done)
}()
_ = d.Configure(dmesgConfig{Enabled: true})
deadline := time.Now().Add(5 * time.Second)
for d.Status()["code"] != "not_found" {
if time.Now().After(deadline) {
t.Fatalf("status %+v", d.Status())
}
time.Sleep(10 * time.Millisecond)
}
// Turning it off clears the error.
_ = d.Configure(dmesgConfig{Enabled: false})
if st := d.Status(); st["code"] != "" || st["enabled"] != false {
t.Fatalf("status %+v", st)
}
cancel()
<-done
}
func TestDmesgPermission(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root reads everything")
}
p := filepath.Join(t.TempDir(), "kmsg")
_ = os.WriteFile(p, []byte("6,1,0,-;x\n"), 0o000)
d := NewDmesg(p, t.TempDir(), time.Hour, func(*Entry) {})
_, err := d.start()
d.fail(err)
if st := d.Status(); st["code"] != "permission" {
t.Fatalf("status %+v", st)
}
}
+19
View File
@@ -37,13 +37,32 @@ services:
DOCKER_HOST: tcp://docker-proxy:2375 # lecture seul de l'API Docker DOCKER_HOST: tcp://docker-proxy:2375 # lecture seul de l'API Docker
DOCKER_BACKFILL: ${DOCKER_BACKFILL:-1h} DOCKER_BACKFILL: ${DOCKER_BACKFILL:-1h}
HOST_LOGS_BACKFILL: ${HOST_LOGS_BACKFILL:-1h} # historique lu a l'activation des logs systeme de l'hote HOST_LOGS_BACKFILL: ${HOST_LOGS_BACKFILL:-1h} # historique lu a l'activation des logs systeme de l'hote
DMESG_BACKFILL: ${DMESG_BACKFILL:-1h} # historique lu a l'activation des messages du noyau (dmesg)
AUDIT_BACKFILL: ${AUDIT_BACKFILL:-1h} # historique lu a l'activation de l'audit Linux (auditd)
AUDIT_HOST: ${AUDIT_HOST:-} # nom d'hote des evenements d'audit (vide : node= des enregistrements, sinon /host/etc/hostname)
group_add: group_add:
- "${HOST_LOGS_GID:-4}" # groupe autorise a lire les logs de l'hote (4 = adm sur Debian/Ubuntu) - "${HOST_LOGS_GID:-4}" # groupe autorise a lire les logs de l'hote (4 = adm sur Debian/Ubuntu)
# messages du noyau de l'hote (dmesg), source a activer dans Reglages > Sources :
# decommenter devices pour donner /dev/kmsg (lecture seule) au conteneur
# devices:
# - /dev/kmsg:/dev/kmsg:r
# si l'hote a kernel.dmesg_restrict=1 (defaut Debian/Ubuntu), soit sur l'hote :
# sysctl -w kernel.dmesg_restrict=0 (et kernel.dmesg_restrict = 0 dans /etc/sysctl.d/ pour le garder)
# soit lancer le conteneur en root avec la capacite SYSLOG (decommenter les deux) :
# user: "0:0"
# cap_add:
# - SYSLOG
volumes: volumes:
- logstream-data:/data # tags.json, docker.json (les choix des conteneurs) - logstream-data:/data # tags.json, docker.json (les choix des conteneurs)
# logs systeme de l'hote, en lecture seule (source a activer dans Reglages > Sources) # logs systeme de l'hote, en lecture seule (source a activer dans Reglages > Sources)
- /var/log:/host/var/log:ro # journal systemd persistant et fichiers texte - /var/log:/host/var/log:ro # journal systemd persistant et fichiers texte
- /run/log/journal:/host/run/log/journal:ro # journal systemd volatile - /run/log/journal:/host/run/log/journal:ro # journal systemd volatile
# audit Linux (auditd), source a activer dans Reglages > Sources ; le fichier doit etre
# lisible par l'UID 10001 : log_group = adm dans /etc/audit/auditd.conf (voir le README)
# - /var/log/audit:/var/log/audit:ro
# noms des utilisateurs et de l'hote pour les evenements d'audit (optionnel)
# - /etc/passwd:/host/etc/passwd:ro
# - /etc/hostname:/host/etc/hostname:ro
# logo de la page de connexion, avec LOGIN_LOGO=/config/logo.png dans .env # logo de la page de connexion, avec LOGIN_LOGO=/config/logo.png dans .env
# - ./logo.png:/config/logo.png:ro # - ./logo.png:/config/logo.png:ro
# prereglages de tags personnalises, avec PRESETS_FILE=/config/presets.json dans .env # prereglages de tags personnalises, avec PRESETS_FILE=/config/presets.json dans .env
+4 -4
View File
@@ -242,7 +242,7 @@ func (m *DockerManager) setError(err error) {
m.mu.Lock() m.mu.Lock()
defer m.mu.Unlock() defer m.mu.Unlock()
if m.lastErr != err.Error() { if m.lastErr != err.Error() {
log.Printf("docker: %v", err) logEvent("docker", 3, "Docker API unreachable: %v", err)
} }
m.connected = false m.connected = false
m.lastErr = err.Error() m.lastErr = err.Error()
@@ -290,7 +290,7 @@ func (m *DockerManager) sync(ctx context.Context) {
m.mu.Lock() m.mu.Lock()
defer m.mu.Unlock() defer m.mu.Unlock()
if !m.connected { if !m.connected {
log.Printf("docker: connected, %d containers", len(list)) logEvent("docker", 6, "connected to the Docker API, %d containers", len(list))
} }
m.connected, m.lastErr = true, "" m.connected, m.lastErr = true, ""
seen := map[string]bool{} seen := map[string]bool{}
@@ -371,7 +371,7 @@ func (m *DockerManager) follow(ctx context.Context, c DockerContainer) {
if ctx.Err() != nil { if ctx.Err() != nil {
return return
} }
log.Printf("docker %s: %v", c.Name, err) logEvent("docker", 4, "container %s: cannot inspect it: %v", c.Name, err)
sleepCtx(ctx, 5*time.Second) sleepCtx(ctx, 5*time.Second)
continue continue
} }
@@ -383,7 +383,7 @@ func (m *DockerManager) follow(ctx context.Context, c DockerContainer) {
return return
} }
if err != nil && !errors.Is(err, io.EOF) && !errors.Is(err, io.ErrUnexpectedEOF) { if err != nil && !errors.Is(err, io.EOF) && !errors.Is(err, io.ErrUnexpectedEOF) {
log.Printf("docker %s: %v", c.Name, err) logEvent("docker", 4, "container %s: log stream lost, reconnecting: %v", c.Name, err)
} }
sleepCtx(ctx, time.Second) sleepCtx(ctx, time.Second)
} }
+1 -1
View File
@@ -2,7 +2,7 @@
# Préréglages de tags de couleur # Préréglages de tags de couleur
Dans **Paramètres › Filtres**, le menu **+ Préréglage…** ajoute d'un clic un groupe de tags de Dans **Paramètres › Motifs de recherche**, le menu **+ Préréglage…** ajoute d'un clic un groupe de tags de
couleur tout faits. Les tags ajoutés sont des tags ordinaires : vous pouvez changer leur couleur, couleur tout faits. Les tags ajoutés sont des tags ordinaires : vous pouvez changer leur couleur,
leur motif ou leurs options, ou les supprimer. Un tag dont le motif est déjà dans la liste n'est leur motif ou leurs options, ou les supprimer. Un tag dont le motif est déjà dans la liste n'est
pas ajouté en double. pas ajouté en double.
+1 -1
View File
@@ -2,7 +2,7 @@
# Color tag presets # Color tag presets
In **Settings › Filters**, the **+ Preset…** menu adds a group of ready-made color tags in one In **Settings › Patterns**, the **+ Preset…** menu adds a group of ready-made color tags in one
click. Added tags are ordinary tags: you can change their color, pattern or options, or delete click. Added tags are ordinary tags: you can change their color, pattern or options, or delete
them. A tag whose pattern is already in the list is not added twice. them. A tag whose pattern is already in the list is not added twice.
+5 -2
View File
@@ -7,7 +7,6 @@ import (
"errors" "errors"
"io" "io"
"io/fs" "io/fs"
"log"
"os" "os"
"path/filepath" "path/filepath"
"sort" "sort"
@@ -116,14 +115,18 @@ func (h *HostLogs) saveState() {
} }
h.dirty = false h.dirty = false
if err := writeJSONFile(h.statePath, h.pos); err != nil { if err := writeJSONFile(h.statePath, h.pos); err != nil {
log.Printf("host logs: saving positions: %v", err) logEvent("host", 4, "saving positions: %v", err)
} }
} }
func (h *HostLogs) setStatus(mode string, files int, code, detail string) { func (h *HostLogs) setStatus(mode string, files int, code, detail string) {
h.mu.Lock() h.mu.Lock()
changed := code != h.errCode
h.mode, h.files, h.errCode, h.errDetail = mode, files, code, detail h.mode, h.files, h.errCode, h.errDetail = mode, files, code, detail
h.mu.Unlock() h.mu.Unlock()
if changed && code != "" && code != "empty" {
logEvent("host", 4, "cannot read the host system logs (%s): %s", code, detail)
}
} }
// scan reads what was added since the previous poll. // scan reads what was added since the previous poll.
+40 -3
View File
@@ -38,6 +38,11 @@ type config struct {
backfill time.Duration backfill time.Duration
hostRoot string hostRoot string
hostFill time.Duration hostFill time.Duration
kmsgDevice string
kmsgFill time.Duration
auditLog string
auditHost string
auditFill time.Duration
batchSize int batchSize int
queueSize int queueSize int
flushEvery time.Duration flushEvery time.Duration
@@ -115,6 +120,11 @@ func main() {
backfill: getenvDuration("DOCKER_BACKFILL", time.Hour), backfill: getenvDuration("DOCKER_BACKFILL", time.Hour),
hostRoot: getenv("HOST_LOGS_ROOT", "/host"), hostRoot: getenv("HOST_LOGS_ROOT", "/host"),
hostFill: getenvDuration("HOST_LOGS_BACKFILL", time.Hour), hostFill: getenvDuration("HOST_LOGS_BACKFILL", time.Hour),
kmsgDevice: getenv("DMESG_DEVICE", "/dev/kmsg"),
kmsgFill: getenvDuration("DMESG_BACKFILL", time.Hour),
auditLog: getenv("AUDIT_LOG", "/var/log/audit/audit.log"),
auditHost: os.Getenv("AUDIT_HOST"),
auditFill: getenvDuration("AUDIT_BACKFILL", time.Hour),
batchSize: getenvInt("BATCH_SIZE", 1000), batchSize: getenvInt("BATCH_SIZE", 1000),
queueSize: getenvInt("QUEUE_SIZE", 100000), queueSize: getenvInt("QUEUE_SIZE", 100000),
flushEvery: time.Duration(getenvInt("FLUSH_MS", 1000)) * time.Millisecond, flushEvery: time.Duration(getenvInt("FLUSH_MS", 1000)) * time.Millisecond,
@@ -134,14 +144,16 @@ func main() {
var spool *Spool var spool *Spool
if cfg.spoolMax > 0 { if cfg.spoolMax > 0 {
if spool, err = OpenSpool(filepath.Join(cfg.dataDir, "spool"), cfg.spoolMax); err != nil { if spool, err = OpenSpool(filepath.Join(cfg.dataDir, "spool"), cfg.spoolMax); err != nil {
log.Printf("disk buffer disabled: %v", err) log.Printf("disk buffer disabled: %v", err) // before the self-logs exist
spool = nil spool = nil
} }
} }
store := NewStore(cfg.vlogsURL, cfg.batchSize, cfg.queueSize, cfg.flushEvery, spool) store := NewStore(cfg.vlogsURL, cfg.batchSize, cfg.queueSize, cfg.flushEvery, spool)
// The store stops after everything else, so the stop event is still stored.
storeCtx, stopStore := context.WithCancel(context.Background())
storeDone := make(chan struct{}) storeDone := make(chan struct{})
go func() { go func() {
store.Run(ctx) store.Run(storeCtx)
close(storeDone) close(storeDone)
}() }()
@@ -159,6 +171,20 @@ func main() {
hub.Publish(e) hub.Publish(e)
}) })
} }
// Internal events of LogStream (host=logstream), WARNING and above unless
// changed in Settings > Sources.
selfLog = NewSelfLog(cfg.dataDir, func(e *Entry) {
store.Enqueue(e)
hub.Publish(e)
})
selfCtx, stopSelf := context.WithCancel(context.Background())
selfDone := make(chan struct{})
go func() {
selfLog.Run(selfCtx)
close(selfDone)
}()
logEvent("main", 5, "LogStream started")
tcpMaxConns = getenvInt("SYSLOG_TCP_MAX_CONNS", tcpMaxConns) tcpMaxConns = getenvInt("SYSLOG_TCP_MAX_CONNS", tcpMaxConns)
if d := getenvDuration("SYSLOG_TCP_IDLE", tcpIdle); d > 0 { if d := getenvDuration("SYSLOG_TCP_IDLE", tcpIdle); d > 0 {
tcpIdle = d tcpIdle = d
@@ -176,7 +202,7 @@ func main() {
if cfg.dockerLogs { if cfg.dockerLogs {
dm, err := NewDockerManager(cfg.dockerHost, cfg.dataDir, cfg.backfill, sink) dm, err := NewDockerManager(cfg.dockerHost, cfg.dataDir, cfg.backfill, sink)
if err != nil { if err != nil {
log.Printf("docker logs disabled: %v", err) logEvent("docker", 3, "Docker logs disabled: %v", err)
} else { } else {
go dm.Run(ctx) go dm.Run(ctx)
api.docker = dm api.docker = dm
@@ -186,6 +212,13 @@ func main() {
// System logs of the host (journal or /var/log), off until enabled in Settings > Sources. // System logs of the host (journal or /var/log), off until enabled in Settings > Sources.
api.host = NewHostLogs(cfg.hostRoot, cfg.dataDir, cfg.hostFill, sink) api.host = NewHostLogs(cfg.hostRoot, cfg.dataDir, cfg.hostFill, sink)
go api.host.Run(ctx) go api.host.Run(ctx)
// Kernel messages of the host (/dev/kmsg), off until enabled in Settings > Sources.
api.dmesg = NewDmesg(cfg.kmsgDevice, cfg.dataDir, cfg.kmsgFill, sink)
go api.dmesg.Run(ctx)
// Linux audit log of the host (auditd), off until enabled in Settings > Sources.
api.audit = NewAudit(cfg.auditLog, cfg.hostRoot, cfg.auditHost, cfg.dataDir, cfg.auditFill, sink)
go api.audit.Run(ctx)
api.selfLog = selfLog
api.Routes(mux) api.Routes(mux)
mux.Handle("GET /", http.FileServer(http.FS(static))) mux.Handle("GET /", http.FileServer(http.FS(static)))
@@ -222,6 +255,10 @@ func main() {
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Fatalf("http: %v", err) log.Fatalf("http: %v", err)
} }
logEvent("main", 5, "LogStream stopping")
stopSelf()
<-selfDone
stopStore()
<-storeDone <-storeDone
log.Println("shutdown complete") log.Println("shutdown complete")
} }
+2 -2
View File
@@ -119,10 +119,10 @@ func (f Filter) filterExpr() string {
parts = append(parts, "app:="+strconv.Quote(f.App)) parts = append(parts, "app:="+strconv.Quote(f.App))
} }
switch f.Source { switch f.Source {
case "docker", "host": case "docker", "host", "kernel", "audit", "logstream":
parts = append(parts, `source_type:=`+strconv.Quote(f.Source)) parts = append(parts, `source_type:=`+strconv.Quote(f.Source))
case "syslog": // also matches logs stored before source_type existed case "syslog": // also matches logs stored before source_type existed
parts = append(parts, `!(source_type:in("docker","host"))`) parts = append(parts, `!(source_type:in("docker","host","kernel","audit","logstream"))`)
} }
if f.Severity >= 0 && f.Severity < 7 { if f.Severity >= 0 && f.Severity < 7 {
names := make([]string, 0, 8) names := make([]string, 0, 8)
+187
View File
@@ -0,0 +1,187 @@
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"path/filepath"
"sync"
"time"
)
// selfLogConfig is saved in /data/selflog.json and edited in Settings > Sources.
type selfLogConfig struct {
Enabled bool `json:"enabled"`
MinLevel string `json:"min_level"` // severity name: "err", "warning", "notice", "info"…
}
// SelfLog turns the internal events of LogStream (storage errors, listener
// failures, sources that stop…) into log entries, with host=logstream and
// app=component. Events go through a bounded queue and a per-message
// throttle: an event about a failing Store is itself stored through that
// Store, so it must never be able to loop or pile up.
type SelfLog struct {
cfgPath string
sink func(*Entry)
queue chan *Entry
interval time.Duration // one event per component and message format per interval
mu sync.Mutex
cfg selfLogConfig
last map[string]time.Time
suppressed map[string]int
emitted uint64
dropped uint64
throttled uint64
}
// selfLog receives the events of logEvent; nil (tests) only writes them to
// the process log.
var selfLog *SelfLog
func NewSelfLog(dataDir string, sink func(*Entry)) *SelfLog {
s := &SelfLog{
cfgPath: filepath.Join(dataDir, "selflog.json"),
sink: sink,
queue: make(chan *Entry, 256),
interval: 30 * time.Second,
cfg: selfLogConfig{Enabled: true, MinLevel: "warning"},
last: map[string]time.Time{},
suppressed: map[string]int{},
}
if b, err := os.ReadFile(s.cfgPath); err == nil {
var cfg selfLogConfig
if json.Unmarshal(b, &cfg) == nil {
s.cfg = normSelfLogConfig(cfg)
}
}
return s
}
func normSelfLogConfig(cfg selfLogConfig) selfLogConfig {
if sevNum(cfg.MinLevel) < 0 {
cfg.MinLevel = "warning"
}
return cfg
}
func sevNum(name string) int {
for i, n := range severityNames {
if n == name {
return i
}
}
return -1
}
// logEvent writes an internal event to the process log and, when the
// self-logs source is on and the level is high enough, emits it as an entry.
// sev is a syslog severity (3 err, 4 warning, 5 notice, 6 info).
func logEvent(component string, sev int, format string, args ...any) {
msg := fmt.Sprintf(format, args...)
log.Printf("%s: %s", component, msg)
if s := selfLog; s != nil {
s.Emit(component, sev, format, msg, time.Now())
}
}
// Emit queues an event without ever blocking. The same component and message
// format are let through once per interval; the next one says how many were
// left out in between.
func (s *SelfLog) Emit(component string, sev int, format, msg string, now time.Time) {
sev = min(max(sev, 0), 7)
key := component + "\x00" + format
s.mu.Lock()
if !s.cfg.Enabled || sev > sevNum(s.cfg.MinLevel) {
s.mu.Unlock()
return
}
if t, ok := s.last[key]; ok && now.Sub(t) < s.interval {
s.suppressed[key]++
s.throttled++
s.mu.Unlock()
return
}
s.last[key] = now
if n := s.suppressed[key]; n > 0 {
msg += fmt.Sprintf(" (%d similar events not shown)", n)
delete(s.suppressed, key)
}
if len(s.last) > 1000 { // formats are constants: only a bug could get here
clear(s.last)
}
s.mu.Unlock()
e := &Entry{
Time: now,
Received: now,
Host: "logstream",
App: component,
Facility: "syslog",
Severity: severityNames[sev],
SevNum: sev,
Message: msg,
Proto: "internal",
SourceType: "logstream",
// Wait stays false: when the Store queue is full, the event is dropped
// instead of blocking the component that reports it.
}
select {
case s.queue <- e:
default:
s.mu.Lock()
s.dropped++
s.mu.Unlock()
}
}
// Run hands the queued events to the sink until the context ends, then
// delivers what is left (the stop event among them).
func (s *SelfLog) Run(ctx context.Context) {
for {
select {
case e := <-s.queue:
s.deliver(e)
case <-ctx.Done():
for {
select {
case e := <-s.queue:
s.deliver(e)
default:
return
}
}
}
}
}
func (s *SelfLog) deliver(e *Entry) {
s.sink(e)
s.mu.Lock()
s.emitted++
s.mu.Unlock()
}
// Configure saves and applies the configuration.
func (s *SelfLog) Configure(cfg selfLogConfig) error {
cfg = normSelfLogConfig(cfg)
s.mu.Lock()
s.cfg = cfg
s.mu.Unlock()
return writeJSONFile(s.cfgPath, cfg)
}
// Status is the state shown in Settings > Sources.
func (s *SelfLog) Status() map[string]any {
s.mu.Lock()
defer s.mu.Unlock()
return map[string]any{
"enabled": s.cfg.Enabled,
"min_level": s.cfg.MinLevel,
"emitted": s.emitted,
"dropped": s.dropped,
"throttled": s.throttled,
}
}
+102
View File
@@ -0,0 +1,102 @@
package main
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func TestSelfLogEmit(t *testing.T) {
now := time.Unix(1759572000, 0)
tests := []struct {
name string
cfg selfLogConfig
events []int // severities, one second apart, same message format
want int // entries queued
suffix string
}{
{"warning and above", selfLogConfig{Enabled: true, MinLevel: "warning"}, []int{3}, 1, ""},
{"below the level", selfLogConfig{Enabled: true, MinLevel: "warning"}, []int{5, 6}, 0, ""},
{"info level", selfLogConfig{Enabled: true, MinLevel: "info"}, []int{6}, 1, ""},
{"disabled", selfLogConfig{Enabled: false, MinLevel: "debug"}, []int{0}, 0, ""},
{"bad level falls back to warning", normSelfLogConfig(selfLogConfig{Enabled: true, MinLevel: "loud"}), []int{4, 5}, 1, ""},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
s := NewSelfLog(t.TempDir(), func(*Entry) {})
s.cfg = tt.cfg
for i, sev := range tt.events {
s.Emit("store", sev, "fmt %d", "msg", now.Add(time.Duration(i)*time.Second))
}
if len(s.queue) != tt.want {
t.Fatalf("%d queued, want %d", len(s.queue), tt.want)
}
})
}
// Same component and format: once per interval, then a count of the others.
s := NewSelfLog(t.TempDir(), func(*Entry) {})
for i := 0; i < 5; i++ {
s.Emit("store", 3, "%d messages lost", "x", now.Add(time.Duration(i)*time.Second))
}
s.Emit("syslog", 3, "%d messages lost", "y", now) // another component
s.Emit("store", 3, "%d messages lost", "z", now.Add(s.interval))
if len(s.queue) != 3 {
t.Fatalf("%d queued", len(s.queue))
}
<-s.queue
if e := <-s.queue; e.App != "syslog" || e.Host != "logstream" || e.SourceType != "logstream" || e.Wait {
t.Fatalf("entry %+v", e)
}
if e := <-s.queue; e.Message != "z (4 similar events not shown)" || e.Severity != "err" {
t.Fatalf("message %q", e.Message)
}
if st := s.Status(); st["throttled"] != uint64(4) {
t.Fatalf("status %v", st)
}
// A full queue drops events instead of blocking.
s = NewSelfLog(t.TempDir(), func(*Entry) {})
for i := 0; i < cap(s.queue)+10; i++ {
s.Emit("store", 3, "event %d", "x", now.Add(time.Duration(i)*s.interval))
}
if st := s.Status(); st["dropped"] != uint64(10) {
t.Fatalf("status %v", st)
}
}
func TestSelfLogRunAndConfigure(t *testing.T) {
dir := t.TempDir()
sink := &entrySink{}
s := NewSelfLog(dir, sink.add)
if err := s.Configure(selfLogConfig{Enabled: true, MinLevel: "notice"}); err != nil {
t.Fatal(err)
}
if b, _ := os.ReadFile(filepath.Join(dir, "selflog.json")); !strings.Contains(string(b), `"notice"`) {
t.Fatalf("saved %s", b)
}
if s2 := NewSelfLog(dir, nil); s2.cfg.MinLevel != "notice" {
t.Fatalf("reloaded %+v", s2.cfg)
}
old := selfLog
selfLog = s
defer func() { selfLog = old }()
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { s.Run(ctx); close(done) }()
logEvent("main", 5, "LogStream started")
logEvent("main", 6, "too verbose")
cancel()
<-done
logEvent("main", 5, "LogStream stopping") // queued after Run: not lost if Run drains it
sink.mu.Lock()
n := len(sink.es)
sink.mu.Unlock()
if n != 1 || s.Status()["emitted"] != uint64(1) {
t.Fatalf("%d entries, status %v", n, s.Status())
}
}
+31 -10
View File
@@ -8,7 +8,6 @@ import (
"errors" "errors"
"fmt" "fmt"
"io" "io"
"log"
"net/http" "net/http"
"net/url" "net/url"
"strings" "strings"
@@ -34,6 +33,7 @@ type Store struct {
ingested atomic.Int64 ingested atomic.Int64
dropped atomic.Int64 dropped atomic.Int64
lastErr atomic.Value // string lastErr atomic.Value // string
failing atomic.Bool // VictoriaLogs refused or missed the last send
} }
func NewStore(base string, batchSize, queueSize int, flushEvery time.Duration, spool *Spool) *Store { func NewStore(base string, batchSize, queueSize int, flushEvery time.Duration, spool *Spool) *Store {
@@ -133,7 +133,7 @@ func (s *Store) store(ctx context.Context, batch []*Entry) {
if err != nil { if err != nil {
s.dropped.Add(int64(len(batch))) s.dropped.Add(int64(len(batch)))
s.lastErr.Store(err.Error()) s.lastErr.Store(err.Error())
log.Printf("victorialogs: %d messages dropped: %v", len(batch), err) logEvent("store", 3, "%d messages lost: %v", len(batch), err)
return return
} }
for _, e := range batch { for _, e := range batch {
@@ -153,9 +153,10 @@ func (s *Store) save(ctx context.Context, body []byte, lines int) error {
} }
if err = s.post(ctx, body); err == nil { if err = s.post(ctx, body); err == nil {
s.ingested.Add(int64(lines)) s.ingested.Add(int64(lines))
s.lastErr.Store("") s.sent()
return nil return nil
} }
s.sendFailed(err)
} }
return err return err
} }
@@ -164,10 +165,10 @@ func (s *Store) save(ctx context.Context, body []byte, lines int) error {
if waiting, _ := s.spool.Pending(); waiting == 0 { if waiting, _ := s.spool.Pending(); waiting == 0 {
if postErr = s.post(ctx, body); postErr == nil { if postErr = s.post(ctx, body); postErr == nil {
s.ingested.Add(int64(lines)) s.ingested.Add(int64(lines))
s.lastErr.Store("") s.sent()
return nil return nil
} }
s.lastErr.Store(postErr.Error()) s.sendFailed(postErr)
} }
err := s.spool.Write(body, lines) err := s.spool.Write(body, lines)
if err == nil { if err == nil {
@@ -192,13 +193,13 @@ func (s *Store) save(ctx context.Context, body []byte, lines int) error {
// an increasing pause (up to 30 s) while it keeps failing. // an increasing pause (up to 30 s) while it keeps failing.
func (s *Store) replay(ctx context.Context) { func (s *Store) replay(ctx context.Context) {
if n, size := s.spool.Pending(); n > 0 { if n, size := s.spool.Pending(); n > 0 {
log.Printf("spool: %d messages (%d bytes) waiting from a previous run", n, size) logEvent("store", 5, "disk spool: %d messages (%d bytes) waiting from a previous run", n, size)
} }
backoff := time.Second backoff := time.Second
for { for {
f, body, ok, err := s.spool.Oldest() f, body, ok, err := s.spool.Oldest()
if err != nil { if err != nil {
log.Printf("spool: %v", err) logEvent("store", 4, "disk spool: %v", err)
} }
if !ok { if !ok {
select { select {
@@ -214,17 +215,17 @@ func (s *Store) replay(ctx context.Context) {
case err == nil: case err == nil:
s.spool.Remove(f) s.spool.Remove(f)
s.ingested.Add(f.lines) s.ingested.Add(f.lines)
s.lastErr.Store("") s.sent()
backoff = time.Second backoff = time.Second
continue continue
case isRejected(err): case isRejected(err):
// VictoriaLogs refuses the data itself: sending it again would not help. // VictoriaLogs refuses the data itself: sending it again would not help.
s.spool.Remove(f) s.spool.Remove(f)
s.dropped.Add(f.lines) s.dropped.Add(f.lines)
log.Printf("spool: %d messages refused by victorialogs: %v", f.lines, err) logEvent("store", 3, "disk spool: %d messages refused by VictoriaLogs and lost: %v", f.lines, err)
continue continue
} }
s.lastErr.Store(err.Error()) s.sendFailed(err)
if !sleepCtx(ctx, backoff) { if !sleepCtx(ctx, backoff) {
return return
} }
@@ -232,6 +233,26 @@ func (s *Store) replay(ctx context.Context) {
} }
} }
// sendFailed and sent report when VictoriaLogs stops and starts taking the
// batches again, once per change.
func (s *Store) sendFailed(err error) {
s.lastErr.Store(err.Error())
if !s.failing.Swap(true) {
if s.spool != nil {
logEvent("store", 4, "VictoriaLogs unreachable, batches are kept in the disk spool: %v", err)
} else {
logEvent("store", 4, "VictoriaLogs unreachable, retrying: %v", err)
}
}
}
func (s *Store) sent() {
s.lastErr.Store("")
if s.failing.Swap(false) {
logEvent("store", 5, "VictoriaLogs reachable again")
}
}
func encodeBatch(batch []*Entry) ([]byte, error) { func encodeBatch(batch []*Entry) ([]byte, error) {
var buf bytes.Buffer var buf bytes.Buffer
enc := json.NewEncoder(&buf) enc := json.NewEncoder(&buf)
+4 -4
View File
@@ -36,7 +36,7 @@ type Entry struct {
Proto string // udp or tcp Proto string // udp or tcp
HostIP string // original host value when it was an IP resolved through DNS HostIP string // original host value when it was an IP resolved through DNS
SourceType string // "syslog" or "docker" SourceType string // "syslog", "docker", "host", "kernel", "audit" or "logstream"
Extra map[string]string // additional fields (docker: container, image, …) Extra map[string]string // additional fields (docker: container, image, …)
// Not stored. Wait: the producer can be slowed down when the queue is full // Not stored. Wait: the producer can be slowed down when the queue is full
@@ -239,7 +239,7 @@ func serveUDP(ctx context.Context, pc net.PacketConn, sink func(*Entry)) {
if ctx.Err() != nil { if ctx.Err() != nil {
return return
} }
log.Printf("syslog udp: %v", err) logEvent("syslog", 4, "UDP read error: %v", err)
time.Sleep(100 * time.Millisecond) time.Sleep(100 * time.Millisecond)
continue continue
} }
@@ -262,14 +262,14 @@ func serveTCP(ctx context.Context, ln net.Listener, sink func(*Entry)) {
if ctx.Err() != nil { if ctx.Err() != nil {
return return
} }
log.Printf("syslog tcp: %v", err) logEvent("syslog", 4, "TCP accept error: %v", err)
time.Sleep(100 * time.Millisecond) time.Sleep(100 * time.Millisecond)
continue continue
} }
select { select {
case slots <- struct{}{}: case slots <- struct{}{}:
default: default:
log.Printf("syslog tcp: %d connections already open, refusing %s", tcpMaxConns, conn.RemoteAddr()) logEvent("syslog", 4, "%d TCP connections already open, refusing %s", tcpMaxConns, conn.RemoteAddr())
conn.Close() conn.Close()
continue continue
} }
+2 -2
View File
@@ -94,7 +94,7 @@ func (s *SyslogServer) apply() {
pc, err := net.ListenPacket("udp", s.addr) pc, err := net.ListenPacket("udp", s.addr)
if err != nil { if err != nil {
s.errUDP = err.Error() s.errUDP = err.Error()
log.Printf("syslog udp: %v", err) logEvent("syslog", 3, "UDP listener failed: %v", err)
} else { } else {
if uc, ok := pc.(*net.UDPConn); ok { if uc, ok := pc.(*net.UDPConn); ok {
_ = uc.SetReadBuffer(8 << 20) _ = uc.SetReadBuffer(8 << 20)
@@ -108,7 +108,7 @@ func (s *SyslogServer) apply() {
ln, err := net.Listen("tcp", s.addr) ln, err := net.Listen("tcp", s.addr)
if err != nil { if err != nil {
s.errTCP = err.Error() s.errTCP = err.Error()
log.Printf("syslog tcp: %v", err) logEvent("syslog", 3, "TCP listener failed: %v", err)
} else { } else {
s.ln = ln s.ln = ln
go serveTCP(ctx, ln, s.sink) go serveTCP(ctx, ln, s.sink)
+333 -39
View File
@@ -75,8 +75,25 @@ const I18N = {
msgTimeTitle: 'Timestamp from the message', msgTimeTitle: 'Timestamp from the message',
fHostName: 'host name (DNS)', fHostIp: 'host IP', fHostName: 'host name (DNS)', fHostIp: 'host IP',
clickHost: 'Click to filter on this host', clickApp: 'Click to filter on this app', clickHost: 'Click to filter on this host', clickApp: 'Click to filter on this app',
sourceAria: 'Source', srcAll: 'All sources', tabSources: 'Sources', sourceAria: 'Source', srcAll: 'All sources', tabSources: 'Log sources',
srcHost: 'Host system', srcHost: 'Host system',
srcKernel: 'Kernel (dmesg)',
auditTitle: 'Linux audit (auditd)',
auditEnabled: 'Collect the audit events of this machine',
auditPath: 'File',
auditOff: 'Off: the audit log of the machine hosting LogStream is not read.',
auditOk: ({ p, r }) => `Following ${p}: ${r} events since startup.`,
audit_not_found: 'Audit log not found: mount /var/log/audit in the container (see docker-compose.yml) and check the path. ',
audit_permission: 'Permission denied on the audit log: set log_group in /etc/audit/auditd.conf, or add an ACL for UID 10001 (see the README). ',
audit_read: 'Cannot read the audit log: ',
auditHelp: 'Reads the log of the Linux audit daemon (auditd): logins, sudo, watched files, syscalls matched by your rules… The records of one event are grouped in one line, with a summary (who, what, result) and the raw record below it; failures are warnings. When turned on, the last hour is read first (AUDIT_BACKFILL); rotations are followed and reading resumes where it stopped.',
selfTitle: 'LogStream internal logs',
selfEnabled: 'Record the events of LogStream itself',
selfLevel: 'Minimum level',
selfOff: 'Off: the events of LogStream only go to its container output.',
selfOk: ({ n }) => `${n} events recorded since startup (host logstream).`,
selfDropped: (n) => ` ${n} left out (repeated or too many at once).`,
selfHelp: 'Turns the events of LogStream into logs (host logstream, app = component): VictoriaLogs unreachable and back, lost batches, syslog listener errors, Docker streams lost, failing sources, start and stop. The same event is recorded at most once every 30 s, with the number of repeats.',
hostTitle: 'Host system logs', hostTitle: 'Host system logs',
hostEnabled: 'Collect the system logs of this machine', hostEnabled: 'Collect the system logs of this machine',
hostOff: 'Off: the system logs of the machine hosting LogStream are not collected.', hostOff: 'Off: the system logs of the machine hosting LogStream are not collected.',
@@ -89,6 +106,15 @@ const I18N = {
host_empty: 'No systemd journal and no log file found in /var/log.', host_empty: 'No systemd journal and no log file found in /var/log.',
host_read: 'Cannot read the host logs: ', host_read: 'Cannot read the host logs: ',
hostHelp: 'Reads the systemd journal of the host (/var/log/journal, /run/log/journal), or the text files of /var/log (syslog, messages, *.log) when there is no journal. Directories are mounted read-only. When turned on, the last hour is read first (HOST_LOGS_BACKFILL).', hostHelp: 'Reads the systemd journal of the host (/var/log/journal, /run/log/journal), or the text files of /var/log (syslog, messages, *.log) when there is no journal. Directories are mounted read-only. When turned on, the last hour is read first (HOST_LOGS_BACKFILL).',
dmesgTitle: 'Kernel messages (dmesg)',
dmesgEnabled: 'Collect the kernel messages of this machine',
dmesgOff: 'Off: the kernel messages of the machine hosting LogStream are not collected.',
dmesgOk: ({ d, r }) => `Reading ${d}: ${r} messages since startup.`,
dmesgLost: (n) => ` ${n} messages were overwritten in the kernel buffer before being read.`,
dmesg_not_found: 'Device /dev/kmsg not found in the container: uncomment its devices section in docker-compose.yml (see the README).',
dmesg_permission: 'Permission denied on /dev/kmsg: set kernel.dmesg_restrict=0 on the host, or run the container as root with the SYSLOG capability (see the README). ',
dmesg_read: 'Cannot read the kernel messages: ',
dmesgHelp: 'Reads the kernel ring buffer of the host through /dev/kmsg, like dmesg: hardware, drivers, OOM killer, firewall… When turned on, the last hour is read first (DMESG_BACKFILL); after a restart, reading resumes where it stopped.',
syslogEnabled: 'Receive syslog messages', syslogProtocols: 'Protocols', syslogPort: 'Port', syslogEnabled: 'Receive syslog messages', syslogProtocols: 'Protocols', syslogPort: 'Port',
syslogListening: ({ port, protos }) => `Listening on port ${port} (${protos}).`, syslogListening: ({ port, protos }) => `Listening on port ${port} (${protos}).`,
syslogOff: 'Syslog reception is off: syslog messages are ignored (Docker logs are still collected).', syslogOff: 'Syslog reception is off: syslog messages are ignored (Docker logs are still collected).',
@@ -116,7 +142,7 @@ const I18N = {
exportNote: (n) => `Up to ${n} logs matching the current filters, newest first. Dates use the time zone chosen in Settings.`, exportNote: (n) => `Up to ${n} logs matching the current filters, newest first. Dates use the time zone chosen in Settings.`,
exporting: 'Exporting…', exportDone: (f) => `Downloaded: ${f}`, exportError: 'Export failed: ', exporting: 'Exporting…', exportDone: (f) => `Downloaded: ${f}`, exportError: 'Export failed: ',
err_export_pipes: 'Export does not support LogsQL pipes (| …): keep only the filter part', err_export_pipes: 'Export does not support LogsQL pipes (| …): keep only the filter part',
tabLocale: 'Localization', tabFilters: 'Filters', tabInterface: 'Interface', tabData: 'Data', tabLocale: 'Localization', tabFilters: 'Patterns', tabInterface: 'Appearance', tabData: 'Database',
themeTitle: 'Theme', themeSystem: 'System', themeLight: 'Light', themeDark: 'Dark', themeTitle: 'Theme', themeSystem: 'System', themeLight: 'Light', themeDark: 'Dark',
themeHelp: 'System follows the light/dark preference of your computer or phone.', themeHelp: 'System follows the light/dark preference of your computer or phone.',
logDisplay: 'Log display', fontSize: 'Font size', logDisplay: 'Log display', fontSize: 'Font size',
@@ -150,22 +176,27 @@ const I18N = {
authOff: 'No authentication: anyone who can reach this page can read the logs and change the settings. Set AUTH_USER / AUTH_PASS or AUTH_MODE=oidc.', authOff: 'No authentication: anyone who can reach this page can read the logs and change the settings. Set AUTH_USER / AUTH_PASS or AUTH_MODE=oidc.',
readOnlyNote: 'Read-only account: these settings can only be changed by an administrator.', readOnlyNote: 'Read-only account: these settings can only be changed by an administrator.',
err_purge_confirm: 'Type PURGE to confirm', err_purge_confirm: 'Type PURGE to confirm',
liveZoomed: 'Live view is not available on a zoomed range', liveZoomed: 'Live view is only available in Stream mode',
connZoom: 'live paused (zoom)', connZoom: 'live paused (time range)',
viewAria: 'Display mode', viewStream: 'Stream', viewPeriod: 'Time range',
viewStreamTitle: 'Stream: the latest logs over a sliding duration, updated live',
viewPeriodTitle: 'Time range: the logs between a start and an end date',
perFrom: 'Start', perTo: 'End', perPrev: 'Previous time range', perNext: 'Next time range',
perOut: 'Zoom out (twice as long)',
histoTitle: 'Timeline', histoTitle: 'Timeline',
hScale: 'Scale', hLinear: 'Linear', hHeight: 'Height', hColor: 'Color', hScale: 'Scale', hLinear: 'Linear', hHeight: 'Height', hColor: 'Color',
hColSeverity: 'Severity', hColIntensity: 'Intensity', hColNone: 'None', hColSeverity: 'Severity', hColIntensity: 'Intensity', hColNone: 'None',
hRender: 'Display', hBars: 'Bars', hArea: 'Area', hRender: 'Display', hBars: 'Bars', hArea: 'Area',
hStep: 'Division', hAuto: 'Automatic', hRefresh: 'Refresh', hOff: 'Off', hRefreshAuto: 'At each new interval', hStep: 'Division', hAuto: 'Automatic', hRefresh: 'Refresh', hOff: 'Off', hRefreshAuto: 'At each new interval',
histoHelp: 'Click a bar to zoom on its interval, or drag across several. The √ scale keeps small volumes visible next to bursts. Intensity compares each interval with the median of the window: calm, burst (more than 3×), anomaly (more than 10×). In live mode, the last interval is updated as messages arrive.', histoHelp: 'Click a bar to show its interval in Time range mode, or drag across several. The √ scale keeps small volumes visible next to bursts. Intensity compares each interval with the median of the window: calm, burst (more than 3×), anomaly (more than 10×). In live mode, the last interval is updated as messages arrive.',
hLeg_err: 'error and above', hLeg_warn: 'warning', hLeg_other: 'notice, info, debug', hLeg_err: 'error and above', hLeg_warn: 'warning', hLeg_other: 'notice, info, debug',
hLeg_calm: 'calm', hLeg_burst: 'burst > 3×', hLeg_anom: 'anomaly > 10×', hLeg_calm: 'calm', hLeg_burst: 'burst > 3×', hLeg_anom: 'anomaly > 10×',
hTotal: 'Total', hRatio: (x) => `${x}× the median`, hUnshown: 'not detailed (high rate)', hTotal: 'Total', hRatio: (x) => `${x}× the median`, hUnshown: 'not detailed (high rate)',
hDivision: (s) => `interval ${s}`, hCapped: 'enlarged', hDivision: (s) => `interval ${s}`, hCapped: 'enlarged',
hCappedTitle: 'The chosen division would exceed 300 intervals over this range: it has been enlarged.', hCappedTitle: 'The chosen division would exceed 300 intervals over this range: it has been enlarged.',
hUnzoom: '× Reset zoom', unitDay: 'd', hUnzoom: '× Back to stream', unitDay: 'd',
toTop: 'Back to top', toTop: 'Back to top',
colRcv: 'Received', colMt: 'Message time', colSev: 'Severity', colHost: 'Host', colApp: 'App', colCodes: 'Filters', colMsg: 'Message', colRcv: 'Received', colMt: 'Message time', colSev: 'Severity', colHost: 'Host', colApp: 'App', colCodes: 'Pattern', colMsg: 'Message',
codesTitle: 'Codes of the color tags found in the message', codesTitle: 'Codes of the color tags found in the message',
tagCodeTitle: 'Code shown on the log lines this tag matches', tagCodeTitle: 'Code shown on the log lines this tag matches',
colGrip: 'Drag to resize, double-click for the automatic width', colGrip: 'Drag to resize, double-click for the automatic width',
@@ -244,8 +275,25 @@ const I18N = {
msgTimeTitle: 'Horodatage contenu dans le message', msgTimeTitle: 'Horodatage contenu dans le message',
fHostName: 'nom d\'hôte (DNS)', fHostIp: 'IP de l\'hôte', fHostName: 'nom d\'hôte (DNS)', fHostIp: 'IP de l\'hôte',
clickHost: 'Cliquer pour filtrer sur cet hôte', clickApp: 'Cliquer pour filtrer sur cette appli', clickHost: 'Cliquer pour filtrer sur cet hôte', clickApp: 'Cliquer pour filtrer sur cette appli',
sourceAria: 'Source', srcAll: 'Toutes les sources', tabSources: 'Sources', sourceAria: 'Source', srcAll: 'Toutes les sources', tabSources: 'Sources de logs',
srcHost: 'Système hôte', srcHost: 'Système hôte',
srcKernel: 'Noyau (dmesg)',
auditTitle: 'Audit Linux (auditd)',
auditEnabled: 'Collecter les événements d\'audit de cette machine',
auditPath: 'Fichier',
auditOff: 'Désactivé : le journal d\'audit de la machine qui héberge LogStream n\'est pas lu.',
auditOk: ({ p, r }) => `Suivi de ${p} : ${r} événements depuis le démarrage.`,
audit_not_found: 'Journal d\'audit introuvable : montez /var/log/audit dans le conteneur (voir docker-compose.yml) et vérifiez le chemin. ',
audit_permission: 'Accès refusé au journal d\'audit : réglez log_group dans /etc/audit/auditd.conf, ou ajoutez une ACL pour l\'UID 10001 (voir le README). ',
audit_read: 'Lecture du journal d\'audit impossible : ',
auditHelp: 'Lit le journal du démon d\'audit Linux (auditd) : connexions, sudo, fichiers surveillés, appels système visés par vos règles… Les enregistrements d\'un même événement sont regroupés sur une ligne, avec un résumé (qui, quoi, résultat) et l\'enregistrement brut en dessous ; les échecs sont des warnings. À l\'activation, la dernière heure est lue d\'abord (AUDIT_BACKFILL) ; les rotations sont suivies et la lecture reprend où elle s\'était arrêtée.',
selfTitle: 'Logs internes de LogStream',
selfEnabled: 'Enregistrer les événements de LogStream lui-même',
selfLevel: 'Niveau minimal',
selfOff: 'Désactivé : les événements de LogStream ne vont que dans la sortie de son conteneur.',
selfOk: ({ n }) => `${n} événements enregistrés depuis le démarrage (hôte logstream).`,
selfDropped: (n) => ` ${n} écartés (répétés ou trop nombreux d\'un coup).`,
selfHelp: 'Transforme les événements de LogStream en logs (hôte logstream, app = composant) : VictoriaLogs injoignable puis de retour, lots perdus, erreurs du listener syslog, flux Docker décrochés, sources en échec, démarrage et arrêt. Un même événement est enregistré au plus une fois toutes les 30 s, avec le nombre de répétitions.',
hostTitle: 'Logs système de l\'hôte', hostTitle: 'Logs système de l\'hôte',
hostEnabled: 'Collecter les logs système de cette machine', hostEnabled: 'Collecter les logs système de cette machine',
hostOff: 'Désactivé : les logs système de la machine qui héberge LogStream ne sont pas collectés.', hostOff: 'Désactivé : les logs système de la machine qui héberge LogStream ne sont pas collectés.',
@@ -258,6 +306,15 @@ const I18N = {
host_empty: 'Ni journal systemd ni fichier de log trouvé dans /var/log.', host_empty: 'Ni journal systemd ni fichier de log trouvé dans /var/log.',
host_read: 'Lecture des logs de l\'hôte impossible : ', host_read: 'Lecture des logs de l\'hôte impossible : ',
hostHelp: 'Lit le journal systemd de l\'hôte (/var/log/journal, /run/log/journal), ou les fichiers texte de /var/log (syslog, messages, *.log) s\'il n\'y a pas de journal. Les répertoires sont montés en lecture seule. À l\'activation, la dernière heure est lue d\'abord (HOST_LOGS_BACKFILL).', hostHelp: 'Lit le journal systemd de l\'hôte (/var/log/journal, /run/log/journal), ou les fichiers texte de /var/log (syslog, messages, *.log) s\'il n\'y a pas de journal. Les répertoires sont montés en lecture seule. À l\'activation, la dernière heure est lue d\'abord (HOST_LOGS_BACKFILL).',
dmesgTitle: 'Messages du noyau (dmesg)',
dmesgEnabled: 'Collecter les messages du noyau de cette machine',
dmesgOff: 'Désactivé : les messages du noyau de la machine qui héberge LogStream ne sont pas collectés.',
dmesgOk: ({ d, r }) => `Lecture de ${d} : ${r} messages depuis le démarrage.`,
dmesgLost: (n) => ` ${n} messages ont été écrasés dans le buffer du noyau avant d'être lus.`,
dmesg_not_found: 'Périphérique /dev/kmsg absent du conteneur : décommentez sa section devices dans docker-compose.yml (voir le README).',
dmesg_permission: 'Accès refusé à /dev/kmsg : réglez kernel.dmesg_restrict=0 sur l\'hôte, ou lancez le conteneur en root avec la capacité SYSLOG (voir le README). ',
dmesg_read: 'Lecture des messages du noyau impossible : ',
dmesgHelp: 'Lit le buffer circulaire du noyau de l\'hôte via /dev/kmsg, comme dmesg : matériel, pilotes, OOM killer, pare-feu… À l\'activation, la dernière heure est lue d\'abord (DMESG_BACKFILL) ; après un redémarrage, la lecture reprend où elle s\'était arrêtée.',
syslogEnabled: 'Recevoir les messages syslog', syslogProtocols: 'Protocoles', syslogPort: 'Port', syslogEnabled: 'Recevoir les messages syslog', syslogProtocols: 'Protocoles', syslogPort: 'Port',
syslogListening: ({ port, protos }) => `Écoute sur le port ${port} (${protos}).`, syslogListening: ({ port, protos }) => `Écoute sur le port ${port} (${protos}).`,
syslogOff: 'Réception syslog désactivée : les messages syslog sont ignorés (les logs Docker sont toujours collectés).', syslogOff: 'Réception syslog désactivée : les messages syslog sont ignorés (les logs Docker sont toujours collectés).',
@@ -285,7 +342,7 @@ const I18N = {
exportNote: (n) => `Jusqu'à ${n} logs correspondant aux filtres, du plus récent au plus ancien. Dates dans le fuseau choisi dans Paramètres.`, exportNote: (n) => `Jusqu'à ${n} logs correspondant aux filtres, du plus récent au plus ancien. Dates dans le fuseau choisi dans Paramètres.`,
exporting: 'Export…', exportDone: (f) => `Téléchargé : ${f}`, exportError: 'Échec de l\'export : ', exporting: 'Export…', exportDone: (f) => `Téléchargé : ${f}`, exportError: 'Échec de l\'export : ',
err_export_pipes: 'L\'export ne gère pas les pipes LogsQL (| …) : gardez seulement la partie filtre', err_export_pipes: 'L\'export ne gère pas les pipes LogsQL (| …) : gardez seulement la partie filtre',
tabLocale: 'Localisation', tabFilters: 'Filtres', tabInterface: 'Interface', tabData: 'Données', tabLocale: 'Localisation et langue', tabFilters: 'Motifs de recherche', tabInterface: 'Affichage', tabData: 'Base de données',
themeTitle: 'Thème', themeSystem: 'Système', themeLight: 'Clair', themeDark: 'Sombre', themeTitle: 'Thème', themeSystem: 'Système', themeLight: 'Clair', themeDark: 'Sombre',
themeHelp: 'Système suit la préférence clair/sombre de votre ordinateur ou de votre téléphone.', themeHelp: 'Système suit la préférence clair/sombre de votre ordinateur ou de votre téléphone.',
logDisplay: 'Affichage des logs', fontSize: 'Taille du texte', logDisplay: 'Affichage des logs', fontSize: 'Taille du texte',
@@ -319,22 +376,27 @@ const I18N = {
authOff: 'Aucune authentification : toute personne qui atteint cette page peut lire les logs et changer les réglages. Définissez AUTH_USER / AUTH_PASS ou AUTH_MODE=oidc.', authOff: 'Aucune authentification : toute personne qui atteint cette page peut lire les logs et changer les réglages. Définissez AUTH_USER / AUTH_PASS ou AUTH_MODE=oidc.',
readOnlyNote: 'Compte en lecture seule : seul un administrateur peut modifier ces réglages.', readOnlyNote: 'Compte en lecture seule : seul un administrateur peut modifier ces réglages.',
err_purge_confirm: 'Tapez PURGE pour confirmer', err_purge_confirm: 'Tapez PURGE pour confirmer',
liveZoomed: 'Le direct n\'est pas disponible sur une plage zoomée', liveZoomed: 'Le direct n\'est disponible qu\'en mode Flux',
connZoom: 'direct en pause (zoom)', connZoom: 'direct en pause (période)',
viewAria: 'Mode d\'affichage', viewStream: 'Flux', viewPeriod: 'Période',
viewStreamTitle: 'Flux : les derniers logs sur une durée glissante, mis à jour en direct',
viewPeriodTitle: 'Période : les logs entre une date de début et une date de fin',
perFrom: 'Début', perTo: 'Fin', perPrev: 'Période précédente', perNext: 'Période suivante',
perOut: 'Élargir (deux fois plus long)',
histoTitle: 'Frise', histoTitle: 'Frise',
hScale: 'Échelle', hLinear: 'Linéaire', hHeight: 'Hauteur', hColor: 'Couleur', hScale: 'Échelle', hLinear: 'Linéaire', hHeight: 'Hauteur', hColor: 'Couleur',
hColSeverity: 'Sévérité', hColIntensity: 'Intensité', hColNone: 'Aucune', hColSeverity: 'Sévérité', hColIntensity: 'Intensité', hColNone: 'Aucune',
hRender: 'Rendu', hBars: 'Barres', hArea: 'Aire', hRender: 'Rendu', hBars: 'Barres', hArea: 'Aire',
hStep: 'Division', hAuto: 'Automatique', hRefresh: 'Rafraîchissement', hOff: 'Désactivé', hRefreshAuto: 'À chaque nouvel intervalle', hStep: 'Division', hAuto: 'Automatique', hRefresh: 'Rafraîchissement', hOff: 'Désactivé', hRefreshAuto: 'À chaque nouvel intervalle',
histoHelp: 'Cliquez sur une barre pour zoomer sur son intervalle, ou glissez sur plusieurs. L\'échelle √ garde visibles les petits volumes à côté des rafales. L\'intensité compare chaque intervalle à la médiane de la fenêtre : calme, rafale (plus de 3×), anomalie (plus de 10×). En direct, le dernier intervalle se met à jour à l\'arrivée des messages.', histoHelp: 'Cliquez sur une barre pour afficher son intervalle en mode Période, ou glissez sur plusieurs. L\'échelle √ garde visibles les petits volumes à côté des rafales. L\'intensité compare chaque intervalle à la médiane de la fenêtre : calme, rafale (plus de 3×), anomalie (plus de 10×). En direct, le dernier intervalle se met à jour à l\'arrivée des messages.',
hLeg_err: 'error et plus', hLeg_warn: 'warning', hLeg_other: 'notice, info, debug', hLeg_err: 'error et plus', hLeg_warn: 'warning', hLeg_other: 'notice, info, debug',
hLeg_calm: 'calme', hLeg_burst: 'rafale > 3×', hLeg_anom: 'anomalie > 10×', hLeg_calm: 'calme', hLeg_burst: 'rafale > 3×', hLeg_anom: 'anomalie > 10×',
hTotal: 'Total', hRatio: (x) => `${x}× la médiane`, hUnshown: 'sans détail (débit élevé)', hTotal: 'Total', hRatio: (x) => `${x}× la médiane`, hUnshown: 'sans détail (débit élevé)',
hDivision: (s) => `intervalle ${s}`, hCapped: 'élargi', hDivision: (s) => `intervalle ${s}`, hCapped: 'élargi',
hCappedTitle: 'La division choisie dépasserait 300 intervalles sur cette plage : elle a été élargie.', hCappedTitle: 'La division choisie dépasserait 300 intervalles sur cette plage : elle a été élargie.',
hUnzoom: '× Annuler le zoom', unitDay: 'j', hUnzoom: '× Revenir au flux', unitDay: 'j',
toTop: 'Revenir en haut', toTop: 'Revenir en haut',
colRcv: 'Réception', colMt: 'Heure message', colSev: 'Sévérité', colHost: 'Hôte', colApp: 'App', colCodes: 'Filtres', colMsg: 'Message', colRcv: 'Réception', colMt: 'Heure message', colSev: 'Sévérité', colHost: 'Hôte', colApp: 'App', colCodes: 'Motif de recherche', colMsg: 'Message',
codesTitle: 'Codes des tags de couleur trouvés dans le message', codesTitle: 'Codes des tags de couleur trouvés dans le message',
tagCodeTitle: 'Code affiché sur les lignes de log où ce tag est trouvé', tagCodeTitle: 'Code affiché sur les lignes de log où ce tag est trouvé',
colGrip: 'Glisser pour redimensionner, double-clic pour la largeur automatique', colGrip: 'Glisser pour redimensionner, double-clic pour la largeur automatique',
@@ -493,7 +555,7 @@ const state = {
live: store.get('live', '1') === '1', live: store.get('live', '1') === '1',
es: null, es: null,
reqId: 0, reqId: 0,
zoom: null, // {from, to} in ms when the timeline is zoomed (range "custom") zoom: null, // {from, to} in ms in Time range mode (range "custom"), null in Stream mode
tookMs: null, tookMs: null,
conn: ['', 'connPaused'], conn: ['', 'connPaused'],
stats: null, stats: null,
@@ -522,7 +584,7 @@ function setLang(next) {
setConn(...state.conn); setConn(...state.conn);
renderList(); renderList();
updateCount(state.tookMs); updateCount(state.tookMs);
setZoomOption(); renderPeriod();
renderHisto(); renderHisto();
renderStats(); renderStats();
renderTagList(); renderTagList();
@@ -533,6 +595,9 @@ function setLang(next) {
renderInterface(); renderInterface();
renderSyslog(); renderSyslog();
renderHost(); renderHost();
renderDmesg();
renderAudit();
renderSelfLog();
renderDocker(); renderDocker();
} }
@@ -840,10 +905,15 @@ function fmtTime(s) {
return `${day} ${p.h}:${p.m}:${p.s}<span class="ms">.${p.ms}</span>`; return `${day} ${p.h}:${p.m}:${p.s}<span class="ms">.${p.ms}</span>`;
} }
// Sources with their own look in the list: a colored edge and a badge
// before the app name (the severity badge stays the same).
const SRC_BADGE = { audit: 'AUDIT', logstream: 'LS' };
function rowHTML(r, isNew) { function rowHTML(r, isNew) {
const sev = r.severity || ''; const sev = r.severity || '';
const cls = SEV_CLASS[sev] || 'info'; const cls = SEV_CLASS[sev] || 'info';
return `<article class="row sev-${cls}${isNew ? ' new' : ''}" tabindex="0">` const badge = SRC_BADGE[r.source_type];
return `<article class="row sev-${cls}${badge ? ' src-' + r.source_type : ''}${isNew ? ' new' : ''}" tabindex="0">`
+ `<time class="rcv" title="${esc(t('rcvTitle'))}">${r.received ? esc(fmtFull(r.received)) : '—'}</time>` + `<time class="rcv" title="${esc(t('rcvTitle'))}">${r.received ? esc(fmtFull(r.received)) : '—'}</time>`
+ `<time class="mt" title="${esc(t('msgTimeTitle'))}">${fmtTime(r.msg_time || r._time)}</time>` + `<time class="mt" title="${esc(t('msgTimeTitle'))}">${fmtTime(r.msg_time || r._time)}</time>`
+ `<span class="sev">${esc(sev || '—')}</span>` + `<span class="sev">${esc(sev || '—')}</span>`
@@ -851,13 +921,20 @@ function rowHTML(r, isNew) {
? `<span class="host" data-act="host" title="${esc(hostTitle(r) + ' · ' + t('clickHost'))}">${esc(r.host_name || r.host)}</span>` ? `<span class="host" data-act="host" title="${esc(hostTitle(r) + ' · ' + t('clickHost'))}">${esc(r.host_name || r.host)}</span>`
: '<span class="host"></span>') : '<span class="host"></span>')
+ (r.app + (r.app
? `<span class="app${r.source_type === 'docker' ? ' proj' : ''}" data-act="app"${r.source_type === 'docker' ? ` style="--h:${hueOf(r.compose_project || r.container || r.app)}"` : ''} title="${esc((r.compose_project ? r.compose_project + '/' : '') + (r.container ? r.container + ' · ' : '') + r.app + ' · ' + t('clickApp'))}">${r.source_type === 'docker' ? DOCKER_ICON : ''}${esc(r.app)}</span>` ? `<span class="app${r.source_type === 'docker' ? ' proj' : ''}" data-act="app"${r.source_type === 'docker' ? ` style="--h:${hueOf(r.compose_project || r.container || r.app)}"` : ''} title="${esc((r.compose_project ? r.compose_project + '/' : '') + (r.container ? r.container + ' · ' : '') + r.app + ' · ' + t('clickApp'))}">${r.source_type === 'docker' ? DOCKER_ICON : ''}${badge ? `<b class="src-badge">${badge}</b>` : ''}${esc(r.app)}</span>`
: '<span class="app"></span>') : '<span class="app"></span>')
+ `<span class="codes">${codesHTML(r._msg)}</span>` + `<span class="codes">${codesHTML(r._msg)}</span>`
+ `<div class="msg">${highlight(r._msg)}</div>` + `<div class="msg">${msgHTML(r)}</div>`
+ '</article>'; + '</article>';
} }
// Message cell; audit events also show their raw record (hidden in compact
// density until the row is opened).
function msgHTML(r) {
return highlight(r._msg)
+ (r.source_type === 'audit' && r.audit_raw ? `<div class="raw">${esc(r.audit_raw)}</div>` : '');
}
// Stable hue (0-359) for a name: the compose project color, shared by the // Stable hue (0-359) for a name: the compose project color, shared by the
// Sources labels and the Docker app names in the log list. // Sources labels and the Docker app names in the log list.
function hueOf(name) { function hueOf(name) {
@@ -921,7 +998,7 @@ function rehighlight() {
for (const row of list.children) { for (const row of list.children) {
const r = recOf.get(row); const r = recOf.get(row);
if (!r) continue; if (!r) continue;
row.querySelector('.msg').innerHTML = highlight(r._msg); row.querySelector('.msg').innerHTML = msgHTML(r);
row.querySelector('.codes').innerHTML = codesHTML(r._msg); row.querySelector('.codes').innerHTML = codesHTML(r._msg);
} }
} }
@@ -948,8 +1025,8 @@ function updateCount(tookMs) {
} }
function detailsHTML(r) { function detailsHTML(r) {
const order = ['received', 'msg_time', '_time', 'host', 'host_name', 'host_ip', 'source_type', 'container', 'image', 'compose_project', 'compose_service', 'stream', 'unit', 'log_file', 'app', 'procid', 'severity', 'facility', 'source', 'proto', 'sd', '_msg']; const order = ['received', 'msg_time', '_time', 'host', 'host_name', 'host_ip', 'source_type', 'audit_type', 'audit_serial', 'auid', 'user', 'exe', 'res', 'key', 'container', 'image', 'compose_project', 'compose_service', 'stream', 'unit', 'log_file', 'subsystem', 'device', 'app', 'procid', 'severity', 'facility', 'source', 'proto', 'sd', '_msg'];
const hidden = new Set(['_stream_id', '_stream', 'sevnum']); const hidden = new Set(['_stream_id', '_stream', 'sevnum', 'audit_raw']); // audit_raw: shown under the message
if (r.msg_time) hidden.add('_time'); // same as the reception time if (r.msg_time) hidden.add('_time'); // same as the reception time
const keys = order.filter((k) => r[k] != null && r[k] !== '' && !hidden.has(k)) const keys = order.filter((k) => r[k] != null && r[k] !== '' && !hidden.has(k))
.concat(Object.keys(r).filter((k) => !order.includes(k) && !hidden.has(k)).sort()); .concat(Object.keys(r).filter((k) => !order.includes(k) && !hidden.has(k)).sort());
@@ -1435,33 +1512,115 @@ function showHistoSel() {
sel.style.width = ((b - a + 1) / d.count) * 100 + '%'; sel.style.width = ((b - a + 1) / d.count) * 100 + '%';
} }
function setZoomOption() { /* ---- Display mode: Stream or Time range ---- */
const sel = $('#range');
let opt = sel.querySelector('option[value="custom"]'); // datetime-local fields hold a wall time in the chosen time zone.
if (!state.zoom) { function toWall(ms, sec) {
if (opt) opt.remove(); const p = zoneParts(new Date(ms));
return; return `${p.Y}-${p.M}-${p.D}T${p.h}:${p.m}${sec ? ':' + p.s : ''}`;
} }
if (!opt) { opt = new Option('', 'custom'); sel.add(opt); }
opt.textContent = fmtSpan(state.zoom.from, state.zoom.to, state.zoom.to - state.zoom.from < 3e5); function zoneOffsetMs(ms) {
sel.value = 'custom'; const m = /^([+-])(\d\d):(\d\d)$/.exec(zoneParts(new Date(ms)).off);
return m ? (m[1] === '-' ? -1 : 1) * (m[2] * 36e5 + m[3] * 6e4) : 0;
}
function fromWall(v) {
const m = /^(\d{4})-(\d\d)-(\d\d)T(\d\d):(\d\d)(?::(\d\d))?/.exec(v || '');
if (!m) return NaN;
const wall = Date.UTC(m[1], m[2] - 1, m[3], m[4], m[5], m[6] || 0);
const guess = wall - zoneOffsetMs(wall);
return wall - zoneOffsetMs(guess); // second pass: right offset around DST changes
}
const serverNow = () => Date.now() + (histo.data ? histo.data.offset : 0);
function renderPeriod() {
const on = !!state.zoom;
for (const b of $('#viewMode').querySelectorAll('[data-view]')) {
b.setAttribute('aria-checked', String((b.dataset.view === 'period') === on));
}
$('#range').hidden = on;
$('#period').hidden = !on;
if (!on) return;
const { from, to } = state.zoom;
const sec = from % 6e4 !== 0 || to % 6e4 !== 0;
for (const [el, v] of [[$('#perFrom'), from], [$('#perTo'), to]]) {
el.step = sec ? 1 : 60;
el.classList.remove('bad');
if (document.activeElement !== el) el.value = toWall(v, sec);
}
$('#perNext').disabled = to >= serverNow();
} }
function zoomTo(from, to) { function zoomTo(from, to) {
state.zoom = { from: Math.round(from), to: Math.round(to) }; state.zoom = { from: Math.round(from), to: Math.round(to) };
setZoomOption(); store.set('period', JSON.stringify(state.zoom));
renderPeriod();
hideHistoTip(); hideHistoTip();
refresh(); refresh();
} }
function unzoom() { function unzoom() {
state.zoom = null; state.zoom = null;
setZoomOption(); store.set('period', '');
renderPeriod();
$('#range').value = store.get('range', '1h'); $('#range').value = store.get('range', '1h');
if (!$('#range').value) $('#range').value = '1h'; if (!$('#range').value) $('#range').value = '1h';
refresh(); refresh();
} }
// Entering Time range mode starts from what the stream was showing.
function enterPeriod() {
if (state.zoom) return;
const d = histo.data;
if (d) return zoomTo(d.start, d.start + d.count * d.step);
const to = Math.ceil(serverNow() / 6e4) * 6e4;
zoomTo(to - (RANGE_MS[$('#range').value] || 36e5), to);
}
$('#viewMode').addEventListener('click', (ev) => {
const b = ev.target.closest('[data-view]');
if (!b) return;
if (b.dataset.view === 'period') enterPeriod();
else if (state.zoom) unzoom();
});
const onPeriodInput = debounce(() => {
if (!state.zoom) return;
const from = fromWall($('#perFrom').value);
const to = fromWall($('#perTo').value);
const ok = !isNaN(from) && !isNaN(to) && to > from;
$('#perFrom').classList.toggle('bad', isNaN(from) || (!isNaN(to) && !ok));
$('#perTo').classList.toggle('bad', isNaN(to) || (!isNaN(from) && !ok));
if (ok && (from !== state.zoom.from || to !== state.zoom.to)) zoomTo(from, to);
}, 500);
for (const id of ['perFrom', 'perTo']) {
$('#' + id).addEventListener('input', onPeriodInput);
$('#' + id).addEventListener('keydown', (ev) => { if (ev.key === 'Enter') ev.target.blur(); });
$('#' + id).addEventListener('blur', () => { if (!$('#' + id).classList.contains('bad')) renderPeriod(); });
}
// ◀ ▶ move by the width of the range, − doubles it around its middle.
function shiftPeriod(dir) {
const { from, to } = state.zoom;
const w = to - from;
zoomTo(from + dir * w, to + dir * w);
}
$('#perPrev').addEventListener('click', () => shiftPeriod(-1));
$('#perNext').addEventListener('click', () => shiftPeriod(1));
$('#perOut').addEventListener('click', () => {
const { from, to } = state.zoom;
const w = to - from;
let a = from - w / 2;
let b = to + w / 2;
const now = serverNow();
const end = Math.max(to, now); // widen into the past rather than the future
if (b > end) { a -= b - end; b = end; }
const r = w >= 12e4 ? 6e4 : 1e3; // whole minutes, or seconds for short ranges
zoomTo(Math.round(a / r) * r, Math.round(b / r) * r);
});
{ {
const el = $('#histo'); const el = $('#histo');
el.innerHTML = '<div class="h-plot"><svg class="h-svg" preserveAspectRatio="none" aria-hidden="true"></svg>' el.innerHTML = '<div class="h-plot"><svg class="h-svg" preserveAspectRatio="none" aria-hidden="true"></svg>'
@@ -1583,10 +1742,6 @@ async function refresh() {
} }
const onFilterChange = (ev) => { const onFilterChange = (ev) => {
if (ev && ev.target.id === 'range' && state.zoom && ev.target.value !== 'custom') {
state.zoom = null; // another range chosen: leave the zoom
setZoomOption();
}
if (!state.zoom) store.set('range', $('#range').value); if (!state.zoom) store.set('range', $('#range').value);
store.set('severity', $('#severity').value); store.set('severity', $('#severity').value);
refresh(); refresh();
@@ -1750,7 +1905,7 @@ function onTimePrefsChange() {
store.set('timefmt', timePrefs.fmt); store.set('timefmt', timePrefs.fmt);
updateTimePreview(); updateTimePreview();
renderList(); renderList();
setZoomOption(); renderPeriod();
refreshHisto(); // intervals are aligned on the local time refreshHisto(); // intervals are aligned on the local time
} }
$('#tzSelect').addEventListener('change', onTimePrefsChange); $('#tzSelect').addEventListener('change', onTimePrefsChange);
@@ -1912,6 +2067,121 @@ async function configureHost(enabled) {
$('#hostEnabled').addEventListener('change', (ev) => configureHost(ev.target.checked)); $('#hostEnabled').addEventListener('change', (ev) => configureHost(ev.target.checked));
/* ================= Kernel messages (dmesg) ================= */
let dmesgState = null;
async function loadDmesg() {
try { dmesgState = await api('/api/dmesg'); } catch { dmesgState = null; }
renderDmesg();
}
function renderDmesg() {
const s = dmesgState;
const status = $('#dmesgStatus');
if (!s) { status.textContent = ''; return; }
$('#dmesgEnabled').checked = s.enabled;
let text = t('hostWaiting');
let cls = 'muted';
if (!s.enabled) {
text = t('dmesgOff');
} else if (s.code) {
text = t('dmesg_' + s.code) + (s.code === 'read' || s.code === 'permission' ? s.error || '' : '');
cls = 'bad';
} else if (s.open) {
text = t('dmesgOk', { d: s.device, r: fmtNum(s.read) }) + (s.lost ? t('dmesgLost', fmtNum(s.lost)) : '');
cls = 'good';
}
status.textContent = text;
status.className = 'docker-status ' + cls;
}
async function configureDmesg(enabled) {
dmesgState = { ...(dmesgState || {}), enabled };
renderDmesg();
try { dmesgState = await api('/api/dmesg', { method: 'PUT', body: { enabled } }); } catch (e) { toast(e.message); }
renderDmesg();
setTimeout(loadDmesg, 1500); // the device is opened in the background
}
$('#dmesgEnabled').addEventListener('change', (ev) => configureDmesg(ev.target.checked));
/* ================= Linux audit (auditd) ================= */
let auditState = null;
async function loadAudit() {
try { auditState = await api('/api/audit'); } catch { auditState = null; }
renderAudit();
}
function renderAudit() {
const s = auditState;
const status = $('#auditStatus');
if (!s) { status.textContent = ''; return; }
$('#auditEnabled').checked = s.enabled;
const input = $('#auditPath');
input.placeholder = s.defaultPath || '';
if (document.activeElement !== input) input.value = s.path || '';
let text = t('hostWaiting');
let cls = 'muted';
if (!s.enabled) {
text = t('auditOff');
} else if (s.code) {
text = t('audit_' + s.code) + (s.error || '');
cls = 'bad';
} else if (s.open) {
text = t('auditOk', { p: s.path || s.defaultPath, r: fmtNum(s.read) });
cls = 'good';
}
status.textContent = text;
status.className = 'docker-status ' + cls;
}
async function configureAudit(change) {
const body = { enabled: auditState?.enabled || false, path: auditState?.path || '', ...change };
auditState = { ...(auditState || {}), ...body };
renderAudit();
try { auditState = await api('/api/audit', { method: 'PUT', body }); } catch (e) { toast(e.message); }
renderAudit();
setTimeout(loadAudit, 1500); // the file is read in the background
}
$('#auditEnabled').addEventListener('change', (ev) => configureAudit({ enabled: ev.target.checked }));
$('#auditPath').addEventListener('change', (ev) => configureAudit({ path: ev.target.value.trim() }));
/* ================= LogStream internal logs ================= */
let selfState = null;
async function loadSelfLog() {
try { selfState = await api('/api/selflog'); } catch { selfState = null; }
renderSelfLog();
}
function renderSelfLog() {
const s = selfState;
const status = $('#selfStatus');
if (!s) { status.textContent = ''; return; }
$('#selfEnabled').checked = s.enabled;
$('#selfLevel').value = s.min_level || 'warning';
$('#selfLevel').disabled = !s.enabled;
const left = (s.dropped || 0) + (s.throttled || 0);
status.textContent = s.enabled ? t('selfOk', { n: fmtNum(s.emitted || 0) }) + (left ? t('selfDropped', fmtNum(left)) : '') : t('selfOff');
status.className = 'docker-status ' + (s.enabled ? 'good' : 'muted');
}
async function configureSelfLog(change) {
const body = { enabled: selfState?.enabled ?? true, min_level: selfState?.min_level || 'warning', ...change };
selfState = { ...(selfState || {}), ...body };
renderSelfLog();
try { selfState = await api('/api/selflog', { method: 'PUT', body }); } catch (e) { toast(e.message); }
renderSelfLog();
}
$('#selfEnabled').addEventListener('change', (ev) => configureSelfLog({ enabled: ev.target.checked }));
$('#selfLevel').addEventListener('change', (ev) => configureSelfLog({ min_level: ev.target.value }));
/* ================= Docker source ================= */ /* ================= Docker source ================= */
let dockerState = null; let dockerState = null;
@@ -2028,6 +2298,9 @@ setInterval(() => {
if ($('#settingsDlg').open && !document.querySelector('[data-panel="sources"]').hidden) { if ($('#settingsDlg').open && !document.querySelector('[data-panel="sources"]').hidden) {
loadDocker(); loadDocker();
loadHost(); loadHost();
loadDmesg();
loadAudit();
loadSelfLog();
} }
}, 4000); }, 4000);
@@ -2203,7 +2476,7 @@ function renderTagList() {
filterTagList(); filterTagList();
} }
// Settings > Filters: shows only the tags whose keyword, label or code contains the filter. // Settings > Pattern: shows only the tags whose keyword, label or code contains the filter.
function filterTagList() { function filterTagList() {
const q = $('#tagFilter').value.trim().toLowerCase(); const q = $('#tagFilter').value.trim().toLowerCase();
let shown = 0; let shown = 0;
@@ -2321,6 +2594,9 @@ $('#settingsBtn').addEventListener('click', () => {
loadPurgeStatus(); loadPurgeStatus();
loadSyslog(); loadSyslog();
loadHost(); loadHost();
loadDmesg();
loadAudit();
loadSelfLog();
loadDocker(); loadDocker();
renderDbStats(); renderDbStats();
loadDbStats(); loadDbStats();
@@ -2350,6 +2626,11 @@ applyLang();
$('#range').value = store.get('range', '1h'); $('#range').value = store.get('range', '1h');
if (!$('#range').value) $('#range').value = '1h'; if (!$('#range').value) $('#range').value = '1h';
$('#severity').value = store.get('severity', ''); $('#severity').value = store.get('severity', '');
try { // Time range mode is kept across reloads
const z = JSON.parse(store.get('period', '') || 'null');
if (z && z.to > z.from) state.zoom = { from: z.from, to: z.to };
} catch { /* stream mode */ }
renderPeriod();
// Without a login, warn that the UI is open to everyone. With a login (local or OIDC), // Without a login, warn that the UI is open to everyone. With a login (local or OIDC),
// show who is logged in and the log out button, and lock the admin settings of a // show who is logged in and the log out button, and lock the admin settings of a
@@ -2357,6 +2638,19 @@ $('#severity').value = store.get('severity', '');
fetch('/auth/me').then((res) => (res.ok ? res.json() : null)).then((me) => { fetch('/auth/me').then((res) => (res.ok ? res.json() : null)).then((me) => {
if (!me) return; if (!me) return;
if (me.mode === 'none' && store.get('authWarnHidden') !== '1') $('#authWarn').hidden = false; if (me.mode === 'none' && store.get('authWarnHidden') !== '1') $('#authWarn').hidden = false;
const who = $('#whoami');
if (me.mode === 'none') {
who.classList.add('none');
who.querySelector('.who-name').textContent = '(no auth)';
who.hidden = false;
} else if (me.user) {
who.querySelector('.who-name').textContent = me.user;
who.title = me.user;
const badge = who.querySelector('.who-badge');
badge.textContent = me.mode;
badge.classList.add(me.mode);
who.hidden = false;
}
if (me.role === 'viewer') { if (me.role === 'viewer') {
document.body.classList.add('read-only'); document.body.classList.add('read-only');
for (const p of document.querySelectorAll('[data-panel="filters"], [data-panel="sources"], [data-panel="data"]')) { for (const p of document.querySelectorAll('[data-panel="filters"], [data-panel="sources"], [data-panel="data"]')) {
+80 -6
View File
@@ -41,6 +41,7 @@
<svg class="sun" viewBox="0 0 24 24"><circle cx="12" cy="12" r="4"/><path d="M12 2v2M12 20v2M4.9 4.9l1.4 1.4M17.7 17.7l1.4 1.4M2 12h2M20 12h2M4.9 19.1l1.4-1.4M17.7 6.3l1.4-1.4"/></svg> <svg class="sun" viewBox="0 0 24 24"><circle cx="12" cy="12" r="4"/><path d="M12 2v2M12 20v2M4.9 4.9l1.4 1.4M17.7 17.7l1.4 1.4M2 12h2M20 12h2M4.9 19.1l1.4-1.4M17.7 6.3l1.4-1.4"/></svg>
<svg class="moon" viewBox="0 0 24 24"><path d="M21 12.8A9 9 0 1 1 11.2 3a7 7 0 0 0 9.8 9.8z"/></svg> <svg class="moon" viewBox="0 0 24 24"><path d="M21 12.8A9 9 0 1 1 11.2 3a7 7 0 0 0 9.8 9.8z"/></svg>
</button> </button>
<span id="whoami" class="whoami" hidden><span class="who-name"></span><span class="who-badge"></span></span>
<a id="logoutBtn" class="icon-btn" href="/auth/logout" hidden data-i18n-title="logout" data-i18n-aria="logout"> <a id="logoutBtn" class="icon-btn" href="/auth/logout" hidden data-i18n-title="logout" data-i18n-aria="logout">
<!-- Log out icon (Lucide "log-out", ISC license) --> <!-- Log out icon (Lucide "log-out", ISC license) -->
<svg viewBox="0 0 24 24"><path d="M9 21H5a2 2 0 0 1-2-2V5a2 2 0 0 1 2-2h4"/><path d="m16 17 5-5-5-5"/><path d="M21 12H9"/></svg> <svg viewBox="0 0 24 24"><path d="M9 21H5a2 2 0 0 1-2-2V5a2 2 0 0 1 2-2h4"/><path d="m16 17 5-5-5-5"/><path d="M21 12H9"/></svg>
@@ -50,6 +51,33 @@
</header> </header>
<section class="filters"> <section class="filters">
<div id="viewMode" class="seg view-mode" role="radiogroup" data-i18n-aria="viewAria">
<button type="button" role="radio" data-view="stream" data-i18n-title="viewStreamTitle">
<!-- Lucide "radio" (ISC license) -->
<svg viewBox="0 0 24 24" aria-hidden="true"><path d="M4.9 19.1C1 15.2 1 8.8 4.9 4.9"/><path d="M7.8 16.2c-2.3-2.3-2.3-6.1 0-8.5"/><circle cx="12" cy="12" r="2"/><path d="M16.2 7.8c2.3 2.3 2.3 6.1 0 8.5"/><path d="M19.1 4.9C23 8.8 23 15.1 19.1 19"/></svg>
<span data-i18n="viewStream">Stream</span>
</button>
<button type="button" role="radio" data-view="period" data-i18n-title="viewPeriodTitle">
<!-- Lucide "calendar-range" (ISC license) -->
<svg viewBox="0 0 24 24" aria-hidden="true"><rect width="18" height="18" x="3" y="4" rx="2"/><path d="M16 2v4M3 10h18M8 2v4M17 14h-6M13 18H7M7 14h.01M17 18h.01"/></svg>
<span data-i18n="viewPeriod">Time range</span>
</button>
</div>
<div id="period" class="period" hidden>
<button id="perPrev" class="btn tool" type="button" data-i18n-title="perPrev" data-i18n-aria="perPrev">
<svg viewBox="0 0 24 24" aria-hidden="true"><path d="m15 18-6-6 6-6"/></svg>
</button>
<input id="perFrom" type="datetime-local" data-i18n-title="perFrom" data-i18n-aria="perFrom">
<span class="muted" aria-hidden="true">→</span>
<input id="perTo" type="datetime-local" data-i18n-title="perTo" data-i18n-aria="perTo">
<button id="perNext" class="btn tool" type="button" data-i18n-title="perNext" data-i18n-aria="perNext">
<svg viewBox="0 0 24 24" aria-hidden="true"><path d="m9 18 6-6-6-6"/></svg>
</button>
<button id="perOut" class="btn tool" type="button" data-i18n-title="perOut" data-i18n-aria="perOut">
<!-- Lucide "zoom-out" (ISC license) -->
<svg viewBox="0 0 24 24" aria-hidden="true"><circle cx="11" cy="11" r="8"/><path d="m21 21-4.3-4.3M8 11h6"/></svg>
</button>
</div>
<select id="range" data-i18n-aria="rangeAria"> <select id="range" data-i18n-aria="rangeAria">
<option value="5m" data-i18n="r5m">5 min</option> <option value="5m" data-i18n="r5m">5 min</option>
<option value="15m" data-i18n="r15m">15 min</option> <option value="15m" data-i18n="r15m">15 min</option>
@@ -75,6 +103,9 @@
<option value="syslog">Syslog</option> <option value="syslog">Syslog</option>
<option value="docker">Docker</option> <option value="docker">Docker</option>
<option value="host" data-i18n="srcHost">Host system</option> <option value="host" data-i18n="srcHost">Host system</option>
<option value="kernel" data-i18n="srcKernel">Kernel (dmesg)</option>
<option value="audit">Audit</option>
<option value="logstream">LogStream</option>
</select> </select>
<span class="spacer"></span> <span class="spacer"></span>
<span id="count" class="muted"></span> <span id="count" class="muted"></span>
@@ -113,7 +144,7 @@
<span data-col="sev"><span class="lbl" data-i18n="colSev">Severity</span><i class="grip" data-i18n-title="colGrip"></i></span> <span data-col="sev"><span class="lbl" data-i18n="colSev">Severity</span><i class="grip" data-i18n-title="colGrip"></i></span>
<span data-col="host"><span class="lbl" data-i18n="colHost">Host</span><i class="grip" data-i18n-title="colGrip"></i></span> <span data-col="host"><span class="lbl" data-i18n="colHost">Host</span><i class="grip" data-i18n-title="colGrip"></i></span>
<span data-col="app"><span class="lbl" data-i18n="colApp">App</span><i class="grip" data-i18n-title="colGrip"></i></span> <span data-col="app"><span class="lbl" data-i18n="colApp">App</span><i class="grip" data-i18n-title="colGrip"></i></span>
<span data-col="codes"><span class="lbl" data-i18n="colCodes" data-i18n-title="codesTitle">Filters</span></span> <span data-col="codes"><span class="lbl" data-i18n="colCodes" data-i18n-title="codesTitle">Pattern</span></span>
<span data-col="msg"><span class="lbl" data-i18n="colMsg">Message</span></span> <span data-col="msg"><span class="lbl" data-i18n="colMsg">Message</span></span>
</div> </div>
<main id="list" class="list"></main> <main id="list" class="list"></main>
@@ -149,19 +180,19 @@
</button> </button>
<button type="button" role="tab" data-tab="filters"> <button type="button" role="tab" data-tab="filters">
<svg viewBox="0 0 24 24"><path d="M3 12V4a1 1 0 0 1 1-1h8l9 9-9 9z"/><circle cx="8" cy="8" r="1.5"/></svg> <svg viewBox="0 0 24 24"><path d="M3 12V4a1 1 0 0 1 1-1h8l9 9-9 9z"/><circle cx="8" cy="8" r="1.5"/></svg>
<span data-i18n="tabFilters">Filters</span> <span data-i18n="tabFilters">Patterns</span>
</button> </button>
<button type="button" role="tab" data-tab="sources"> <button type="button" role="tab" data-tab="sources">
<svg viewBox="0 0 24 24"><path d="M21 8 12 3 3 8v8l9 5 9-5z"/><path d="m3 8 9 5 9-5M12 13v8"/></svg> <svg viewBox="0 0 24 24"><path d="M21 8 12 3 3 8v8l9 5 9-5z"/><path d="m3 8 9 5 9-5M12 13v8"/></svg>
<span data-i18n="tabSources">Sources</span> <span data-i18n="tabSources">Log sources</span>
</button> </button>
<button type="button" role="tab" data-tab="interface"> <button type="button" role="tab" data-tab="interface">
<svg viewBox="0 0 24 24"><rect x="3" y="4" width="18" height="12" rx="2"/><path d="M8 20h8M12 16v4"/></svg> <svg viewBox="0 0 24 24"><rect x="3" y="4" width="18" height="12" rx="2"/><path d="M8 20h8M12 16v4"/></svg>
<span data-i18n="tabInterface">Interface</span> <span data-i18n="tabInterface">Appearance</span>
</button> </button>
<button type="button" role="tab" data-tab="data"> <button type="button" role="tab" data-tab="data">
<svg viewBox="0 0 24 24"><ellipse cx="12" cy="5" rx="8" ry="3"/><path d="M4 5v14c0 1.7 3.6 3 8 3s8-1.3 8-3V5M4 12c0 1.7 3.6 3 8 3s8-1.3 8-3"/></svg> <svg viewBox="0 0 24 24"><ellipse cx="12" cy="5" rx="8" ry="3"/><path d="M4 5v14c0 1.7 3.6 3 8 3s8-1.3 8-3V5M4 12c0 1.7 3.6 3 8 3s8-1.3 8-3"/></svg>
<span data-i18n="tabData">Data</span> <span data-i18n="tabData">Database</span>
</button> </button>
</nav> </nav>
@@ -188,7 +219,7 @@
</section> </section>
</div> </div>
<!-- Filters --> <!-- Pattern (color tags; the "filters" tab id is kept for saved settings) -->
<div class="set-panel wide" role="tabpanel" data-panel="filters" hidden> <div class="set-panel wide" role="tabpanel" data-panel="filters" hidden>
<section class="set-section tags-section"> <section class="set-section tags-section">
<div class="sec-head"> <div class="sec-head">
@@ -238,6 +269,49 @@
<p class="muted small" data-i18n="hostHelp"></p> <p class="muted small" data-i18n="hostHelp"></p>
</section> </section>
<section class="set-section">
<h3 data-i18n="dmesgTitle">Kernel messages (dmesg)</h3>
<p id="dmesgStatus" class="docker-status"></p>
<label class="switch-row">
<input id="dmesgEnabled" type="checkbox" class="switch">
<span data-i18n="dmesgEnabled">Collect the kernel messages of this machine</span>
</label>
<p class="muted small" data-i18n="dmesgHelp"></p>
</section>
<section class="set-section">
<h3 data-i18n="auditTitle">Linux audit (auditd)</h3>
<p id="auditStatus" class="docker-status"></p>
<label class="switch-row">
<input id="auditEnabled" type="checkbox" class="switch">
<span data-i18n="auditEnabled">Collect the audit events of this machine</span>
</label>
<div class="sl-grid">
<label class="lbl" for="auditPath" data-i18n="auditPath">File</label>
<input id="auditPath" type="text" class="dk-filter" autocomplete="off" spellcheck="false">
</div>
<p class="muted small src-help" data-i18n="auditHelp"></p>
</section>
<section class="set-section">
<h3 data-i18n="selfTitle">LogStream internal logs</h3>
<p id="selfStatus" class="docker-status"></p>
<label class="switch-row">
<input id="selfEnabled" type="checkbox" class="switch">
<span data-i18n="selfEnabled">Record the events of LogStream itself</span>
</label>
<div class="sl-grid">
<label class="lbl" for="selfLevel" data-i18n="selfLevel">Minimum level</label>
<select id="selfLevel" class="field self-level">
<option value="err" data-i18n="sev3">Errors and above</option>
<option value="warning" data-i18n="sev4">Warnings and above</option>
<option value="notice" data-i18n="sev5">Notice and above</option>
<option value="info" data-i18n="sev6">Info and above</option>
</select>
</div>
<p class="muted small src-help" data-i18n="selfHelp"></p>
</section>
<section class="set-section span"> <section class="set-section span">
<h3 data-i18n="dockerTitle">Docker containers</h3> <h3 data-i18n="dockerTitle">Docker containers</h3>
<p id="dockerStatus" class="docker-status"></p> <p id="dockerStatus" class="docker-status"></p>
+62 -3
View File
@@ -22,6 +22,9 @@
--hit: rgba(99, 102, 241, .18); --hit: rgba(99, 102, 241, .18);
--shadow: 0 10px 30px rgba(15, 23, 42, .12); --shadow: 0 10px 30px rgba(15, 23, 42, .12);
--chip-l: 38%; /* lightness of project-colored text (labels, Docker apps) */ --chip-l: 38%; /* lightness of project-colored text (labels, Docker apps) */
--src-audit: #7c4dbd; /* audit events (auditd) */
--src-ls: #1f9aa3; /* LogStream internal logs */
--src-tint: 6%; /* background tint of their rows */
--sev-crit: #be123c; --sev-crit: #be123c;
--sev-err: #dc2626; --sev-err: #dc2626;
@@ -70,6 +73,9 @@
--hit: rgba(129, 140, 248, .28); --hit: rgba(129, 140, 248, .28);
--shadow: 0 10px 30px rgba(0, 0, 0, .5); --shadow: 0 10px 30px rgba(0, 0, 0, .5);
--chip-l: 70%; --chip-l: 70%;
--src-audit: #a983e6;
--src-ls: #3cc3cc;
--src-tint: 9%;
--sev-crit: #fb7185; --sev-crit: #fb7185;
--sev-err: #f87171; --sev-err: #f87171;
--sev-warning: #fbbf24; --sev-warning: #fbbf24;
@@ -102,6 +108,9 @@
--hit: rgba(129, 140, 248, .28); --hit: rgba(129, 140, 248, .28);
--shadow: 0 10px 30px rgba(0, 0, 0, .5); --shadow: 0 10px 30px rgba(0, 0, 0, .5);
--chip-l: 70%; --chip-l: 70%;
--src-audit: #a983e6;
--src-ls: #3cc3cc;
--src-tint: 9%;
--sev-crit: #fb7185; --sev-crit: #fb7185;
--sev-err: #f87171; --sev-err: #f87171;
--sev-warning: #fbbf24; --sev-warning: #fbbf24;
@@ -180,6 +189,16 @@ svg { fill: none; stroke: currentColor; stroke-width: 2; stroke-linecap: round;
.chip.on { color: var(--accent); border-color: var(--accent); background: var(--accent-soft); } .chip.on { color: var(--accent); border-color: var(--accent); background: var(--accent-soft); }
.actions { display: flex; align-items: center; gap: 6px; margin-left: auto; } .actions { display: flex; align-items: center; gap: 6px; margin-left: auto; }
/* Logged-in user, with how they logged in (oidc / local) */
.whoami { display: flex; align-items: center; gap: 6px; min-width: 0; margin-left: 6px; font-size: 13px; color: var(--muted); }
.whoami .who-name { font-weight: 600; color: var(--text); max-width: 18ch; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.whoami .who-badge {
flex: none; border: 1px solid var(--border); border-radius: 6px;
padding: 1px 6px; font-size: 10.5px; font-weight: 650; letter-spacing: .02em;
}
.whoami .who-badge.oidc { color: var(--accent); border-color: var(--accent); background: var(--accent-soft); }
.whoami .who-badge:empty { display: none; }
.whoami.none .who-name { font-weight: 400; color: var(--muted); }
.icon-btn { .icon-btn {
width: 36px; height: 36px; display: grid; place-items: center; width: 36px; height: 36px; display: grid; place-items: center;
border: 1px solid transparent; background: transparent; border-radius: 9px; color: var(--muted); border: 1px solid transparent; background: transparent; border-radius: 9px; color: var(--muted);
@@ -240,6 +259,19 @@ body.busy .progress::after {
font-size: 13px; max-width: 220px; font-size: 13px; max-width: 220px;
} }
.filters select.set { border-color: var(--accent); background-color: var(--accent-soft); } .filters select.set { border-color: var(--accent); background-color: var(--accent-soft); }
/* Display mode: Stream or Time range, then the start → end fields of the range */
.view-mode { padding: 2px; gap: 2px; border-radius: 9px; }
.view-mode button { display: inline-flex; align-items: center; gap: 6px; height: 26px; padding: 0 10px; }
.view-mode svg { width: 15px; height: 15px; }
.period { display: inline-flex; align-items: center; gap: 6px; }
.period .btn.tool { width: 32px; padding: 0; justify-content: center; }
.period input {
height: 32px; padding: 0 8px;
border: 1px solid var(--accent); border-radius: 8px; background: var(--accent-soft);
font-size: 13px; font-variant-numeric: tabular-nums; color-scheme: light dark;
}
.period input:focus { outline: 0; box-shadow: 0 0 0 3px var(--accent-soft); }
.period input.bad { border-color: var(--sev-err); background: color-mix(in srgb, var(--sev-err) 10%, transparent); }
.spacer { flex: 1; } .spacer { flex: 1; }
#count { font-size: 12.5px; font-variant-numeric: tabular-nums; } #count { font-size: 12.5px; font-variant-numeric: tabular-nums; }
@@ -388,6 +420,25 @@ body.col-resizing, body.col-resizing * { cursor: col-resize !important; user-sel
.row.sev-crit, .row.sev-err, .row.sev-warning { border-left-color: var(--sev); } .row.sev-crit, .row.sev-err, .row.sev-warning { border-left-color: var(--sev); }
.row.sev-crit, .row.sev-err { background: color-mix(in srgb, var(--sev) 7%, transparent); } .row.sev-crit, .row.sev-err { background: color-mix(in srgb, var(--sev) 7%, transparent); }
.row .host[data-act], .row .app[data-act] { cursor: pointer; } .row .host[data-act], .row .app[data-act] { cursor: pointer; }
/* Audit events and LogStream internal logs: colored edge (over the severity
one), light tint, badge before the app name. */
.row.src-audit { --src: var(--src-audit); }
.row.src-logstream { --src: var(--src-ls); }
.row.src-audit, .row.src-logstream { border-left-color: var(--src); background: color-mix(in srgb, var(--src) var(--src-tint), transparent); }
.row.src-audit:hover, .row.src-logstream:hover { background: color-mix(in srgb, var(--src) 13%, var(--row-hover)); }
.src-badge {
display: inline-block; margin-right: 5px; padding: 0 4px; border-radius: 4px; vertical-align: 1px;
font: 700 9.5px/1.45 var(--sans); letter-spacing: .04em;
color: var(--src); background: color-mix(in srgb, var(--src) 16%, transparent);
box-shadow: inset 0 0 0 1px color-mix(in srgb, var(--src) 40%, transparent);
}
.row .raw {
margin-top: 2px; font-size: .85em; line-height: 1.35; color: var(--muted);
white-space: pre-wrap; word-break: break-all;
}
:root[data-density="compact"] .row:not(.open) .raw { display: none; }
.src-help { margin-top: 10px !important; }
.self-level { width: auto; justify-self: start; }
/* Codes of the tags found in the message: grey badges, fixed size */ /* Codes of the tags found in the message: grey badges, fixed size */
.row .codes { white-space: nowrap; overflow: hidden; } .row .codes { white-space: nowrap; overflow: hidden; }
.row .codes > span { display: inline-flex; gap: 3px; vertical-align: 1px; } .row .codes > span { display: inline-flex; gap: 3px; vertical-align: 1px; }
@@ -487,7 +538,7 @@ dialog.settings {
} }
dialog.settings .dlg { display: flex; flex-direction: column; height: 100%; padding: 0; } dialog.settings .dlg { display: flex; flex-direction: column; height: 100%; padding: 0; }
dialog.settings .dlg > header { padding: 14px 16px 14px 22px; border-bottom: 1px solid var(--border); } dialog.settings .dlg > header { padding: 14px 16px 14px 22px; border-bottom: 1px solid var(--border); }
.set-body { flex: 1; min-height: 0; display: grid; grid-template-columns: 176px minmax(0, 1fr); } .set-body { flex: 1; min-height: 0; display: grid; grid-template-columns: minmax(176px, max-content) minmax(0, 1fr); }
.set-nav { .set-nav {
display: flex; flex-direction: column; gap: 2px; display: flex; flex-direction: column; gap: 2px;
padding: 12px 10px; border-right: 1px solid var(--border); padding: 12px 10px; border-right: 1px solid var(--border);
@@ -644,7 +695,7 @@ input.switch:checked::after { transform: translateX(14px); }
input.switch:disabled { cursor: not-allowed; } input.switch:disabled { cursor: not-allowed; }
input.switch:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; } input.switch:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
/* Settings > Filters: toolbar kept in view while the list scrolls, dense rows. */ /* Settings > Pattern: toolbar kept in view while the list scrolls, dense rows. */
.tags-section { padding-top: 12px !important; } .tags-section { padding-top: 12px !important; }
.tag-toolbar { .tag-toolbar {
position: sticky; top: -16px; z-index: 2; position: sticky; top: -16px; z-index: 2;
@@ -709,8 +760,16 @@ input.switch:focus-visible { outline: 2px solid var(--accent); outline-offset: 2
.topbar { flex-wrap: wrap; padding: 8px 16px; gap: 8px 10px; } .topbar { flex-wrap: wrap; padding: 8px 16px; gap: 8px 10px; }
.search { order: 3; flex-basis: 100%; max-width: none; } .search { order: 3; flex-basis: 100%; max-width: none; }
.live .lbl { display: none; } .live .lbl { display: none; }
.whoami .who-name { max-width: 12ch; }
.filters { padding: 10px 16px 6px; } .filters { padding: 10px 16px 6px; }
.filters select { flex: 1 1 calc(50% - 8px); max-width: none; min-width: 0; } .filters select { flex: 1 1 calc(50% - 8px); max-width: none; min-width: 0; }
.view-mode { flex: 1 1 100%; }
.view-mode button { flex: 1; justify-content: center; }
/* Phones: the two dates on one line, the ◀ ▶ − buttons below */
.period { flex: 1 1 100%; flex-wrap: wrap; }
.period > span { display: none; }
.period input { flex: 1 1 calc(50% - 3px); min-width: 0; }
.period .btn.tool { order: 1; flex: 1; }
.spacer { display: none; } .spacer { display: none; }
#count { flex: 1 1 50%; min-width: 0; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; } #count { flex: 1 1 50%; min-width: 0; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
.histo { padding: 0 16px 6px; } .histo { padding: 0 16px 6px; }
@@ -746,7 +805,7 @@ input.switch:focus-visible { outline: 2px solid var(--accent); outline-offset: 2
flex-direction: column; justify-content: center; height: auto; padding: 6px 2px; gap: 3px; flex-direction: column; justify-content: center; height: auto; padding: 6px 2px; gap: 3px;
font-size: 11.5px; text-align: center; font-size: 11.5px; text-align: center;
} }
.set-nav button span { max-width: 100%; overflow: hidden; text-overflow: ellipsis; } .set-nav button span { max-width: 100%; white-space: normal; line-height: 1.15; overflow: hidden; text-overflow: ellipsis; }
.set-panels { padding: 12px 12px 18px; } .set-panels { padding: 12px 12px 18px; }
.set-panel { grid-template-columns: minmax(0, 1fr); gap: 10px; } .set-panel { grid-template-columns: minmax(0, 1fr); gap: 10px; }
.set-panel > .set-section, .set-panel > .set-section + .set-section { padding: 12px; } .set-panel > .set-section, .set-panel > .set-section + .set-section { padding: 12px; }