Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
392fc645ce | ||
|
|
81806a6991 | ||
|
|
6abf1f47fd |
No files matched your search
@@ -19,9 +19,3 @@ EXPORT_MAX=100000
|
|||||||
DOCKER_LOGS=on
|
DOCKER_LOGS=on
|
||||||
# History read from a container seen for the first time (e.g. 30m, 1h, 24h; 0 = only new lines)
|
# History read from a container seen for the first time (e.g. 30m, 1h, 24h; 0 = only new lines)
|
||||||
DOCKER_BACKFILL=1h
|
DOCKER_BACKFILL=1h
|
||||||
# Host system logs (enable them in Settings > Sources)
|
|
||||||
# Group allowed to read the host logs: 4 = adm on Debian/Ubuntu; or the systemd-journal group
|
|
||||||
# (getent group systemd-journal | cut -d: -f3)
|
|
||||||
HOST_LOGS_GID=4
|
|
||||||
# History read when the source is turned on (e.g. 30m, 1h, 24h; 0 = only new entries)
|
|
||||||
HOST_LOGS_BACKFILL=1h
|
|
||||||
-344
@@ -1,344 +0,0 @@
|
|||||||
[English](README.md) | Français
|
|
||||||
|
|
||||||
# Logstream
|
|
||||||
|
|
||||||
Un collecteur syslog simple : il reçoit les logs en UDP/TCP sur le port 514, les stocke dans
|
|
||||||
[VictoriaLogs](https://docs.victoriametrics.com/victorialogs/) et sert une interface web
|
|
||||||
soignée (thème clair/sombre, recherche instantanée, direct, tags de couleur, interface en
|
|
||||||
anglais et en français).
|
|
||||||
|
|
||||||
```
|
|
||||||
devices ──514 udp/tcp──▶ logstream (Go) ──HTTP batches──▶ VictoriaLogs
|
|
||||||
▲ └── SSE (live) ──▶ browser
|
|
||||||
└──── API / UI ◀─────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||

|
|
||||||
|
|
||||||
- **Ingestion** : les messages syslog (UDP/TCP) et les logs des conteneurs Docker passent tous
|
|
||||||
par `sink()` (résolution DNS inverse des hôtes donnés par leur IP), puis par la file du
|
|
||||||
`Store`, qui les envoie par lots à VictoriaLogs.
|
|
||||||
- **Direct** : `sink()` publie aussi chaque message dans le `Hub`, qui le diffuse aux
|
|
||||||
navigateurs en SSE.
|
|
||||||
- **Recherche** : l'API HTTP traduit les filtres de l'interface en requêtes LogsQL envoyées à
|
|
||||||
VictoriaLogs.
|
|
||||||
- **Frise** : `/api/histogram` (`histogram.go`) compte les logs par intervalle et par sévérité,
|
|
||||||
alignés sur l'heure locale ; les bornes du zoom (`from`/`to`) s'appliquent aussi à la liste
|
|
||||||
et à l'export.
|
|
||||||
- **État** : les tags et les réglages des sources sont dans `/data` (volume `logstream-data`) ;
|
|
||||||
les logs eux-mêmes dans le volume `vlogs-data`.
|
|
||||||
|
|
||||||
La source modifiable du schéma est
|
|
||||||
[`docs/architecture.excalidraw`](docs/architecture.excalidraw)
|
|
||||||
(à ouvrir sur [excalidraw.com](https://excalidraw.com)).
|
|
||||||
|
|
||||||
## Démarrage
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cp .env.example .env # optional
|
|
||||||
docker compose up -d --build
|
|
||||||
./tools/send-test-logs.sh # sends 100 test messages
|
|
||||||
```
|
|
||||||
|
|
||||||
Ouvrez ensuite <http://localhost:8080>.
|
|
||||||
|
|
||||||
## Envoyer des logs
|
|
||||||
|
|
||||||
- **rsyslog** (Linux) : ajoutez `*.* @SERVER_IP:514` (UDP) ou `*.* @@SERVER_IP:514` (TCP)
|
|
||||||
dans `/etc/rsyslog.d/90-logstream.conf`, puis lancez `systemctl restart rsyslog`.
|
|
||||||
- **Équipements réseau, NAS, pare-feu** : indiquez l'IP du serveur et le port 514 dans leurs
|
|
||||||
réglages « syslog distant » (remote syslog).
|
|
||||||
- **Test manuel** : `logger -n 127.0.0.1 -P 514 -d "hello error"` (util-linux) ou
|
|
||||||
`echo "<14>test ok" | nc -u -w1 127.0.0.1 514`.
|
|
||||||
|
|
||||||
**Paramètres > Sources > Syslog** active ou désactive la réception syslog et choisit les
|
|
||||||
protocoles (UDP, TCP) sans redémarrage ; le choix est enregistré dans `/data/syslog.json`. Cet
|
|
||||||
onglet affiche aussi l'état d'écoute (et l'erreur si le port est déjà utilisé). Le port
|
|
||||||
lui-même est publié par docker-compose : modifiez `SYSLOG_PORT` dans `.env`, puis lancez
|
|
||||||
`docker compose up -d`.
|
|
||||||
|
|
||||||
Formats pris en charge : RFC 3164 (BSD) et RFC 5424. En TCP, les deux découpages sont acceptés :
|
|
||||||
« un message par ligne » et « octet counting » (RFC 6587).
|
|
||||||
|
|
||||||
## Recherche
|
|
||||||
|
|
||||||
**Mode simple** (par défaut) : chaque mot est cherché comme sous-chaîne, sans tenir compte de
|
|
||||||
la casse, dans le message, l'hôte et l'application. Les mots sont combinés avec ET.
|
|
||||||
|
|
||||||
| Saisie | Signification |
|
|
||||||
|---|---|
|
|
||||||
| `error disk` | contient « error » **et** « disk » |
|
|
||||||
| `"disk full"` | contient la phrase exacte |
|
|
||||||
| `error -timeout` | contient « error » mais pas « timeout » |
|
|
||||||
|
|
||||||
**Mode LogsQL** (bouton `Simple` / `LogsQL`) : le langage de requête complet de VictoriaLogs,
|
|
||||||
par exemple `error AND host:web-01`, `app:~"ssh|nginx"` ou `* | stats by (host) count()`.
|
|
||||||
Le direct est désactivé dans ce mode.
|
|
||||||
|
|
||||||
Chaque ligne affiche, de gauche à droite : l'**heure de réception** (horloge du serveur),
|
|
||||||
l'horodatage trouvé dans le message lui-même (`msg_time`), la sévérité, l'hôte, l'application
|
|
||||||
et le message. Un clic sur un hôte ou une application filtre dessus.
|
|
||||||
|
|
||||||
Les logs sont indexés, recherchés et triés par **heure de réception** : les équipements dont
|
|
||||||
l'horloge est fausse (par exemple des points d'accès dont le NTP échoue) apparaissent quand même
|
|
||||||
dans la bonne plage de temps. Les logs stockés par les versions antérieures à ce changement sont
|
|
||||||
indexés par l'horodatage de leur message ; purgez-les depuis les Paramètres pour repartir sur
|
|
||||||
une base propre.
|
|
||||||
|
|
||||||
Les badges de sévérité `err`/`crit` et `warning` reprennent les couleurs des tags `error` et
|
|
||||||
`warning`.
|
|
||||||
|
|
||||||
Raccourcis : `/` place le curseur dans la recherche, `Esc` la vide. Un clic sur une ligne
|
|
||||||
affiche tous ses champs.
|
|
||||||
|
|
||||||
Les en-têtes de colonnes restent visibles pendant le défilement. Faites glisser le bord d'un
|
|
||||||
en-tête (réception, heure message, sévérité, hôte, app) pour redimensionner la colonne, et
|
|
||||||
double-cliquez dessus pour revenir à la largeur automatique ; le message occupe l'espace
|
|
||||||
restant. Les largeurs sont mémorisées par le navigateur (**Paramètres > Interface >
|
|
||||||
Réinitialiser les colonnes** les rétablit toutes). Sur téléphone, la liste garde sa
|
|
||||||
présentation sur deux lignes, sans colonnes.
|
|
||||||
|
|
||||||
## Frise
|
|
||||||
|
|
||||||
La frise au-dessus de la liste montre le volume de logs par intervalle, compté selon l'**heure
|
|
||||||
de réception** sur l'horloge du serveur (elle correspond donc aux heures affichées dans les
|
|
||||||
lignes).
|
|
||||||
|
|
||||||
- Au survol d'un intervalle : ses bornes, son total et le détail par sévérité.
|
|
||||||
- Un clic sur une barre zoome sur cet intervalle ; un glisser sur plusieurs barres zoome sur la
|
|
||||||
sélection. La plage de temps affiche alors la période zoomée (« × Annuler le zoom » ou le
|
|
||||||
choix d'une autre plage en sort) ; la liste, les compteurs et l'export CSV suivent le zoom,
|
|
||||||
et le direct se met en pause.
|
|
||||||
- En direct, le dernier intervalle grandit à l'arrivée des messages, et la frise se recharge à
|
|
||||||
chaque nouvel intervalle. Rien n'est rafraîchi tant que l'onglet du navigateur est masqué ;
|
|
||||||
la frise se met à jour dès qu'il redevient visible.
|
|
||||||
|
|
||||||
**Paramètres > Interface > Frise** (mémorisé par navigateur) : échelle (linéaire, √ par défaut,
|
|
||||||
log), hauteur (S/M/L : 40/80/120 px), couleur (empilement par sévérité, intensité comparée à
|
|
||||||
la médiane de la fenêtre : calme, rafale au-delà de 3×, anomalie au-delà de 10×, ou aucune),
|
|
||||||
barres ou aire, division (automatique, environ 100 intervalles, ou fixe : 1 s, 10 s, 1 min,
|
|
||||||
5 min, 1 h, 1 jour) et rafraîchissement (désactivé, 5 s, 15 s, 30 s, 1 min, ou à chaque nouvel
|
|
||||||
intervalle). Une division fixe qui dépasserait 300 intervalles sur la plage est élargie
|
|
||||||
(signalé par « élargi »). Les intervalles sont alignés sur l'heure locale du fuseau horaire
|
|
||||||
choisi dans les Paramètres (les jours commencent à minuit, heure locale).
|
|
||||||
|
|
||||||
API : `curl 'localhost:8080/api/histogram?range=24h&step=auto&tz=Europe/Paris'` renvoie
|
|
||||||
`step` (ms), `start` (ms), `count`, `now` (horloge du serveur) et les `buckets` non vides
|
|
||||||
(`i` = numéro d'intervalle, `n` = total, `sev` = nombre par sévérité). Toutes les routes de
|
|
||||||
logs acceptent aussi `from` / `to` (millisecondes Unix) à la place de `range`.
|
|
||||||
|
|
||||||
## Logs des conteneurs Docker
|
|
||||||
|
|
||||||
Logstream collecte aussi les logs des conteneurs Docker qui tournent sur la machine où il est
|
|
||||||
installé (`DOCKER_LOGS=on`, le défaut dans `docker-compose.yml`). Ils sont recherchés, filtrés,
|
|
||||||
colorés et exportés comme les messages syslog :
|
|
||||||
|
|
||||||
- **host** est le nom de l'hôte Docker, **app** le service compose (ou le nom du conteneur), et
|
|
||||||
chaque log porte aussi `container`, `container_id`, `image`, `compose_project`,
|
|
||||||
`compose_service` et `stream` (stdout/stderr), visibles dans le détail de la ligne.
|
|
||||||
- Le filtre **Source** n'affiche que les logs syslog ou que les logs Docker ; les lignes Docker
|
|
||||||
ont un petit cube devant le nom de l'application, de la couleur de son projet compose.
|
|
||||||
- La sévérité vient de la ligne elle-même quand l'application l'écrit : JSON
|
|
||||||
(`"level":"error"`), logfmt (`level=warn`), `[ERROR]`, ou un niveau en majuscules au début de
|
|
||||||
la ligne (`ERROR`, `WARN`…). Sinon, elle vaut `info`. Les codes de couleur du terminal sont
|
|
||||||
supprimés.
|
|
||||||
- **Paramètres > Sources** affiche une étiquette par conteneur (`project/service`) dans un seul
|
|
||||||
champ : les conteneurs suivis en couleur, puis les non suivis en gris ; un clic bascule une
|
|
||||||
étiquette. La couleur identifie le projet compose, et les noms d'applications Docker de la
|
|
||||||
liste utilisent la même couleur. Les conteneurs arrêtés sont masqués par défaut (« Afficher
|
|
||||||
les conteneurs arrêtés » les montre, en pointillés, et ils restent modifiables). Une zone de
|
|
||||||
filtre et « Tout activer » / « Tout désactiver » (appliqués aux étiquettes affichées) aident
|
|
||||||
quand les conteneurs sont nombreux. Les nouveaux conteneurs sont suivis automatiquement, sauf
|
|
||||||
si cette option est désactivée. Les choix sont enregistrés par service compose (ou nom de
|
|
||||||
conteneur) dans `/data/docker.json` : ils survivent aux recréations.
|
|
||||||
- Logstream mémorise la position lue dans chaque conteneur (`/data/docker-state.json`) : après
|
|
||||||
un redémarrage, il reprend sans perdre ni dupliquer de lignes. Un conteneur vu pour la
|
|
||||||
première fois est lu à partir de `DOCKER_BACKFILL` en arrière (1 heure par défaut).
|
|
||||||
- Logstream lui-même et le proxy ci-dessous ne sont jamais collectés ; ajoutez l'étiquette
|
|
||||||
`logstream.exclude=true` à tout autre conteneur pour l'exclure définitivement.
|
|
||||||
|
|
||||||
**Sécurité** : l'accès au socket Docker équivaut à un accès root sur la machine. Logstream passe
|
|
||||||
donc par [docker-socket-proxy](https://github.com/Tecnativa/docker-socket-proxy), qui ne laisse
|
|
||||||
passer que la liste des conteneurs, la lecture des logs, les événements et les informations du
|
|
||||||
moteur (`GET` uniquement).
|
|
||||||
|
|
||||||
## Logs système de l'hôte
|
|
||||||
|
|
||||||
Logstream peut aussi collecter les logs système de la machine qui héberge la stack, sans rien
|
|
||||||
configurer sur l'hôte. La source est **désactivée par défaut** : activez-la dans
|
|
||||||
**Paramètres > Sources > Logs système de l'hôte** (enregistré dans `/data/hostlogs.json`).
|
|
||||||
|
|
||||||
- `docker-compose.yml` monte `/var/log` et `/run/log/journal` en lecture seule sous `/host`.
|
|
||||||
- Si l'hôte utilise systemd, Logstream lit directement les fichiers du **journal systemd**
|
|
||||||
(pas besoin de `journalctl` dans l'image) : **host** est le nom de la machine, **app** le
|
|
||||||
programme (`SYSLOG_IDENTIFIER`), la sévérité et la facility viennent du journal, et l'unité
|
|
||||||
systemd est conservée dans `unit`. Sinon, il suit les fichiers texte de `/var/log` (`syslog`,
|
|
||||||
`messages`, `*.log`), analysés comme des lignes syslog, avec le nom du fichier dans `log_file`.
|
|
||||||
- Ces logs ont la source `host` : le filtre **Source** permet de les afficher seuls.
|
|
||||||
- À l'activation, la dernière heure est lue d'abord (`HOST_LOGS_BACKFILL`) ; la position
|
|
||||||
atteinte est enregistrée dans `/data/hostlogs-state.json`, un redémarrage ne perd donc ni ne
|
|
||||||
duplique d'entrées.
|
|
||||||
- **Droits** : le conteneur tourne avec un utilisateur sans privilège et reçoit le groupe `adm`
|
|
||||||
(gid 4), qui peut lire le journal et `/var/log` sur Debian et Ubuntu. Sur d'autres systèmes,
|
|
||||||
réglez `HOST_LOGS_GID` dans `.env` sur le gid de `systemd-journal`
|
|
||||||
(`getent group systemd-journal | cut -d: -f3`). Paramètres > Sources affiche un message clair
|
|
||||||
si l'accès est refusé.
|
|
||||||
- Limites : les champs du journal compressés par journald (messages de plus de 512 octets,
|
|
||||||
compressés en zstd/lz4/xz) ne peuvent pas être décodés sans bibliothèque supplémentaire ; ils
|
|
||||||
sont comptés dans les Paramètres et affichés comme « (compressed journal entry) ». Les fichiers
|
|
||||||
texte tournés ou compressés (`*.1`, `*.gz`) ne sont pas lus.
|
|
||||||
|
|
||||||
## Export CSV
|
|
||||||
|
|
||||||
Le bouton **Exporter** (à côté du nombre de logs) télécharge tous les logs stockés qui
|
|
||||||
correspondent aux filtres en cours (recherche, plage de temps, sévérité, hôte, application),
|
|
||||||
du plus récent au plus ancien, jusqu'à `EXPORT_MAX` lignes (100 000 par défaut), et pas
|
|
||||||
seulement les lignes à l'écran. Deux variantes :
|
|
||||||
|
|
||||||
- **CSV** : séparateur virgule, UTF-8.
|
|
||||||
- **CSV pour Excel** : séparateur point-virgule avec un BOM UTF-8, pour qu'Excel en français
|
|
||||||
l'ouvre directement avec les accents. Les cellules qui commencent par `=`, `+`, `-` ou `@`
|
|
||||||
sont préfixées par `'` pour qu'un message de log forgé ne puisse pas s'exécuter comme une
|
|
||||||
formule.
|
|
||||||
|
|
||||||
Colonnes : `received`, `message_time` (toutes deux au format `YYYY-MM-DD HH:MM:SS.mmm` dans le
|
|
||||||
fuseau horaire choisi dans les Paramètres), `severity`, `facility`, `host`, `host_ip`, `app`,
|
|
||||||
`pid`, `source_ip`, `proto`, `message`. En mode LogsQL, seule la partie filtre est prise en
|
|
||||||
charge (pas de `| pipes`).
|
|
||||||
|
|
||||||
En ligne de commande : `curl -o logs.csv 'localhost:8080/api/export.csv?q=error&range=24h&tz=Europe/Paris'`.
|
|
||||||
|
|
||||||
## Paramètres
|
|
||||||
|
|
||||||
L'icône en forme d'engrenage ouvre les paramètres, organisés en onglets. Tout, sauf les tags de
|
|
||||||
couleur, est mémorisé par navigateur.
|
|
||||||
|
|
||||||
- **Localisation**
|
|
||||||
- *Langue* : anglais ou français.
|
|
||||||
- *Date et heure* : fuseau horaire (celui du navigateur, UTC ou environ 80 fuseaux courants)
|
|
||||||
et format d'affichage de l'heure de réception : `DD/MM/YYYY HH:MM:SS` (par défaut,
|
|
||||||
l'affichage français habituel), avec millisecondes, `YYYY-MM-DD`, horloge sur 12 heures,
|
|
||||||
ISO 8601 ou epoch Unix. Le fuseau horaire s'applique à toutes les dates affichées.
|
|
||||||
- **Filtres** : tags de couleur. Chaque tag a un mot-clé, une couleur de fond (le texte passe
|
|
||||||
automatiquement en noir ou en blanc pour rester lisible) et des options : mot entier, respect
|
|
||||||
de la casse, expression régulière, actif. Les tags sont stockés sur le serveur dans
|
|
||||||
`/data/tags.json` (volume `logstream-data`) : ils sont donc partagés par tous les navigateurs.
|
|
||||||
Tags par défaut (pastel) : `warning` (orange), `error` (rouge), `ok` (vert). Les tags par
|
|
||||||
défaut qui utilisent encore les couleurs des versions précédentes passent automatiquement aux
|
|
||||||
couleurs pastel.
|
|
||||||
- **Interface**
|
|
||||||
- *Thème* : Système (suit la préférence de l'ordinateur ou du téléphone), Clair ou Sombre. Le
|
|
||||||
bouton soleil/lune de l'en-tête bascule entre clair et sombre.
|
|
||||||
- *Affichage des logs* : taille du texte (très petite, petite, moyenne, grande) et police :
|
|
||||||
la police monospace du système, ou l'une des 12 polices libres conçues pour le texte dense
|
|
||||||
(JetBrains Mono, Fira Code, Source Code Pro, IBM Plex Mono, Cascadia Code, Roboto Mono,
|
|
||||||
Ubuntu Mono, Inconsolata, Red Hat Mono, Noto Sans Mono, Victor Mono, DM Mono). Elles sont
|
|
||||||
chargées par le navigateur depuis [Bunny Fonts](https://fonts.bunny.net), un service
|
|
||||||
européen de polices respectueux de la vie privée ; sans accès à internet, la police du
|
|
||||||
système est utilisée. Les ligatures sont désactivées pour que `->` ou `!=` s'affichent tels
|
|
||||||
quels.
|
|
||||||
- **Données** : « Supprimer tous les logs » efface définitivement tous les logs stockés (il faut
|
|
||||||
taper `PURGE` pour confirmer). Les tags et les paramètres sont conservés. VictoriaLogs doit
|
|
||||||
être lancé avec `-delete.enable` (déjà présent dans `docker-compose.yml`) ; mettez
|
|
||||||
`ALLOW_PURGE=false` pour désactiver la fonction. Toute personne qui peut ouvrir l'interface
|
|
||||||
peut purger : définissez `AUTH_USER` / `AUTH_PASS` si l'interface est accessible à d'autres.
|
|
||||||
|
|
||||||
## Noms d'hôtes (DNS inverse)
|
|
||||||
|
|
||||||
Quand un équipement envoie son adresse IP comme nom d'hôte (ou pas de nom d'hôte du tout),
|
|
||||||
Logstream cherche son nom DNS (enregistrement PTR) et stocke le nom dans `host` et l'IP dans
|
|
||||||
`host_ip`. Les résultats sont mis en cache (1 heure, 10 minutes quand il n'y a pas de nom). Les
|
|
||||||
logs stockés auparavant avec une IP sont résolus à l'affichage, et le filtre d'hôte affiche
|
|
||||||
`name (IP)`.
|
|
||||||
|
|
||||||
Le conteneur utilise le DNS de Docker, qui relaie vers les résolveurs de l'hôte. Si vos noms
|
|
||||||
locaux ne sont connus que de votre routeur ou d'un DNS local (Pi-hole, AdGuard, Unbound…),
|
|
||||||
définissez `DNS_SERVER=192.168.1.1` (son adresse). Mettez `RDNS=off` pour désactiver les
|
|
||||||
résolutions.
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
| Variable | Défaut | Rôle |
|
|
||||||
|---|---|---|
|
|
||||||
| `SYSLOG_PORT` | `514` | port syslog publié sur l'hôte |
|
|
||||||
| `HTTP_PORT` | `8080` | port de l'interface web |
|
|
||||||
| `RETENTION` | `30d` | durée de conservation des logs dans VictoriaLogs |
|
|
||||||
| `AUTH_USER` / `AUTH_PASS` | vide | authentification HTTP Basic pour l'interface |
|
|
||||||
| `RDNS` | `on` | résoudre les hôtes donnés par leur IP en noms DNS |
|
|
||||||
| `DNS_SERVER` | vide | serveur DNS pour les résolutions inverses (`ip` ou `ip:port`) |
|
|
||||||
| `ALLOW_PURGE` | `true` | autoriser « Supprimer tous les logs » dans les Paramètres |
|
|
||||||
| `EXPORT_MAX` | `100000` | nombre maximal de lignes dans un export CSV |
|
|
||||||
| `DOCKER_LOGS` | `on` dans compose | collecter les logs des conteneurs Docker locaux |
|
|
||||||
| `DOCKER_HOST` | `tcp://docker-proxy:2375` dans compose | adresse de l'API Docker (`unix:///var/run/docker.sock` hors compose) |
|
|
||||||
| `DOCKER_BACKFILL` | `1h` | historique lu pour un conteneur vu pour la première fois |
|
|
||||||
| `HOST_LOGS_GID` | `4` (adm) dans compose | groupe donné au conteneur pour lire les logs de l'hôte |
|
|
||||||
| `HOST_LOGS_BACKFILL` | `1h` | historique lu à l'activation de la source « logs système de l'hôte » |
|
|
||||||
| `HOST_LOGS_ROOT` | `/host` | emplacement de montage des répertoires de l'hôte |
|
|
||||||
| `TZ` | `Europe/Paris` | fuseau horaire des horodatages RFC 3164 (qui n'en portent pas) |
|
|
||||||
| `BATCH_SIZE`, `FLUSH_MS`, `QUEUE_SIZE` | `1000`, `1000`, `100000` | réglage de l'ingestion |
|
|
||||||
|
|
||||||
## Débogage
|
|
||||||
|
|
||||||
- `docker compose logs -f logstream` : erreurs de réception et erreurs d'envoi vers
|
|
||||||
VictoriaLogs.
|
|
||||||
- La barre du bas affiche les compteurs reçus / stockés / perdus et la dernière erreur de
|
|
||||||
stockage.
|
|
||||||
- <http://localhost:9428/select/vmui> : l'interface de VictoriaLogs, pour essayer des requêtes
|
|
||||||
LogsQL.
|
|
||||||
- API :
|
|
||||||
```bash
|
|
||||||
curl 'localhost:8080/api/logs?q=error&range=1h&limit=5' # the response includes the generated LogsQL query
|
|
||||||
curl localhost:8080/api/stats
|
|
||||||
curl localhost:8080/api/tags
|
|
||||||
```
|
|
||||||
- Lancement hors Docker (Go 1.22+) : `VLOGS_URL=http://localhost:9428 DATA_DIR=./data SYSLOG_ADDR=:5514 go run .`
|
|
||||||
|
|
||||||
## Mise à jour
|
|
||||||
|
|
||||||
Toutes les versions d'images sont épinglées : un `docker compose pull` ou une reconstruction ne
|
|
||||||
change jamais un composant à votre insu.
|
|
||||||
|
|
||||||
| Emplacement | Image | Version |
|
|
||||||
|---|---|---|
|
|
||||||
| `docker-compose.yml` | `victoriametrics/victoria-logs` | `v1.52.0` |
|
|
||||||
| `docker-compose.yml` | `tecnativa/docker-socket-proxy` | `v0.5.0` |
|
|
||||||
| `Dockerfile` (build) | `golang` | `1.27.1-alpine3.24` |
|
|
||||||
| `Dockerfile` (exécution) | `alpine` | `3.24.2` |
|
|
||||||
|
|
||||||
Pour mettre à jour l'une d'elles :
|
|
||||||
|
|
||||||
1. Lisez les notes de version : [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/changelog/),
|
|
||||||
[docker-socket-proxy](https://github.com/Tecnativa/docker-socket-proxy/releases),
|
|
||||||
[Go](https://go.dev/doc/devel/release), [Alpine](https://alpinelinux.org/releases/).
|
|
||||||
VictoriaLogs conserve son format de stockage entre versions mineures ; lisez le changelog
|
|
||||||
avant un changement de version majeure.
|
|
||||||
2. Modifiez la version dans le fichier indiqué ci-dessus, puis lancez `docker compose up -d --build`.
|
|
||||||
3. Vérifiez la barre du bas (reçus / stockés / perdus) et **Paramètres > Sources**. Pour revenir
|
|
||||||
en arrière, remettez la version précédente et lancez la même commande.
|
|
||||||
|
|
||||||
## Organisation du code
|
|
||||||
|
|
||||||
| Fichier | Contenu |
|
|
||||||
|---|---|
|
|
||||||
| `main.go` | configuration, démarrage, authentification |
|
|
||||||
| `syslog.go` | écoute UDP/TCP et analyse RFC 3164 / 5424 |
|
|
||||||
| `store.go` | insertions par lots dans VictoriaLogs et requêtes LogsQL |
|
|
||||||
| `query.go` | traduit les filtres de l'interface en LogsQL ; filtre du direct |
|
|
||||||
| `histogram.go` | frise : division, alignement des intervalles, `/api/histogram` |
|
|
||||||
| `hub.go` | envoie les nouveaux messages aux navigateurs (SSE) |
|
|
||||||
| `rdns.go` | résolutions DNS inverses avec cache |
|
|
||||||
| `export.go` | export CSV en flux |
|
|
||||||
| `docker.go` | logs des conteneurs Docker (API, lecteurs, positions, détection du niveau) |
|
|
||||||
| `syslogserver.go` | écoutes syslog ouvertes et fermées depuis Paramètres > Sources |
|
|
||||||
| `hostlogs.go`, `journal.go` | logs système de l'hôte : lecteur du journal systemd (sans `journalctl`) et suivi de `/var/log` |
|
|
||||||
| `tags.go` | stockage des tags de couleur |
|
|
||||||
| `api.go` | routes HTTP `/api/*` |
|
|
||||||
| `web/` | interface (HTML, CSS, JavaScript simple, sans étape de build), embarquée dans le binaire ; les traductions sont dans `web/app.js` (`I18N`) |
|
|
||||||
|
|
||||||
## Remarque
|
|
||||||
|
|
||||||
Avec Docker Desktop (macOS/Windows), l'IP source vue par le conteneur pour les paquets UDP est
|
|
||||||
la passerelle Docker. Le champ `host` vient toujours de l'en-tête syslog, qui porte normalement
|
|
||||||
le vrai nom de l'émetteur.
|
|
||||||
@@ -1,5 +1,3 @@
|
|||||||
English | [Français](README.fr.md)
|
|
||||||
|
|
||||||
# Logstream
|
# Logstream
|
||||||
|
|
||||||
A simple syslog sink: receives logs over UDP/TCP on port 514, stores them in
|
A simple syslog sink: receives logs over UDP/TCP on port 514, stores them in
|
||||||
@@ -14,20 +12,18 @@ devices ──514 udp/tcp──▶ logstream (Go) ──HTTP batches──▶ Vi
|
|||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
- **Ingestion**: syslog (UDP/TCP) and Docker container logs both go through `sink()` (reverse DNS
|
- **Ingestion**: syslog (UDP/TCP) and Docker container logs both go through `sink()` (reverse DNS
|
||||||
on IP hosts), then the `Store` queue, which sends them in batches to VictoriaLogs.
|
on IP hosts), then the `Store` queue, which sends them in batches to VictoriaLogs.
|
||||||
- **Live view**: `sink()` also publishes each message to the `Hub`, which streams it to the
|
- **Live view**: `sink()` also publishes each message to the `Hub`, which streams it to the
|
||||||
browsers over SSE.
|
browsers over SSE.
|
||||||
- **Search**: the HTTP API turns the UI filters into LogsQL queries sent to VictoriaLogs.
|
- **Search**: the HTTP API turns the UI filters into LogsQL queries sent to VictoriaLogs.
|
||||||
- **Timeline**: `/api/histogram` (`histogram.go`) counts the logs per interval and severity,
|
|
||||||
aligned on the local time; the zoom bounds (`from`/`to`) also apply to the list and the export.
|
|
||||||
- **State**: tags and source settings live in `/data` (`logstream-data` volume); the logs
|
- **State**: tags and source settings live in `/data` (`logstream-data` volume); the logs
|
||||||
themselves in the `vlogs-data` volume.
|
themselves in the `vlogs-data` volume.
|
||||||
|
|
||||||
The editable source of the diagram is
|
The editable source of the diagram is
|
||||||
[`docs/architecture.excalidraw`](docs/architecture.excalidraw)
|
[`docs/logstream-schema-logique.excalidraw`](docs/logstream-schema-logique.excalidraw)
|
||||||
(open it on [excalidraw.com](https://excalidraw.com)).
|
(open it on [excalidraw.com](https://excalidraw.com)).
|
||||||
|
|
||||||
## Getting started
|
## Getting started
|
||||||
@@ -147,32 +143,6 @@ colored and exported like syslog messages:
|
|||||||
therefore goes through [docker-socket-proxy](https://github.com/Tecnativa/docker-socket-proxy),
|
therefore goes through [docker-socket-proxy](https://github.com/Tecnativa/docker-socket-proxy),
|
||||||
which only lets through listing containers, reading logs, events and engine info (`GET` only).
|
which only lets through listing containers, reading logs, events and engine info (`GET` only).
|
||||||
|
|
||||||
## Host system logs
|
|
||||||
|
|
||||||
Logstream can also collect the system logs of the machine hosting the stack, without
|
|
||||||
configuring anything on the host. The source is **off by default**: turn it on in
|
|
||||||
**Settings > Sources > Host system logs** (saved in `/data/hostlogs.json`).
|
|
||||||
|
|
||||||
- `docker-compose.yml` mounts `/var/log` and `/run/log/journal` read-only under `/host`.
|
|
||||||
- When the host runs systemd, Logstream reads the **systemd journal** files directly (no
|
|
||||||
`journalctl` needed in the image): **host** is the machine name, **app** the program
|
|
||||||
(`SYSLOG_IDENTIFIER`), severity and facility come from the journal, and the systemd unit is
|
|
||||||
kept in `unit`. Otherwise it follows the text files of `/var/log` (`syslog`, `messages`,
|
|
||||||
`*.log`), parsed like syslog lines, with the file name in `log_file`.
|
|
||||||
- These logs have the `host` source: the **Source** filter shows them alone.
|
|
||||||
- When the source is turned on, the last hour is read first (`HOST_LOGS_BACKFILL`); the
|
|
||||||
position reached is saved in `/data/hostlogs-state.json`, so a restart neither loses nor
|
|
||||||
duplicates entries.
|
|
||||||
- **Permissions**: the container runs as an unprivileged user and gets the `adm` group
|
|
||||||
(gid 4), which can read the journal and `/var/log` on Debian and Ubuntu. On other systems,
|
|
||||||
set `HOST_LOGS_GID` in `.env` to the gid of `systemd-journal`
|
|
||||||
(`getent group systemd-journal | cut -d: -f3`). Settings > Sources shows a clear message when
|
|
||||||
access is denied.
|
|
||||||
- Limits: journal fields compressed by journald (messages longer than 512 bytes, compressed
|
|
||||||
with zstd/lz4/xz) cannot be decoded without extra libraries; they are counted in Settings and
|
|
||||||
shown as "(compressed journal entry)". Rotated or compressed text files (`*.1`, `*.gz`) are
|
|
||||||
not read.
|
|
||||||
|
|
||||||
## CSV export
|
## CSV export
|
||||||
|
|
||||||
The **Export** button (next to the log count) downloads every stored log matching the
|
The **Export** button (next to the log count) downloads every stored log matching the
|
||||||
@@ -232,6 +202,11 @@ on display, and the host filter shows `name (IP)`.
|
|||||||
The container uses Docker's DNS, which forwards to the host's resolvers. If your local names
|
The container uses Docker's DNS, which forwards to the host's resolvers. If your local names
|
||||||
are only known by your router or a local DNS (Pi-hole, AdGuard, Unbound…), set
|
are only known by your router or a local DNS (Pi-hole, AdGuard, Unbound…), set
|
||||||
`DNS_SERVER=192.168.1.1` (its address). Set `RDNS=off` to disable lookups.
|
`DNS_SERVER=192.168.1.1` (its address). Set `RDNS=off` to disable lookups.
|
||||||
|
- **Color tags**: each tag has a keyword, a background color (the text automatically
|
||||||
|
switches to black or white to stay readable) and options: whole word, match case,
|
||||||
|
regular expression, active. Tags are stored in `/data/tags.json` (`logstream-data` volume).
|
||||||
|
Default tags (pastel): `warning` (orange), `error` (red), `ok` (green). Default tags
|
||||||
|
still using the colors of earlier versions are switched to the pastel ones automatically.
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
@@ -248,9 +223,6 @@ are only known by your router or a local DNS (Pi-hole, AdGuard, Unbound…), set
|
|||||||
| `DOCKER_LOGS` | `on` in compose | collect the logs of the local Docker containers |
|
| `DOCKER_LOGS` | `on` in compose | collect the logs of the local Docker containers |
|
||||||
| `DOCKER_HOST` | `tcp://docker-proxy:2375` in compose | Docker API address (`unix:///var/run/docker.sock` outside compose) |
|
| `DOCKER_HOST` | `tcp://docker-proxy:2375` in compose | Docker API address (`unix:///var/run/docker.sock` outside compose) |
|
||||||
| `DOCKER_BACKFILL` | `1h` | history read from a container seen for the first time |
|
| `DOCKER_BACKFILL` | `1h` | history read from a container seen for the first time |
|
||||||
| `HOST_LOGS_GID` | `4` (adm) in compose | group given to the container to read the host logs |
|
|
||||||
| `HOST_LOGS_BACKFILL` | `1h` | history read when the host system logs source is turned on |
|
|
||||||
| `HOST_LOGS_ROOT` | `/host` | where the host directories are mounted |
|
|
||||||
| `TZ` | `Europe/Paris` | time zone for RFC 3164 timestamps (which carry none) |
|
| `TZ` | `Europe/Paris` | time zone for RFC 3164 timestamps (which carry none) |
|
||||||
| `BATCH_SIZE`, `FLUSH_MS`, `QUEUE_SIZE` | `1000`, `1000`, `100000` | ingestion tuning |
|
| `BATCH_SIZE`, `FLUSH_MS`, `QUEUE_SIZE` | `1000`, `1000`, `100000` | ingestion tuning |
|
||||||
|
|
||||||
@@ -304,7 +276,6 @@ To update one of them:
|
|||||||
| `export.go` | streamed CSV export |
|
| `export.go` | streamed CSV export |
|
||||||
| `docker.go` | Docker container logs (API, followers, positions, level detection) |
|
| `docker.go` | Docker container logs (API, followers, positions, level detection) |
|
||||||
| `syslogserver.go` | syslog listeners opened and closed from Settings > Sources |
|
| `syslogserver.go` | syslog listeners opened and closed from Settings > Sources |
|
||||||
| `hostlogs.go`, `journal.go` | host system logs: systemd journal reader (no `journalctl`) and `/var/log` follower |
|
|
||||||
| `tags.go` | color tag storage |
|
| `tags.go` | color tag storage |
|
||||||
| `api.go` | `/api/*` HTTP routes |
|
| `api.go` | `/api/*` HTTP routes |
|
||||||
| `web/` | UI (HTML, CSS, plain JavaScript, no build step), embedded in the binary; translations live in `web/app.js` (`I18N`) |
|
| `web/` | UI (HTML, CSS, plain JavaScript, no build step), embedded in the binary; translations live in `web/app.js` (`I18N`) |
|
||||||
|
|||||||
@@ -21,7 +21,6 @@ type API struct {
|
|||||||
exportMax int
|
exportMax int
|
||||||
docker *DockerManager // nil when DOCKER_LOGS is off
|
docker *DockerManager // nil when DOCKER_LOGS is off
|
||||||
syslog *SyslogServer
|
syslog *SyslogServer
|
||||||
host *HostLogs
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (a *API) Routes(mux *http.ServeMux) {
|
func (a *API) Routes(mux *http.ServeMux) {
|
||||||
@@ -43,28 +42,6 @@ func (a *API) Routes(mux *http.ServeMux) {
|
|||||||
mux.HandleFunc("PUT /api/syslog", a.syslogConfigure)
|
mux.HandleFunc("PUT /api/syslog", a.syslogConfigure)
|
||||||
mux.HandleFunc("GET /api/docker", a.dockerStatus)
|
mux.HandleFunc("GET /api/docker", a.dockerStatus)
|
||||||
mux.HandleFunc("PUT /api/docker", a.dockerConfigure)
|
mux.HandleFunc("PUT /api/docker", a.dockerConfigure)
|
||||||
mux.HandleFunc("GET /api/hostlogs", a.hostLogsStatus)
|
|
||||||
mux.HandleFunc("PUT /api/hostlogs", a.hostLogsConfigure)
|
|
||||||
}
|
|
||||||
|
|
||||||
// GET /api/hostlogs: state of the host system logs source (Settings > Sources).
|
|
||||||
func (a *API) hostLogsStatus(w http.ResponseWriter, r *http.Request) {
|
|
||||||
writeJSON(w, http.StatusOK, a.host.Status())
|
|
||||||
}
|
|
||||||
|
|
||||||
// PUT /api/hostlogs {"enabled": bool}
|
|
||||||
func (a *API) hostLogsConfigure(w http.ResponseWriter, r *http.Request) {
|
|
||||||
var cfg hostLogsConfig
|
|
||||||
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&cfg); err != nil {
|
|
||||||
writeErr(w, http.StatusBadRequest, err)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if err := a.host.Configure(cfg); err != nil {
|
|
||||||
writeErr(w, http.StatusInternalServerError, err)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
log.Printf("host system logs changed from %s: %+v", r.RemoteAddr, cfg)
|
|
||||||
writeJSON(w, http.StatusOK, a.host.Status())
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// GET /api/syslog: syslog reception state (Settings > Sources).
|
// GET /api/syslog: syslog reception state (Settings > Sources).
|
||||||
|
|||||||
+15
-18
@@ -12,29 +12,24 @@ services:
|
|||||||
- "${HTTP_PORT:-8080}:8080"
|
- "${HTTP_PORT:-8080}:8080"
|
||||||
environment:
|
environment:
|
||||||
VLOGS_URL: http://victorialogs:9428
|
VLOGS_URL: http://victorialogs:9428
|
||||||
SYSLOG_PUBLIC_PORT: ${SYSLOG_PORT:-514} # le port d'ecoute syslog par defaut (attention aux ports <1024)
|
SYSLOG_PUBLIC_PORT: ${SYSLOG_PORT:-514} # shown in Settings > Sources (the mapping is set above)
|
||||||
TZ: ${TZ:-Europe/Paris}
|
TZ: ${TZ:-Europe/Paris}
|
||||||
AUTH_USER: ${AUTH_USER:-} # vide = pas d'authentification, on delegue ca au reverse proxy traefik
|
AUTH_USER: ${AUTH_USER:-} # leave empty to disable authentication
|
||||||
AUTH_PASS: ${AUTH_PASS:-}
|
AUTH_PASS: ${AUTH_PASS:-}
|
||||||
RDNS: ${RDNS:-on} # resol dns
|
RDNS: ${RDNS:-on} # replace IP hosts with their DNS name (PTR)
|
||||||
DNS_SERVER: ${DNS_SERVER:-} # si resolv directe
|
DNS_SERVER: ${DNS_SERVER:-} # e.g. 192.168.1.1 to query your LAN DNS; empty = system resolver
|
||||||
ALLOW_PURGE: ${ALLOW_PURGE:-true}
|
ALLOW_PURGE: ${ALLOW_PURGE:-true}
|
||||||
EXPORT_MAX: ${EXPORT_MAX:-100000}
|
EXPORT_MAX: ${EXPORT_MAX:-100000}
|
||||||
DOCKER_LOGS: ${DOCKER_LOGS:-on} # collecte des logs des conteneurs Docker
|
DOCKER_LOGS: ${DOCKER_LOGS:-on} # collect the logs of this machine's containers
|
||||||
DOCKER_HOST: tcp://docker-proxy:2375 # lecture seul de l'API Docker
|
DOCKER_HOST: tcp://docker-proxy:2375 # read-only Docker API gateway (below)
|
||||||
DOCKER_BACKFILL: ${DOCKER_BACKFILL:-1h}
|
DOCKER_BACKFILL: ${DOCKER_BACKFILL:-1h} # history read from a container seen for the first time
|
||||||
HOST_LOGS_BACKFILL: ${HOST_LOGS_BACKFILL:-1h} # historique lu a l'activation des logs systeme de l'hote
|
|
||||||
group_add:
|
|
||||||
- "${HOST_LOGS_GID:-4}" # groupe autorise a lire les logs de l'hote (4 = adm sur Debian/Ubuntu)
|
|
||||||
volumes:
|
volumes:
|
||||||
- logstream-data:/data # tags.json, docker.json (les choix des conteneurs)
|
- logstream-data:/data # tags.json, docker.json (container choices)
|
||||||
# logs systeme de l'hote, en lecture seule (source a activer dans Reglages > Sources)
|
|
||||||
- /var/log:/host/var/log:ro # journal systemd persistant et fichiers texte
|
|
||||||
- /run/log/journal:/host/run/log/journal:ro # journal systemd volatile
|
|
||||||
labels:
|
labels:
|
||||||
logstream.exclude: "true" # pas de collect des logs logstream
|
logstream.exclude: "true" # never collect Logstream's own logs
|
||||||
|
|
||||||
victorialogs:
|
victorialogs:
|
||||||
|
# Pinned version: see "Updating" in the README before changing it
|
||||||
image: victoriametrics/victoria-logs:v1.52.0
|
image: victoriametrics/victoria-logs:v1.52.0
|
||||||
container_name: logstream-victorialogs
|
container_name: logstream-victorialogs
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
@@ -42,7 +37,7 @@ services:
|
|||||||
- -storageDataPath=/vlogs
|
- -storageDataPath=/vlogs
|
||||||
- -retentionPeriod=${RETENTION:-30d}
|
- -retentionPeriod=${RETENTION:-30d}
|
||||||
- -httpListenAddr=:9428
|
- -httpListenAddr=:9428
|
||||||
- -delete.enable # obliger pour autoriser la purge de la base via l'interface
|
- -delete.enable # required by "Delete all logs" in Settings
|
||||||
volumes:
|
volumes:
|
||||||
- vlogs-data:/vlogs
|
- vlogs-data:/vlogs
|
||||||
ports:
|
ports:
|
||||||
@@ -50,7 +45,9 @@ services:
|
|||||||
- "127.0.0.1:9428:9428"
|
- "127.0.0.1:9428:9428"
|
||||||
|
|
||||||
docker-proxy:
|
docker-proxy:
|
||||||
# Docker proxy pour eviter de solliciter directement l'API docker
|
# Read-only gateway to the Docker API: Logstream can only list containers,
|
||||||
|
# read their logs, receive events and engine info. Anything else (start,
|
||||||
|
# stop, exec, images, volumes…) is refused.
|
||||||
image: tecnativa/docker-socket-proxy:v0.5.0
|
image: tecnativa/docker-socket-proxy:v0.5.0
|
||||||
container_name: logstream-docker-proxy
|
container_name: logstream-docker-proxy
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
@@ -62,7 +59,7 @@ services:
|
|||||||
volumes:
|
volumes:
|
||||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||||
labels:
|
labels:
|
||||||
logstream.exclude: "true" # pour exclure les logs propres de logstream
|
logstream.exclude: "true" # its access log would only echo Logstream's own requests
|
||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
logstream-data:
|
logstream-data:
|
||||||
|
|||||||
File diff suppressed because it is too large.
Load diff
Binary file not shown.
|
Before Width: | Height: | Size: 884 KiB |
File diff suppressed because it is too large.
Load diff
Binary file not shown.
|
After Width: | Height: | Size: 597 KiB |
-452
@@ -1,452 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"context"
|
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
|
||||||
"io"
|
|
||||||
"io/fs"
|
|
||||||
"log"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"sort"
|
|
||||||
"strconv"
|
|
||||||
"strings"
|
|
||||||
"sync"
|
|
||||||
"syscall"
|
|
||||||
"time"
|
|
||||||
"unicode/utf8"
|
|
||||||
)
|
|
||||||
|
|
||||||
// hostLogsConfig is saved in /data/hostlogs.json and edited in Settings > Sources.
|
|
||||||
type hostLogsConfig struct {
|
|
||||||
Enabled bool `json:"enabled"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// hostPos is where reading resumes in a journal file (by file ID) or a text
|
|
||||||
// file (by name, with its inode to detect rotations). Off -1: archived file
|
|
||||||
// read to the end.
|
|
||||||
type hostPos struct {
|
|
||||||
Off int64 `json:"off"`
|
|
||||||
Ino uint64 `json:"ino,omitempty"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// HostLogs collects the system logs of the machine hosting the stack: the
|
|
||||||
// systemd journal when there is one, else the text files of /var/log. The
|
|
||||||
// host directories are mounted read-only under root (/host by default).
|
|
||||||
type HostLogs struct {
|
|
||||||
root string
|
|
||||||
sink func(*Entry)
|
|
||||||
backfill time.Duration
|
|
||||||
cfgPath string
|
|
||||||
statePath string
|
|
||||||
wake chan struct{}
|
|
||||||
|
|
||||||
// Owned by the Run goroutine.
|
|
||||||
pos map[string]hostPos
|
|
||||||
dirty bool
|
|
||||||
started bool // a first scan was done since enabling: new files are read from their start
|
|
||||||
|
|
||||||
mu sync.Mutex
|
|
||||||
cfg hostLogsConfig
|
|
||||||
reset bool
|
|
||||||
mode string // "journal", "files" or "" (nothing found)
|
|
||||||
files int
|
|
||||||
read uint64
|
|
||||||
compressed uint64
|
|
||||||
errCode string
|
|
||||||
errDetail string
|
|
||||||
}
|
|
||||||
|
|
||||||
func NewHostLogs(root, dataDir string, backfill time.Duration, sink func(*Entry)) *HostLogs {
|
|
||||||
h := &HostLogs{
|
|
||||||
root: root,
|
|
||||||
sink: sink,
|
|
||||||
backfill: backfill,
|
|
||||||
cfgPath: filepath.Join(dataDir, "hostlogs.json"),
|
|
||||||
statePath: filepath.Join(dataDir, "hostlogs-state.json"),
|
|
||||||
wake: make(chan struct{}, 1),
|
|
||||||
pos: map[string]hostPos{},
|
|
||||||
}
|
|
||||||
if b, err := os.ReadFile(h.cfgPath); err == nil {
|
|
||||||
_ = json.Unmarshal(b, &h.cfg)
|
|
||||||
}
|
|
||||||
if b, err := os.ReadFile(h.statePath); err == nil {
|
|
||||||
_ = json.Unmarshal(b, &h.pos)
|
|
||||||
h.started = len(h.pos) > 0
|
|
||||||
}
|
|
||||||
return h
|
|
||||||
}
|
|
||||||
|
|
||||||
// Run polls the host logs every second while the source is enabled.
|
|
||||||
func (h *HostLogs) Run(ctx context.Context) {
|
|
||||||
tick := time.NewTicker(time.Second)
|
|
||||||
defer tick.Stop()
|
|
||||||
n := 0
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
h.saveState()
|
|
||||||
return
|
|
||||||
case <-tick.C:
|
|
||||||
case <-h.wake:
|
|
||||||
}
|
|
||||||
h.mu.Lock()
|
|
||||||
enabled, reset := h.cfg.Enabled, h.reset
|
|
||||||
h.reset = false
|
|
||||||
h.mu.Unlock()
|
|
||||||
if reset {
|
|
||||||
// Disabled then enabled again: start over from HOST_LOGS_BACKFILL ago
|
|
||||||
// rather than reading everything written in between.
|
|
||||||
h.pos, h.started, h.dirty = map[string]hostPos{}, false, true
|
|
||||||
}
|
|
||||||
if enabled {
|
|
||||||
h.scan(time.Now())
|
|
||||||
}
|
|
||||||
if n++; n%5 == 0 || reset {
|
|
||||||
h.saveState()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (h *HostLogs) saveState() {
|
|
||||||
if !h.dirty {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
h.dirty = false
|
|
||||||
if err := writeJSONFile(h.statePath, h.pos); err != nil {
|
|
||||||
log.Printf("host logs: saving positions: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (h *HostLogs) setStatus(mode string, files int, code, detail string) {
|
|
||||||
h.mu.Lock()
|
|
||||||
h.mode, h.files, h.errCode, h.errDetail = mode, files, code, detail
|
|
||||||
h.mu.Unlock()
|
|
||||||
}
|
|
||||||
|
|
||||||
// scan reads what was added since the previous poll.
|
|
||||||
func (h *HostLogs) scan(now time.Time) {
|
|
||||||
notBefore := time.Time{}
|
|
||||||
if !h.started {
|
|
||||||
notBefore = now.Add(-h.backfill)
|
|
||||||
}
|
|
||||||
defer func() { h.started = true }()
|
|
||||||
|
|
||||||
var journals []string
|
|
||||||
for _, dir := range []string{"var/log/journal", "run/log/journal"} {
|
|
||||||
base := filepath.Join(h.root, dir)
|
|
||||||
for _, pat := range []string{"*.journal", "*/*.journal"} {
|
|
||||||
m, _ := filepath.Glob(filepath.Join(base, pat))
|
|
||||||
journals = append(journals, m...)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if len(journals) > 0 {
|
|
||||||
h.scanJournals(journals, notBefore)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
h.scanFiles(notBefore.IsZero())
|
|
||||||
}
|
|
||||||
|
|
||||||
func (h *HostLogs) scanJournals(paths []string, notBefore time.Time) {
|
|
||||||
seen := map[string]bool{}
|
|
||||||
var firstErr error
|
|
||||||
for _, p := range paths {
|
|
||||||
f, err := os.Open(p)
|
|
||||||
if err != nil {
|
|
||||||
firstErr = keepFirst(firstErr, err)
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
hdr, err := readJournalHeader(f)
|
|
||||||
f.Close()
|
|
||||||
if err != nil {
|
|
||||||
continue // file being created, or not a journal
|
|
||||||
}
|
|
||||||
key := "j:" + hdr.fileID
|
|
||||||
seen[key] = true
|
|
||||||
pos, known := h.pos[key]
|
|
||||||
if known && pos.Off < 0 {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
from, nb := pos.Off, time.Time{}
|
|
||||||
if !known {
|
|
||||||
if hdr.archived && !notBefore.IsZero() && time.UnixMicro(int64(hdr.tailRealtimeUS)).Before(notBefore) {
|
|
||||||
h.pos[key], h.dirty = hostPos{Off: -1}, true // archived before the backfill window
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
nb = notBefore
|
|
||||||
}
|
|
||||||
next, hdr, err := readJournal(p, from, nb, h.emitJournal)
|
|
||||||
if err != nil {
|
|
||||||
firstErr = keepFirst(firstErr, err)
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if hdr.archived {
|
|
||||||
next = -1
|
|
||||||
}
|
|
||||||
if !known || next != pos.Off {
|
|
||||||
h.pos[key], h.dirty = hostPos{Off: next}, true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
h.prune("j:", seen)
|
|
||||||
code, detail := "", ""
|
|
||||||
if firstErr != nil && len(seen) == 0 {
|
|
||||||
code, detail = errCode(firstErr), firstErr.Error()
|
|
||||||
}
|
|
||||||
h.setStatus("journal", len(seen), code, detail)
|
|
||||||
}
|
|
||||||
|
|
||||||
func keepFirst(first, err error) error {
|
|
||||||
if first != nil {
|
|
||||||
return first
|
|
||||||
}
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
func errCode(err error) string {
|
|
||||||
if errors.Is(err, fs.ErrPermission) {
|
|
||||||
return "permission"
|
|
||||||
}
|
|
||||||
return "read"
|
|
||||||
}
|
|
||||||
|
|
||||||
func (h *HostLogs) prune(prefix string, seen map[string]bool) {
|
|
||||||
for k := range h.pos {
|
|
||||||
if strings.HasPrefix(k, prefix) && !seen[k] {
|
|
||||||
delete(h.pos, k)
|
|
||||||
h.dirty = true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// hostLogFile reports the classic text logs of /var/log (rotated and
|
|
||||||
// compressed copies are left out).
|
|
||||||
func hostLogFile(name string) bool {
|
|
||||||
return name == "syslog" || name == "messages" || strings.HasSuffix(name, ".log")
|
|
||||||
}
|
|
||||||
|
|
||||||
const maxHostRead = 4 << 20 // per file and per poll
|
|
||||||
|
|
||||||
// scanFiles follows the text files of /var/log, for hosts without journald.
|
|
||||||
// Files present at the first scan are read from their end (only new lines).
|
|
||||||
func (h *HostLogs) scanFiles(fromStart bool) {
|
|
||||||
dir := filepath.Join(h.root, "var/log")
|
|
||||||
ents, err := os.ReadDir(dir)
|
|
||||||
if err != nil {
|
|
||||||
code := errCode(err)
|
|
||||||
if errors.Is(err, fs.ErrNotExist) {
|
|
||||||
code = "not_mounted"
|
|
||||||
}
|
|
||||||
h.setStatus("", 0, code, err.Error())
|
|
||||||
return
|
|
||||||
}
|
|
||||||
seen := map[string]bool{}
|
|
||||||
var firstErr error
|
|
||||||
for _, de := range ents {
|
|
||||||
if !de.Type().IsRegular() || !hostLogFile(de.Name()) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
name := de.Name()
|
|
||||||
key := "f:" + name
|
|
||||||
fi, err := de.Info()
|
|
||||||
if err != nil {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
var ino uint64
|
|
||||||
if st, ok := fi.Sys().(*syscall.Stat_t); ok {
|
|
||||||
ino = uint64(st.Ino)
|
|
||||||
}
|
|
||||||
pos, known := h.pos[key]
|
|
||||||
switch {
|
|
||||||
case !known && !fromStart:
|
|
||||||
pos = hostPos{Off: fi.Size(), Ino: ino}
|
|
||||||
case !known || pos.Ino != ino || fi.Size() < pos.Off:
|
|
||||||
pos = hostPos{Off: 0, Ino: ino} // new, rotated or truncated file
|
|
||||||
}
|
|
||||||
if fi.Size() > pos.Off {
|
|
||||||
off, err := h.readFile(filepath.Join(dir, name), name, pos.Off)
|
|
||||||
if err != nil {
|
|
||||||
firstErr = keepFirst(firstErr, err)
|
|
||||||
continue // not readable: not counted as followed
|
|
||||||
}
|
|
||||||
pos.Off = off
|
|
||||||
}
|
|
||||||
seen[key] = true
|
|
||||||
if old, ok := h.pos[key]; !ok || old != pos {
|
|
||||||
h.pos[key], h.dirty = pos, true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
h.prune("f:", seen)
|
|
||||||
code, detail := "", ""
|
|
||||||
if firstErr != nil && len(seen) == 0 {
|
|
||||||
code, detail = errCode(firstErr), firstErr.Error()
|
|
||||||
}
|
|
||||||
mode := "files"
|
|
||||||
if len(seen) == 0 && code == "" {
|
|
||||||
mode, code = "", "empty"
|
|
||||||
}
|
|
||||||
h.setStatus(mode, len(seen), code, detail)
|
|
||||||
}
|
|
||||||
|
|
||||||
// readFile sends the complete lines written after off and returns the new offset.
|
|
||||||
func (h *HostLogs) readFile(path, name string, off int64) (int64, error) {
|
|
||||||
f, err := os.Open(path)
|
|
||||||
if err != nil {
|
|
||||||
return off, err
|
|
||||||
}
|
|
||||||
defer f.Close()
|
|
||||||
buf, err := io.ReadAll(io.NewSectionReader(f, off, maxHostRead))
|
|
||||||
if err != nil {
|
|
||||||
return off, err
|
|
||||||
}
|
|
||||||
end := bytes.LastIndexByte(buf, '\n')
|
|
||||||
if end < 0 {
|
|
||||||
if len(buf) < maxHostRead {
|
|
||||||
return off, nil // partial line: wait for its end
|
|
||||||
}
|
|
||||||
end = len(buf) - 1 // line longer than the read window: cut it
|
|
||||||
}
|
|
||||||
now := time.Now()
|
|
||||||
fac := fileFacility(name)
|
|
||||||
for _, line := range bytes.Split(buf[:end+1], []byte{'\n'}) {
|
|
||||||
if len(bytes.TrimSpace(line)) == 0 {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
e := ParseSyslog(line, "", "file", now)
|
|
||||||
if e.Host == "" {
|
|
||||||
e.Host = "localhost"
|
|
||||||
}
|
|
||||||
if n, ok := detectLevel(e.Message); ok {
|
|
||||||
e.SevNum, e.Severity = n, severityNames[n]
|
|
||||||
} else {
|
|
||||||
e.SevNum, e.Severity = 6, "info"
|
|
||||||
}
|
|
||||||
if fac >= 0 {
|
|
||||||
e.Facility = facilityNames[fac]
|
|
||||||
}
|
|
||||||
e.SourceType = "host"
|
|
||||||
e.Extra = map[string]string{"log_file": "/var/log/" + name}
|
|
||||||
h.sink(e)
|
|
||||||
h.count(1, 0)
|
|
||||||
}
|
|
||||||
return off + int64(end) + 1, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// fileFacility guesses the facility from the usual Debian/RHEL file names.
|
|
||||||
func fileFacility(name string) int {
|
|
||||||
switch strings.TrimSuffix(name, ".log") {
|
|
||||||
case "kern":
|
|
||||||
return 0
|
|
||||||
case "mail", "maillog":
|
|
||||||
return 2
|
|
||||||
case "daemon":
|
|
||||||
return 3
|
|
||||||
case "auth", "secure":
|
|
||||||
return 4
|
|
||||||
case "cron":
|
|
||||||
return 9
|
|
||||||
}
|
|
||||||
return -1
|
|
||||||
}
|
|
||||||
|
|
||||||
func (h *HostLogs) count(read, compressed uint64) {
|
|
||||||
h.mu.Lock()
|
|
||||||
h.read += read
|
|
||||||
h.compressed += compressed
|
|
||||||
h.mu.Unlock()
|
|
||||||
}
|
|
||||||
|
|
||||||
// emitJournal turns a journal entry into an Entry.
|
|
||||||
func (h *HostLogs) emitJournal(je *journalEntry) {
|
|
||||||
f := je.Fields
|
|
||||||
msg := f["MESSAGE"]
|
|
||||||
if msg == "" && je.Compressed > 0 {
|
|
||||||
msg = "(compressed journal entry: read it with journalctl)"
|
|
||||||
}
|
|
||||||
h.count(1, uint64(je.Compressed))
|
|
||||||
if strings.TrimSpace(msg) == "" {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if !utf8.ValidString(msg) {
|
|
||||||
msg = strings.ToValidUTF8(msg, "�")
|
|
||||||
}
|
|
||||||
sev := 6
|
|
||||||
if n, err := strconv.Atoi(f["PRIORITY"]); err == nil && n >= 0 && n <= 7 {
|
|
||||||
sev = n
|
|
||||||
}
|
|
||||||
fac := 1 // user
|
|
||||||
switch {
|
|
||||||
case f["_TRANSPORT"] == "kernel":
|
|
||||||
fac = 0
|
|
||||||
case f["_SYSTEMD_UNIT"] != "":
|
|
||||||
fac = 3 // daemon
|
|
||||||
}
|
|
||||||
if n, err := strconv.Atoi(f["SYSLOG_FACILITY"]); err == nil && n >= 0 && n < len(facilityNames) {
|
|
||||||
fac = n
|
|
||||||
}
|
|
||||||
app := firstNonEmpty(f["SYSLOG_IDENTIFIER"], f["_COMM"], strings.TrimSuffix(f["_SYSTEMD_UNIT"], ".service"))
|
|
||||||
host := firstNonEmpty(f["_HOSTNAME"], "localhost")
|
|
||||||
h.sink(&Entry{
|
|
||||||
Time: je.Realtime,
|
|
||||||
Received: time.Now(),
|
|
||||||
Host: host,
|
|
||||||
App: app,
|
|
||||||
ProcID: firstNonEmpty(f["SYSLOG_PID"], f["_PID"]),
|
|
||||||
Facility: facilityNames[fac],
|
|
||||||
Severity: severityNames[sev],
|
|
||||||
SevNum: sev,
|
|
||||||
Message: strings.TrimRight(msg, "\n"),
|
|
||||||
Proto: "journal",
|
|
||||||
SourceType: "host",
|
|
||||||
Extra: map[string]string{"unit": f["_SYSTEMD_UNIT"]},
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
func firstNonEmpty(vals ...string) string {
|
|
||||||
for _, v := range vals {
|
|
||||||
if v != "" {
|
|
||||||
return v
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
// Configure saves and applies the configuration.
|
|
||||||
func (h *HostLogs) Configure(cfg hostLogsConfig) error {
|
|
||||||
h.mu.Lock()
|
|
||||||
if h.cfg.Enabled && !cfg.Enabled {
|
|
||||||
h.reset = true
|
|
||||||
h.mode, h.files, h.errCode, h.errDetail = "", 0, "", ""
|
|
||||||
}
|
|
||||||
h.cfg = cfg
|
|
||||||
h.mu.Unlock()
|
|
||||||
select {
|
|
||||||
case h.wake <- struct{}{}:
|
|
||||||
default:
|
|
||||||
}
|
|
||||||
return writeJSONFile(h.cfgPath, cfg)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Status is the state shown in Settings > Sources.
|
|
||||||
func (h *HostLogs) Status() map[string]any {
|
|
||||||
h.mu.Lock()
|
|
||||||
defer h.mu.Unlock()
|
|
||||||
dirs := []string{}
|
|
||||||
for _, d := range []string{"var/log/journal", "run/log/journal", "var/log"} {
|
|
||||||
if _, err := os.Stat(filepath.Join(h.root, d)); err == nil {
|
|
||||||
dirs = append(dirs, "/"+d)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
sort.Strings(dirs)
|
|
||||||
return map[string]any{
|
|
||||||
"enabled": h.cfg.Enabled,
|
|
||||||
"mode": h.mode,
|
|
||||||
"files": h.files,
|
|
||||||
"read": h.read,
|
|
||||||
"compressed": h.compressed,
|
|
||||||
"code": h.errCode,
|
|
||||||
"error": h.errDetail,
|
|
||||||
"mounted": dirs,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,206 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"encoding/binary"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// fakeJournal builds a minimal journal file: header, data objects, entries.
|
|
||||||
type fakeJournal struct {
|
|
||||||
compact bool
|
|
||||||
buf []byte
|
|
||||||
seq uint64
|
|
||||||
tail uint64
|
|
||||||
}
|
|
||||||
|
|
||||||
func newFakeJournal(compact bool) *fakeJournal {
|
|
||||||
j := &fakeJournal{compact: compact, buf: make([]byte, 272)}
|
|
||||||
copy(j.buf, jSignature)
|
|
||||||
if compact {
|
|
||||||
binary.LittleEndian.PutUint32(j.buf[12:], jIncompatCompact)
|
|
||||||
}
|
|
||||||
j.buf[16] = 1 // online
|
|
||||||
copy(j.buf[24:40], "0123456789abcdef")
|
|
||||||
binary.LittleEndian.PutUint64(j.buf[88:], 272)
|
|
||||||
return j
|
|
||||||
}
|
|
||||||
|
|
||||||
func (j *fakeJournal) object(typ, flags byte, body []byte) uint64 {
|
|
||||||
for len(j.buf)%8 != 0 {
|
|
||||||
j.buf = append(j.buf, 0)
|
|
||||||
}
|
|
||||||
off := uint64(len(j.buf))
|
|
||||||
h := make([]byte, jObjHeaderSize)
|
|
||||||
h[0], h[1] = typ, flags
|
|
||||||
binary.LittleEndian.PutUint64(h[8:], uint64(jObjHeaderSize+len(body)))
|
|
||||||
j.buf = append(append(j.buf, h...), body...)
|
|
||||||
j.tail = off
|
|
||||||
binary.LittleEndian.PutUint64(j.buf[136:], off)
|
|
||||||
return off
|
|
||||||
}
|
|
||||||
|
|
||||||
func (j *fakeJournal) data(field string, flags byte) uint64 {
|
|
||||||
n := 48
|
|
||||||
if j.compact {
|
|
||||||
n = 56
|
|
||||||
}
|
|
||||||
return j.object(jObjData, flags, append(make([]byte, n), field...))
|
|
||||||
}
|
|
||||||
|
|
||||||
// entry appends an entry; linked=false leaves it unfinished (tail seqnum not updated).
|
|
||||||
func (j *fakeJournal) entry(t time.Time, linked bool, fields ...string) {
|
|
||||||
var items []byte
|
|
||||||
for _, f := range fields {
|
|
||||||
flags := byte(0)
|
|
||||||
if strings.HasPrefix(f, "!") { // compressed data object
|
|
||||||
f, flags = f[1:], 4
|
|
||||||
}
|
|
||||||
off := j.data(f, flags)
|
|
||||||
if j.compact {
|
|
||||||
items = binary.LittleEndian.AppendUint32(items, uint32(off))
|
|
||||||
} else {
|
|
||||||
items = binary.LittleEndian.AppendUint64(items, off)
|
|
||||||
items = binary.LittleEndian.AppendUint64(items, 0)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
j.seq++
|
|
||||||
body := make([]byte, 48)
|
|
||||||
binary.LittleEndian.PutUint64(body[0:], j.seq)
|
|
||||||
binary.LittleEndian.PutUint64(body[8:], uint64(t.UnixMicro()))
|
|
||||||
j.object(jObjEntry, 0, append(body, items...))
|
|
||||||
if linked {
|
|
||||||
binary.LittleEndian.PutUint64(j.buf[160:], j.seq)
|
|
||||||
binary.LittleEndian.PutUint64(j.buf[192:], uint64(t.UnixMicro()))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (j *fakeJournal) write(t *testing.T, path string) {
|
|
||||||
t.Helper()
|
|
||||||
if err := os.WriteFile(path, j.buf, 0o644); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestReadJournal(t *testing.T) {
|
|
||||||
for _, compact := range []bool{false, true} {
|
|
||||||
path := filepath.Join(t.TempDir(), "system.journal")
|
|
||||||
now := time.Now()
|
|
||||||
j := newFakeJournal(compact)
|
|
||||||
j.entry(now.Add(-2*time.Hour), true, "MESSAGE=too old", "PRIORITY=6")
|
|
||||||
j.entry(now.Add(-time.Minute), true, "MESSAGE=Started cron.", "PRIORITY=5", "SYSLOG_IDENTIFIER=systemd", "_HOSTNAME=srv1", "_PID=1")
|
|
||||||
j.write(t, path)
|
|
||||||
|
|
||||||
var got []*journalEntry
|
|
||||||
emit := func(e *journalEntry) { got = append(got, e) }
|
|
||||||
next, _, err := readJournal(path, 0, now.Add(-time.Hour), emit)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if len(got) != 1 || got[0].Fields["MESSAGE"] != "Started cron." || got[0].Fields["_HOSTNAME"] != "srv1" {
|
|
||||||
t.Fatalf("compact=%v: got %+v", compact, got)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A new complete entry and one still being written.
|
|
||||||
j.entry(now, true, "MESSAGE=second", "!MESSAGE=big")
|
|
||||||
j.entry(now, false, "MESSAGE=unfinished")
|
|
||||||
j.write(t, path)
|
|
||||||
got = nil
|
|
||||||
next2, _, err := readJournal(path, next, time.Time{}, emit)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if len(got) != 1 || got[0].Fields["MESSAGE"] != "second" || got[0].Compressed != 1 {
|
|
||||||
t.Fatalf("compact=%v: resumed read got %d entries", compact, len(got))
|
|
||||||
}
|
|
||||||
|
|
||||||
// Once linked, the unfinished entry is read from where reading stopped.
|
|
||||||
binary.LittleEndian.PutUint64(j.buf[160:], j.seq)
|
|
||||||
j.write(t, path)
|
|
||||||
got = nil
|
|
||||||
if _, _, err := readJournal(path, next2, time.Time{}, emit); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if len(got) != 1 || got[0].Fields["MESSAGE"] != "unfinished" {
|
|
||||||
t.Fatalf("compact=%v: unfinished entry got %+v", compact, got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestHostLogsJournal(t *testing.T) {
|
|
||||||
root := t.TempDir()
|
|
||||||
dir := filepath.Join(root, "var/log/journal/machine")
|
|
||||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
j := newFakeJournal(true)
|
|
||||||
j.entry(time.Now(), true, "MESSAGE=Accepted publickey", "PRIORITY=6", "SYSLOG_FACILITY=10", "SYSLOG_IDENTIFIER=sshd", "_PID=42", "_HOSTNAME=srv1", "_SYSTEMD_UNIT=ssh.service")
|
|
||||||
j.entry(time.Now(), true, "MESSAGE=oops", "PRIORITY=3", "_TRANSPORT=kernel", "_HOSTNAME=srv1")
|
|
||||||
j.write(t, filepath.Join(dir, "system.journal"))
|
|
||||||
|
|
||||||
var got []*Entry
|
|
||||||
h := NewHostLogs(root, t.TempDir(), time.Hour, func(e *Entry) { got = append(got, e) })
|
|
||||||
h.scan(time.Now())
|
|
||||||
if len(got) != 2 {
|
|
||||||
t.Fatalf("got %d entries", len(got))
|
|
||||||
}
|
|
||||||
e := got[0]
|
|
||||||
if e.App != "sshd" || e.ProcID != "42" || e.Facility != "authpriv" || e.Severity != "info" || e.Host != "srv1" || e.SourceType != "host" || e.Extra["unit"] != "ssh.service" {
|
|
||||||
t.Fatalf("entry %+v", e)
|
|
||||||
}
|
|
||||||
if got[1].Facility != "kern" || got[1].Severity != "err" {
|
|
||||||
t.Fatalf("kernel entry %+v", got[1])
|
|
||||||
}
|
|
||||||
h.scan(time.Now()) // nothing new
|
|
||||||
if len(got) != 2 {
|
|
||||||
t.Fatalf("re-read: %d entries", len(got))
|
|
||||||
}
|
|
||||||
if st := h.Status(); st["mode"] != "journal" || st["files"] != 1 {
|
|
||||||
t.Fatalf("status %+v", st)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestHostLogsFiles(t *testing.T) {
|
|
||||||
root := t.TempDir()
|
|
||||||
dir := filepath.Join(root, "var/log")
|
|
||||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
auth := filepath.Join(dir, "auth.log")
|
|
||||||
if err := os.WriteFile(auth, []byte("Oct 3 07:00:00 srv1 sshd[1]: old line\n"), 0o644); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
_ = os.WriteFile(filepath.Join(dir, "auth.log.1"), []byte("rotated\n"), 0o644)
|
|
||||||
|
|
||||||
var got []*Entry
|
|
||||||
h := NewHostLogs(root, t.TempDir(), time.Hour, func(e *Entry) { got = append(got, e) })
|
|
||||||
h.scan(time.Now()) // existing content is skipped
|
|
||||||
if len(got) != 0 {
|
|
||||||
t.Fatalf("first scan read %d lines", len(got))
|
|
||||||
}
|
|
||||||
f, _ := os.OpenFile(auth, os.O_APPEND|os.O_WRONLY, 0)
|
|
||||||
_, _ = f.WriteString("Oct 3 08:00:00 srv1 sshd[7]: Failed password for root\nOct 3 08:00:01 srv1 sshd[7]: partial")
|
|
||||||
f.Close()
|
|
||||||
h.scan(time.Now())
|
|
||||||
if len(got) != 1 {
|
|
||||||
t.Fatalf("got %d lines", len(got))
|
|
||||||
}
|
|
||||||
e := got[0]
|
|
||||||
if e.Host != "srv1" || e.App != "sshd" || e.ProcID != "7" || e.Facility != "auth" || e.Extra["log_file"] != "/var/log/auth.log" || e.Proto != "file" {
|
|
||||||
t.Fatalf("entry %+v", e)
|
|
||||||
}
|
|
||||||
if st := h.Status(); st["mode"] != "files" || st["files"] != 1 {
|
|
||||||
t.Fatalf("status %+v", st)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestHostLogsNotMounted(t *testing.T) {
|
|
||||||
h := NewHostLogs(filepath.Join(t.TempDir(), "none"), t.TempDir(), time.Hour, func(*Entry) {})
|
|
||||||
h.scan(time.Now())
|
|
||||||
if st := h.Status(); st["code"] != "not_mounted" {
|
|
||||||
t.Fatalf("status %+v", st)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
-174
@@ -1,174 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bufio"
|
|
||||||
"bytes"
|
|
||||||
"encoding/binary"
|
|
||||||
"encoding/hex"
|
|
||||||
"errors"
|
|
||||||
"io"
|
|
||||||
"os"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Minimal reader of systemd journal files (*.journal), enough to follow them
|
|
||||||
// without journalctl: objects are walked in file order, which is the order
|
|
||||||
// entries were written. Format: https://systemd.io/JOURNAL_FILE_FORMAT/
|
|
||||||
|
|
||||||
const (
|
|
||||||
jHeaderMin = 208 // header fields used below end at offset 208
|
|
||||||
jObjHeaderSize = 16
|
|
||||||
|
|
||||||
jObjData = 1
|
|
||||||
jObjEntry = 3
|
|
||||||
|
|
||||||
jIncompatCompact = 1 << 4
|
|
||||||
jStateArchived = 2
|
|
||||||
jObjCompressed = 1<<0 | 1<<1 | 1<<2 // XZ, LZ4, ZSTD: not decoded (no stdlib codec)
|
|
||||||
)
|
|
||||||
|
|
||||||
var jSignature = []byte("LPKSHHRH")
|
|
||||||
|
|
||||||
// jHeader holds the header fields the reader needs.
|
|
||||||
type jHeader struct {
|
|
||||||
compact bool
|
|
||||||
archived bool
|
|
||||||
fileID string
|
|
||||||
headerSize uint64
|
|
||||||
tailObject uint64 // offset of the last object
|
|
||||||
tailSeqnum uint64 // seqnum of the last complete entry
|
|
||||||
tailRealtimeUS uint64 // realtime of the last entry (µs since the epoch)
|
|
||||||
}
|
|
||||||
|
|
||||||
func readJournalHeader(f io.ReaderAt) (jHeader, error) {
|
|
||||||
b := make([]byte, jHeaderMin)
|
|
||||||
if _, err := f.ReadAt(b, 0); err != nil {
|
|
||||||
return jHeader{}, err
|
|
||||||
}
|
|
||||||
if !bytes.Equal(b[:8], jSignature) {
|
|
||||||
return jHeader{}, errors.New("not a journal file")
|
|
||||||
}
|
|
||||||
le := binary.LittleEndian
|
|
||||||
h := jHeader{
|
|
||||||
compact: le.Uint32(b[12:])&jIncompatCompact != 0,
|
|
||||||
archived: b[16] == jStateArchived,
|
|
||||||
fileID: hex.EncodeToString(b[24:40]),
|
|
||||||
headerSize: le.Uint64(b[88:]),
|
|
||||||
tailObject: le.Uint64(b[136:]),
|
|
||||||
tailSeqnum: le.Uint64(b[160:]),
|
|
||||||
tailRealtimeUS: le.Uint64(b[192:]),
|
|
||||||
}
|
|
||||||
if h.headerSize < jHeaderMin {
|
|
||||||
return h, errors.New("journal header too small")
|
|
||||||
}
|
|
||||||
return h, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// journalEntry is one entry: its time and its "FIELD=value" pairs.
|
|
||||||
type journalEntry struct {
|
|
||||||
Realtime time.Time
|
|
||||||
Fields map[string]string
|
|
||||||
Compressed int // fields that could not be read (compressed data)
|
|
||||||
}
|
|
||||||
|
|
||||||
// readJournal reads the entries complete after offset `from` (0 = start of
|
|
||||||
// the file) and returns the offset to resume from. Entries older than
|
|
||||||
// `notBefore` are skipped without decoding their fields.
|
|
||||||
func readJournal(path string, from int64, notBefore time.Time, emit func(*journalEntry)) (next int64, h jHeader, err error) {
|
|
||||||
f, err := os.Open(path)
|
|
||||||
if err != nil {
|
|
||||||
return from, h, err
|
|
||||||
}
|
|
||||||
defer f.Close()
|
|
||||||
if h, err = readJournalHeader(f); err != nil {
|
|
||||||
return from, h, err
|
|
||||||
}
|
|
||||||
if from < int64(h.headerSize) {
|
|
||||||
from = int64(h.headerSize)
|
|
||||||
}
|
|
||||||
end := int64(h.tailObject)
|
|
||||||
r := bufio.NewReaderSize(io.NewSectionReader(f, from, 1<<62), 64*1024)
|
|
||||||
le := binary.LittleEndian
|
|
||||||
hdr := make([]byte, jObjHeaderSize)
|
|
||||||
off := from
|
|
||||||
for off <= end {
|
|
||||||
if _, err := io.ReadFull(r, hdr); err != nil {
|
|
||||||
return off, h, nil // object still being written
|
|
||||||
}
|
|
||||||
typ, size := hdr[0], int64(le.Uint64(hdr[8:]))
|
|
||||||
if size < jObjHeaderSize {
|
|
||||||
return off, h, nil
|
|
||||||
}
|
|
||||||
body := size - jObjHeaderSize
|
|
||||||
if typ == jObjEntry && body >= 48 && body < 1<<20 {
|
|
||||||
obj := make([]byte, body)
|
|
||||||
if _, err := io.ReadFull(r, obj); err != nil {
|
|
||||||
return off, h, nil
|
|
||||||
}
|
|
||||||
seq := le.Uint64(obj[0:])
|
|
||||||
if seq == 0 || seq > h.tailSeqnum {
|
|
||||||
return off, h, nil // not linked yet: retry at the next poll
|
|
||||||
}
|
|
||||||
rt := time.UnixMicro(int64(le.Uint64(obj[8:])))
|
|
||||||
if !rt.Before(notBefore) {
|
|
||||||
emit(readEntryFields(f, h.compact, rt, obj[48:]))
|
|
||||||
}
|
|
||||||
} else if _, err := r.Discard(int(body)); err != nil {
|
|
||||||
return off, h, nil
|
|
||||||
}
|
|
||||||
next := (off + size + 7) &^ 7 // objects are 8-byte aligned
|
|
||||||
if pad := next - off - size; pad > 0 {
|
|
||||||
if _, err := r.Discard(int(pad)); err != nil {
|
|
||||||
return next, h, nil // the object is complete: resume after it
|
|
||||||
}
|
|
||||||
}
|
|
||||||
off = next
|
|
||||||
}
|
|
||||||
return off, h, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// readEntryFields resolves the data objects referenced by an entry.
|
|
||||||
func readEntryFields(f io.ReaderAt, compact bool, rt time.Time, items []byte) *journalEntry {
|
|
||||||
le := binary.LittleEndian
|
|
||||||
e := &journalEntry{Realtime: rt, Fields: make(map[string]string, 16)}
|
|
||||||
step := 16
|
|
||||||
if compact {
|
|
||||||
step = 4
|
|
||||||
}
|
|
||||||
payloadAt := int64(64)
|
|
||||||
if compact {
|
|
||||||
payloadAt = 72
|
|
||||||
}
|
|
||||||
hdr := make([]byte, jObjHeaderSize)
|
|
||||||
for i := 0; i+step <= len(items); i += step {
|
|
||||||
var off int64
|
|
||||||
if compact {
|
|
||||||
off = int64(le.Uint32(items[i:]))
|
|
||||||
} else {
|
|
||||||
off = int64(le.Uint64(items[i:]))
|
|
||||||
}
|
|
||||||
if off == 0 {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if _, err := f.ReadAt(hdr, off); err != nil || hdr[0] != jObjData {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if hdr[1]&jObjCompressed != 0 {
|
|
||||||
e.Compressed++
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
size := int64(le.Uint64(hdr[8:]))
|
|
||||||
n := size - payloadAt
|
|
||||||
if n <= 0 || n > 1<<20 {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
p := make([]byte, n)
|
|
||||||
if _, err := f.ReadAt(p, off+payloadAt); err != nil {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if k := bytes.IndexByte(p, '='); k > 0 {
|
|
||||||
e.Fields[string(p[:k])] = string(p[k+1:])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return e
|
|
||||||
}
|
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
// Logstream: a syslog (UDP/TCP), Docker and host system logs sink stored in VictoriaLogs,
|
// Logstream: a syslog (UDP/TCP) sink stored in VictoriaLogs,
|
||||||
// with a web interface to browse and search the logs.
|
// with a web interface to browse and search the logs.
|
||||||
package main
|
package main
|
||||||
|
|
||||||
@@ -38,8 +38,6 @@ type config struct {
|
|||||||
dockerLogs bool
|
dockerLogs bool
|
||||||
dockerHost string
|
dockerHost string
|
||||||
backfill time.Duration
|
backfill time.Duration
|
||||||
hostRoot string
|
|
||||||
hostFill time.Duration
|
|
||||||
batchSize int
|
batchSize int
|
||||||
queueSize int
|
queueSize int
|
||||||
flushEvery time.Duration
|
flushEvery time.Duration
|
||||||
@@ -91,8 +89,6 @@ func main() {
|
|||||||
dockerLogs: getenvBool("DOCKER_LOGS", false),
|
dockerLogs: getenvBool("DOCKER_LOGS", false),
|
||||||
dockerHost: getenv("DOCKER_HOST", "unix:///var/run/docker.sock"),
|
dockerHost: getenv("DOCKER_HOST", "unix:///var/run/docker.sock"),
|
||||||
backfill: getenvDuration("DOCKER_BACKFILL", time.Hour),
|
backfill: getenvDuration("DOCKER_BACKFILL", time.Hour),
|
||||||
hostRoot: getenv("HOST_LOGS_ROOT", "/host"),
|
|
||||||
hostFill: getenvDuration("HOST_LOGS_BACKFILL", time.Hour),
|
|
||||||
batchSize: getenvInt("BATCH_SIZE", 1000),
|
batchSize: getenvInt("BATCH_SIZE", 1000),
|
||||||
queueSize: getenvInt("QUEUE_SIZE", 100000),
|
queueSize: getenvInt("QUEUE_SIZE", 100000),
|
||||||
flushEvery: time.Duration(getenvInt("FLUSH_MS", 1000)) * time.Millisecond,
|
flushEvery: time.Duration(getenvInt("FLUSH_MS", 1000)) * time.Millisecond,
|
||||||
@@ -143,9 +139,6 @@ func main() {
|
|||||||
log.Printf("docker logs enabled through %s", cfg.dockerHost)
|
log.Printf("docker logs enabled through %s", cfg.dockerHost)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// System logs of the host (journal or /var/log), off until enabled in Settings > Sources.
|
|
||||||
api.host = NewHostLogs(cfg.hostRoot, cfg.dataDir, cfg.hostFill, sink)
|
|
||||||
go api.host.Run(ctx)
|
|
||||||
api.Routes(mux)
|
api.Routes(mux)
|
||||||
mux.Handle("GET /", http.FileServer(http.FS(static)))
|
mux.Handle("GET /", http.FileServer(http.FS(static)))
|
||||||
|
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ type Filter struct {
|
|||||||
Host string
|
Host string
|
||||||
App string
|
App string
|
||||||
Severity int // highest severity number included (0 = emerg … 7 = debug), -1 = all
|
Severity int // highest severity number included (0 = emerg … 7 = debug), -1 = all
|
||||||
Source string // "syslog", "docker", "host" or "" for all
|
Source string // "syslog", "docker" or "" for all
|
||||||
}
|
}
|
||||||
|
|
||||||
var rangeDurations = map[string]time.Duration{
|
var rangeDurations = map[string]time.Duration{
|
||||||
@@ -119,10 +119,10 @@ func (f Filter) filterExpr() string {
|
|||||||
parts = append(parts, "app:="+strconv.Quote(f.App))
|
parts = append(parts, "app:="+strconv.Quote(f.App))
|
||||||
}
|
}
|
||||||
switch f.Source {
|
switch f.Source {
|
||||||
case "docker", "host":
|
case "docker":
|
||||||
parts = append(parts, `source_type:=`+strconv.Quote(f.Source))
|
parts = append(parts, `source_type:="docker"`)
|
||||||
case "syslog": // also matches logs stored before source_type existed
|
case "syslog": // also matches logs stored before source_type existed
|
||||||
parts = append(parts, `!(source_type:in("docker","host"))`)
|
parts = append(parts, `!(source_type:="docker")`)
|
||||||
}
|
}
|
||||||
if f.Severity >= 0 && f.Severity < 7 {
|
if f.Severity >= 0 && f.Severity < 7 {
|
||||||
names := make([]string, 0, 8)
|
names := make([]string, 0, 8)
|
||||||
@@ -209,7 +209,7 @@ func (m *Matcher) Match(e *Entry) bool {
|
|||||||
if m.f.App != "" && e.App != m.f.App {
|
if m.f.App != "" && e.App != m.f.App {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
if m.f.Source != "" && m.f.Source != e.SourceType {
|
if (m.f.Source == "docker") != (e.SourceType == "docker") && m.f.Source != "" {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
if m.f.Severity >= 0 && e.SevNum > m.f.Severity {
|
if m.f.Severity >= 0 && e.SevNum > m.f.Severity {
|
||||||
|
|||||||
+3
-78
@@ -67,19 +67,6 @@ const I18N = {
|
|||||||
fHostName: 'host name (DNS)', fHostIp: 'host IP',
|
fHostName: 'host name (DNS)', fHostIp: 'host IP',
|
||||||
clickHost: 'Click to filter on this host', clickApp: 'Click to filter on this app',
|
clickHost: 'Click to filter on this host', clickApp: 'Click to filter on this app',
|
||||||
sourceAria: 'Source', srcAll: 'All sources', tabSources: 'Sources',
|
sourceAria: 'Source', srcAll: 'All sources', tabSources: 'Sources',
|
||||||
srcHost: 'Host system',
|
|
||||||
hostTitle: 'Host system logs',
|
|
||||||
hostEnabled: 'Collect the system logs of this machine',
|
|
||||||
hostOff: 'Off: the system logs of the machine hosting Logstream are not collected.',
|
|
||||||
hostWaiting: 'Starting…',
|
|
||||||
hostJournal: ({ n, r }) => `Reading the systemd journal (${n} files): ${r} entries since startup.`,
|
|
||||||
hostFiles: ({ n, r }) => `Following ${n} files of /var/log: ${r} lines since startup.`,
|
|
||||||
hostCompressed: (n) => ` ${n} compressed fields skipped (long messages, readable with journalctl).`,
|
|
||||||
host_not_mounted: 'No host log directory found: mount /var/log and /run/log/journal under /host (see docker-compose.yml).',
|
|
||||||
host_permission: 'Permission denied: give the container a group allowed to read the logs (HOST_LOGS_GID in .env, see the README). ',
|
|
||||||
host_empty: 'No systemd journal and no log file found in /var/log.',
|
|
||||||
host_read: 'Cannot read the host logs: ',
|
|
||||||
hostHelp: 'Reads the systemd journal of the host (/var/log/journal, /run/log/journal), or the text files of /var/log (syslog, messages, *.log) when there is no journal. Directories are mounted read-only. When turned on, the last hour is read first (HOST_LOGS_BACKFILL).',
|
|
||||||
syslogEnabled: 'Receive syslog messages', syslogProtocols: 'Protocols', syslogPort: 'Port',
|
syslogEnabled: 'Receive syslog messages', syslogProtocols: 'Protocols', syslogPort: 'Port',
|
||||||
syslogListening: ({ port, protos }) => `Listening on port ${port} (${protos}).`,
|
syslogListening: ({ port, protos }) => `Listening on port ${port} (${protos}).`,
|
||||||
syslogOff: 'Syslog reception is off: syslog messages are ignored (Docker logs are still collected).',
|
syslogOff: 'Syslog reception is off: syslog messages are ignored (Docker logs are still collected).',
|
||||||
@@ -101,7 +88,6 @@ const I18N = {
|
|||||||
dockerHelp: 'Logs are read through docker-socket-proxy, a read-only gateway: Logstream can list containers and read their logs, nothing else. Choices apply per compose service (or container name), so they survive container re-creations.',
|
dockerHelp: 'Logs are read through docker-socket-proxy, a read-only gateway: Logstream can list containers and read their logs, nothing else. Choices apply per compose service (or container name), so they survive container re-creations.',
|
||||||
fContainer: 'container', fContainerId: 'container ID', fImage: 'image', fProject: 'compose project',
|
fContainer: 'container', fContainerId: 'container ID', fImage: 'image', fProject: 'compose project',
|
||||||
fService: 'compose service', fStream: 'stream', fSourceType: 'source',
|
fService: 'compose service', fStream: 'stream', fSourceType: 'source',
|
||||||
fUnit: 'systemd unit', fLogFile: 'log file',
|
|
||||||
export: 'Export', exportCsvHint: 'Comma separated, UTF-8',
|
export: 'Export', exportCsvHint: 'Comma separated, UTF-8',
|
||||||
exportExcel: 'CSV for Excel', exportExcelHint: 'Semicolon separated, for Excel in French',
|
exportExcel: 'CSV for Excel', exportExcelHint: 'Semicolon separated, for Excel in French',
|
||||||
exportNote: (n) => `Up to ${n} logs matching the current filters, newest first. Dates use the time zone chosen in Settings.`,
|
exportNote: (n) => `Up to ${n} logs matching the current filters, newest first. Dates use the time zone chosen in Settings.`,
|
||||||
@@ -210,19 +196,6 @@ const I18N = {
|
|||||||
fHostName: 'nom d\'hôte (DNS)', fHostIp: 'IP de l\'hôte',
|
fHostName: 'nom d\'hôte (DNS)', fHostIp: 'IP de l\'hôte',
|
||||||
clickHost: 'Cliquer pour filtrer sur cet hôte', clickApp: 'Cliquer pour filtrer sur cette appli',
|
clickHost: 'Cliquer pour filtrer sur cet hôte', clickApp: 'Cliquer pour filtrer sur cette appli',
|
||||||
sourceAria: 'Source', srcAll: 'Toutes les sources', tabSources: 'Sources',
|
sourceAria: 'Source', srcAll: 'Toutes les sources', tabSources: 'Sources',
|
||||||
srcHost: 'Système hôte',
|
|
||||||
hostTitle: 'Logs système de l\'hôte',
|
|
||||||
hostEnabled: 'Collecter les logs système de cette machine',
|
|
||||||
hostOff: 'Désactivé : les logs système de la machine qui héberge Logstream ne sont pas collectés.',
|
|
||||||
hostWaiting: 'Démarrage…',
|
|
||||||
hostJournal: ({ n, r }) => `Lecture du journal systemd (${n} fichiers) : ${r} entrées depuis le démarrage.`,
|
|
||||||
hostFiles: ({ n, r }) => `Suivi de ${n} fichiers de /var/log : ${r} lignes depuis le démarrage.`,
|
|
||||||
hostCompressed: (n) => ` ${n} champs compressés ignorés (messages longs, lisibles avec journalctl).`,
|
|
||||||
host_not_mounted: 'Aucun répertoire de logs de l\'hôte trouvé : montez /var/log et /run/log/journal sous /host (voir docker-compose.yml).',
|
|
||||||
host_permission: 'Accès refusé : donnez au conteneur un groupe autorisé à lire les logs (HOST_LOGS_GID dans .env, voir le README). ',
|
|
||||||
host_empty: 'Ni journal systemd ni fichier de log trouvé dans /var/log.',
|
|
||||||
host_read: 'Lecture des logs de l\'hôte impossible : ',
|
|
||||||
hostHelp: 'Lit le journal systemd de l\'hôte (/var/log/journal, /run/log/journal), ou les fichiers texte de /var/log (syslog, messages, *.log) s\'il n\'y a pas de journal. Les répertoires sont montés en lecture seule. À l\'activation, la dernière heure est lue d\'abord (HOST_LOGS_BACKFILL).',
|
|
||||||
syslogEnabled: 'Recevoir les messages syslog', syslogProtocols: 'Protocoles', syslogPort: 'Port',
|
syslogEnabled: 'Recevoir les messages syslog', syslogProtocols: 'Protocoles', syslogPort: 'Port',
|
||||||
syslogListening: ({ port, protos }) => `Écoute sur le port ${port} (${protos}).`,
|
syslogListening: ({ port, protos }) => `Écoute sur le port ${port} (${protos}).`,
|
||||||
syslogOff: 'Réception syslog désactivée : les messages syslog sont ignorés (les logs Docker sont toujours collectés).',
|
syslogOff: 'Réception syslog désactivée : les messages syslog sont ignorés (les logs Docker sont toujours collectés).',
|
||||||
@@ -244,7 +217,6 @@ const I18N = {
|
|||||||
dockerHelp: 'Les logs sont lus via docker-socket-proxy, une passerelle en lecture seule : Logstream peut lister les conteneurs et lire leurs logs, rien d\'autre. Les choix s\'appliquent par service compose (ou nom de conteneur), ils survivent donc à la recréation des conteneurs.',
|
dockerHelp: 'Les logs sont lus via docker-socket-proxy, une passerelle en lecture seule : Logstream peut lister les conteneurs et lire leurs logs, rien d\'autre. Les choix s\'appliquent par service compose (ou nom de conteneur), ils survivent donc à la recréation des conteneurs.',
|
||||||
fContainer: 'conteneur', fContainerId: 'ID du conteneur', fImage: 'image', fProject: 'projet compose',
|
fContainer: 'conteneur', fContainerId: 'ID du conteneur', fImage: 'image', fProject: 'projet compose',
|
||||||
fService: 'service compose', fStream: 'flux', fSourceType: 'source',
|
fService: 'service compose', fStream: 'flux', fSourceType: 'source',
|
||||||
fUnit: 'unité systemd', fLogFile: 'fichier de log',
|
|
||||||
export: 'Exporter', exportCsvHint: 'Séparateur virgule, UTF-8',
|
export: 'Exporter', exportCsvHint: 'Séparateur virgule, UTF-8',
|
||||||
exportExcel: 'CSV pour Excel', exportExcelHint: 'Séparateur point-virgule, pour Excel en français',
|
exportExcel: 'CSV pour Excel', exportExcelHint: 'Séparateur point-virgule, pour Excel en français',
|
||||||
exportNote: (n) => `Jusqu'à ${n} logs correspondant aux filtres, du plus récent au plus ancien. Dates dans le fuseau choisi dans Paramètres.`,
|
exportNote: (n) => `Jusqu'à ${n} logs correspondant aux filtres, du plus récent au plus ancien. Dates dans le fuseau choisi dans Paramètres.`,
|
||||||
@@ -475,7 +447,6 @@ function setLang(next) {
|
|||||||
renderPurge();
|
renderPurge();
|
||||||
renderInterface();
|
renderInterface();
|
||||||
renderSyslog();
|
renderSyslog();
|
||||||
renderHost();
|
|
||||||
renderDocker();
|
renderDocker();
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -834,14 +805,14 @@ function updateCount(tookMs) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
function detailsHTML(r) {
|
function detailsHTML(r) {
|
||||||
const order = ['received', 'msg_time', '_time', 'host', 'host_name', 'host_ip', 'source_type', 'container', 'image', 'compose_project', 'compose_service', 'stream', 'unit', 'log_file', 'app', 'procid', 'severity', 'facility', 'source', 'proto', 'sd', '_msg'];
|
const order = ['received', 'msg_time', '_time', 'host', 'host_name', 'host_ip', 'source_type', 'container', 'image', 'compose_project', 'compose_service', 'stream', 'app', 'procid', 'severity', 'facility', 'source', 'proto', 'sd', '_msg'];
|
||||||
const hidden = new Set(['_stream_id', '_stream', 'sevnum']);
|
const hidden = new Set(['_stream_id', '_stream', 'sevnum']);
|
||||||
if (r.msg_time) hidden.add('_time'); // same as the reception time
|
if (r.msg_time) hidden.add('_time'); // same as the reception time
|
||||||
const keys = order.filter((k) => r[k] != null && r[k] !== '' && !hidden.has(k))
|
const keys = order.filter((k) => r[k] != null && r[k] !== '' && !hidden.has(k))
|
||||||
.concat(Object.keys(r).filter((k) => !order.includes(k) && !hidden.has(k)).sort());
|
.concat(Object.keys(r).filter((k) => !order.includes(k) && !hidden.has(k)).sort());
|
||||||
const label = {
|
const label = {
|
||||||
container: t('fContainer'), container_id: t('fContainerId'), image: t('fImage'), compose_project: t('fProject'),
|
container: t('fContainer'), container_id: t('fContainerId'), image: t('fImage'), compose_project: t('fProject'),
|
||||||
compose_service: t('fService'), stream: t('fStream'), source_type: t('fSourceType'), unit: t('fUnit'), log_file: t('fLogFile'),
|
compose_service: t('fService'), stream: t('fStream'), source_type: t('fSourceType'),
|
||||||
msg_time: t('fTime'), host_name: t('fHostName'), host_ip: t('fHostIp'), received: t('fReceived'), _time: t('fTime'), _msg: t('fMsg'), procid: t('fPid'), sd: t('fSd') };
|
msg_time: t('fTime'), host_name: t('fHostName'), host_ip: t('fHostIp'), received: t('fReceived'), _time: t('fTime'), _msg: t('fMsg'), procid: t('fPid'), sd: t('fSd') };
|
||||||
const val = (k) => (k === '_time' || k === 'received' || k === 'msg_time'
|
const val = (k) => (k === '_time' || k === 'received' || k === 'msg_time'
|
||||||
? `${esc(fmtFull(r[k]))} <span class="muted">(${esc(r[k])})</span>`
|
? `${esc(fmtFull(r[k]))} <span class="muted">(${esc(r[k])})</span>`
|
||||||
@@ -1755,48 +1726,6 @@ $('#syslogProtos').addEventListener('click', (ev) => {
|
|||||||
configureSyslog({ [b.dataset.proto]: !syslogState[b.dataset.proto] });
|
configureSyslog({ [b.dataset.proto]: !syslogState[b.dataset.proto] });
|
||||||
});
|
});
|
||||||
|
|
||||||
/* ================= Host system logs ================= */
|
|
||||||
|
|
||||||
let hostState = null;
|
|
||||||
|
|
||||||
async function loadHost() {
|
|
||||||
try { hostState = await api('/api/hostlogs'); } catch { hostState = null; }
|
|
||||||
renderHost();
|
|
||||||
}
|
|
||||||
|
|
||||||
function renderHost() {
|
|
||||||
const s = hostState;
|
|
||||||
const status = $('#hostStatus');
|
|
||||||
if (!s) { status.textContent = ''; return; }
|
|
||||||
$('#hostEnabled').checked = s.enabled;
|
|
||||||
let text = t('hostWaiting');
|
|
||||||
let cls = 'muted';
|
|
||||||
if (!s.enabled) {
|
|
||||||
text = t('hostOff');
|
|
||||||
} else if (s.code) {
|
|
||||||
text = t('host_' + s.code) + (s.code === 'read' || s.code === 'permission' ? s.error || '' : '');
|
|
||||||
cls = 'bad';
|
|
||||||
} else if (s.mode === 'journal') {
|
|
||||||
text = t('hostJournal', { n: s.files, r: fmtNum(s.read) }) + (s.compressed ? t('hostCompressed', fmtNum(s.compressed)) : '');
|
|
||||||
cls = 'good';
|
|
||||||
} else if (s.mode === 'files') {
|
|
||||||
text = t('hostFiles', { n: s.files, r: fmtNum(s.read) });
|
|
||||||
cls = 'good';
|
|
||||||
}
|
|
||||||
status.textContent = text;
|
|
||||||
status.className = 'docker-status ' + cls;
|
|
||||||
}
|
|
||||||
|
|
||||||
async function configureHost(enabled) {
|
|
||||||
hostState = { ...(hostState || {}), enabled };
|
|
||||||
renderHost();
|
|
||||||
try { hostState = await api('/api/hostlogs', { method: 'PUT', body: { enabled } }); } catch (e) { toast(e.message); }
|
|
||||||
renderHost();
|
|
||||||
setTimeout(loadHost, 1500); // the first scan runs in the background
|
|
||||||
}
|
|
||||||
|
|
||||||
$('#hostEnabled').addEventListener('change', (ev) => configureHost(ev.target.checked));
|
|
||||||
|
|
||||||
/* ================= Docker source ================= */
|
/* ================= Docker source ================= */
|
||||||
|
|
||||||
let dockerState = null;
|
let dockerState = null;
|
||||||
@@ -1910,10 +1839,7 @@ for (const [id, on] of [['#dockerAll', true], ['#dockerNone', false]]) {
|
|||||||
}
|
}
|
||||||
// Keep the list fresh while the Sources tab is open.
|
// Keep the list fresh while the Sources tab is open.
|
||||||
setInterval(() => {
|
setInterval(() => {
|
||||||
if ($('#settingsDlg').open && !document.querySelector('[data-panel="sources"]').hidden) {
|
if ($('#settingsDlg').open && !document.querySelector('[data-panel="sources"]').hidden) loadDocker();
|
||||||
loadDocker();
|
|
||||||
loadHost();
|
|
||||||
}
|
|
||||||
}, 4000);
|
}, 4000);
|
||||||
|
|
||||||
/* ================= Purge ================= */
|
/* ================= Purge ================= */
|
||||||
@@ -2076,7 +2002,6 @@ $('#settingsBtn').addEventListener('click', () => {
|
|||||||
renderInterface();
|
renderInterface();
|
||||||
loadPurgeStatus();
|
loadPurgeStatus();
|
||||||
loadSyslog();
|
loadSyslog();
|
||||||
loadHost();
|
|
||||||
loadDocker();
|
loadDocker();
|
||||||
showSettingsTab(store.get('settingsTab', 'locale'));
|
showSettingsTab(store.get('settingsTab', 'locale'));
|
||||||
$('#settingsDlg').showModal();
|
$('#settingsDlg').showModal();
|
||||||
|
|||||||
@@ -70,7 +70,6 @@
|
|||||||
<option value="" data-i18n="srcAll">All sources</option>
|
<option value="" data-i18n="srcAll">All sources</option>
|
||||||
<option value="syslog">Syslog</option>
|
<option value="syslog">Syslog</option>
|
||||||
<option value="docker">Docker</option>
|
<option value="docker">Docker</option>
|
||||||
<option value="host" data-i18n="srcHost">Host system</option>
|
|
||||||
</select>
|
</select>
|
||||||
<span class="spacer"></span>
|
<span class="spacer"></span>
|
||||||
<span id="count" class="muted"></span>
|
<span id="count" class="muted"></span>
|
||||||
@@ -210,16 +209,6 @@
|
|||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section class="set-section">
|
|
||||||
<h3 data-i18n="hostTitle">Host system logs</h3>
|
|
||||||
<p id="hostStatus" class="docker-status"></p>
|
|
||||||
<label class="switch-row">
|
|
||||||
<input id="hostEnabled" type="checkbox" class="switch">
|
|
||||||
<span data-i18n="hostEnabled">Collect the system logs of this machine</span>
|
|
||||||
</label>
|
|
||||||
<p class="muted small" data-i18n="hostHelp"></p>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<section class="set-section">
|
<section class="set-section">
|
||||||
<h3 data-i18n="dockerTitle">Docker containers</h3>
|
<h3 data-i18n="dockerTitle">Docker containers</h3>
|
||||||
<p id="dockerStatus" class="docker-status"></p>
|
<p id="dockerStatus" class="docker-status"></p>
|
||||||
|
|||||||
Reference in new issue
Block a user