WIP: Source « Fichiers » : logs de Traefik, nginx et autres programmes #23

Closed
claude Bot wants to merge 0 commits from feat/custom-log-source into main
pull from: feat/custom-log-source

Requested by Cédric

Before: LogStream could only read syslog, Docker, host system logs, dmesg and auditd; the log files of Traefik, nginx or an application stayed out of reach.

After: Settings > Log sources > Log files lists files to follow, each with a name (= app, the file name by default), the path of one file (no wildcards for now), a format (Automatic by default), an optional host and a switch. As soon as a path is typed, the file is checked and the row shows whether it was found, the detected format and its last line. Lines show in the list with a green edge and a FILE badge, and the Source filter has a Files entry.

Formats: Automatic (detected from the last 50 lines: Traefik access JSON, JSON lines, CLF or logfmt when 80 % of them follow it, else plain text), or forced: plain text (level and leading timestamp detected, indented lines joined to the previous entry), JSON lines, logfmt, access log CLF (nginx, Apache, Traefik), Traefik access log JSON, regular expression with named groups (time, level, msg, host, app, others as fields). Access logs: 5xx = err, 4xx = warning.

How: filesource.go follows each file like the audit source (inode + offset in /data/files-state.json, rotation via .1, copytruncate, FILES_BACKFILL of dated lines when turned on); fileformats.go holds the parsers and the detection. GET/PUT /api/files for the settings, POST /api/files/detect for the check of a path (POST so read-only accounts cannot read files through it). docker-compose, .env.example and both READMEs document the mount and formats.

Checked: go vet, go build, go test -race (tests for every format, detection, rotation, restart, truncation, backfill, forced format); UI checked in a static preview with stubbed API (rows, path check, auto name, FR/EN).

🤖 Generated with Claude Code

<!-- ccr-projects-attribution: {"github_login":"vblogio"} --> _Requested by **Cédric**_ Before: LogStream could only read syslog, Docker, host system logs, dmesg and auditd; the log files of Traefik, nginx or an application stayed out of reach. After: Settings > Log sources > **Log files** lists files to follow, each with a name (= app, the file name by default), the path of one file (no wildcards for now), a format (**Automatic** by default), an optional host and a switch. As soon as a path is typed, the file is checked and the row shows whether it was found, the detected format and its last line. Lines show in the list with a green edge and a **FILE** badge, and the Source filter has a **Files** entry. Formats: Automatic (detected from the last 50 lines: Traefik access JSON, JSON lines, CLF or logfmt when 80 % of them follow it, else plain text), or forced: plain text (level and leading timestamp detected, indented lines joined to the previous entry), JSON lines, logfmt, access log CLF (nginx, Apache, Traefik), Traefik access log JSON, regular expression with named groups (`time`, `level`, `msg`, `host`, `app`, others as fields). Access logs: 5xx = err, 4xx = warning. How: `filesource.go` follows each file like the audit source (inode + offset in `/data/files-state.json`, rotation via `.1`, copytruncate, `FILES_BACKFILL` of dated lines when turned on); `fileformats.go` holds the parsers and the detection. `GET/PUT /api/files` for the settings, `POST /api/files/detect` for the check of a path (POST so read-only accounts cannot read files through it). docker-compose, `.env.example` and both READMEs document the mount and formats. Checked: `go vet`, `go build`, `go test -race` (tests for every format, detection, rotation, restart, truncation, backfill, forced format); UI checked in a static preview with stubbed API (rows, path check, auto name, FR/EN). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude Bot added 1 commit 2026-10-05 10:28:04 +02:00
Files source (source_type=file), declared in Settings > Log sources > Log
files (/data/files.json, 50 at most): name (= app), path as mounted in the
container (file or glob, rotated copies .1/.gz left out, 200 files per
glob), format, optional host, switch.

- formats: plain text (level detected, leading timestamp read and removed),
  JSON lines (msg/time/level keys, numeric pino levels, other keys as
  fields, nested ones dotted), logfmt, access log CLF (nginx, Apache,
  Traefik CLF with router/backend/duration), Traefik access log in JSON
  (summary "GET host/path 200 · 12.5 ms · router · client"), regular
  expression with named groups time/level/msg/host/app/status; 5xx = err,
  4xx = warning; fields named like LogStream's own renamed field_<name>
- indented lines (text) and non-matching lines (regex) added to the
  previous entry, which waits 2 s for them
- inode + offset per file in /data/files-state.json, rotation (end of
  .1 read first), copytruncate, file renamed within the glob; last
  FILES_BACKFILL (1h) of dated lines read when turned on, new files read
  from their start
- status per file (files followed, entries read, not found / permission)

UI: rows with switch, name, path, format, host (regex field when needed),
status line; green edge, tint and FILE badge in the list, "Files" in the
source filter, HTTP fields first in the details, FR/EN. docker-compose,
.env.example and README (EN/FR): mounts, formats, FILES_BACKFILL.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
claude Bot added 1 commit 2026-10-05 10:33:27 +02:00
- a path is one file (wildcards refused for now); the name defaults to
  the file name
- format "auto" (the default): detected from the last 50 lines of the
  file, the structured format followed by at least 80 % of them (Traefik
  access JSON, JSON lines, CLF, logfmt), else plain text; done again when
  the format changes, shown in the status ("Format detected: …"); any
  other format forces it
- POST /api/files/detect {path}: found / permission / empty, detected
  format and last line, shown under the row as soon as the path is typed
  (POST so that read-only accounts cannot read files through it)
- README (EN/FR) updated

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
claude Bot closed this pull request 2026-10-05 10:42:23 +02:00

Pull request closed

Please reopen this pull request to perform a merge.
This pull request is marked as a work in progress.
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vLab-BZH/logstream#23