Log files source: follow the logs of Traefik, nginx and other programs
Files source (source_type=file), declared in Settings > Log sources > Log files (/data/files.json, 50 at most): name (= app), path as mounted in the container (file or glob, rotated copies .1/.gz left out, 200 files per glob), format, optional host, switch. - formats: plain text (level detected, leading timestamp read and removed), JSON lines (msg/time/level keys, numeric pino levels, other keys as fields, nested ones dotted), logfmt, access log CLF (nginx, Apache, Traefik CLF with router/backend/duration), Traefik access log in JSON (summary "GET host/path 200 · 12.5 ms · router · client"), regular expression with named groups time/level/msg/host/app/status; 5xx = err, 4xx = warning; fields named like LogStream's own renamed field_<name> - indented lines (text) and non-matching lines (regex) added to the previous entry, which waits 2 s for them - inode + offset per file in /data/files-state.json, rotation (end of .1 read first), copytruncate, file renamed within the glob; last FILES_BACKFILL (1h) of dated lines read when turned on, new files read from their start - status per file (files followed, entries read, not found / permission) UI: rows with switch, name, path, format, host (regex field when needed), status line; green edge, tint and FILE badge in the list, "Files" in the source filter, HTTP fields first in the details, FR/EN. docker-compose, .env.example and README (EN/FR): mounts, formats, FILES_BACKFILL. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
e87186e476
commit
087d9e7b30
14 files changed
+1934
-9
No files matched your search
@@ -43,6 +43,7 @@ type config struct {
|
||||
auditLog string
|
||||
auditHost string
|
||||
auditFill time.Duration
|
||||
filesFill time.Duration
|
||||
batchSize int
|
||||
queueSize int
|
||||
flushEvery time.Duration
|
||||
@@ -125,6 +126,7 @@ func main() {
|
||||
auditLog: getenv("AUDIT_LOG", "/var/log/audit/audit.log"),
|
||||
auditHost: os.Getenv("AUDIT_HOST"),
|
||||
auditFill: getenvDuration("AUDIT_BACKFILL", time.Hour),
|
||||
filesFill: getenvDuration("FILES_BACKFILL", time.Hour),
|
||||
batchSize: getenvInt("BATCH_SIZE", 1000),
|
||||
queueSize: getenvInt("QUEUE_SIZE", 100000),
|
||||
flushEvery: time.Duration(getenvInt("FLUSH_MS", 1000)) * time.Millisecond,
|
||||
@@ -218,6 +220,10 @@ func main() {
|
||||
// Linux audit log of the host (auditd), off until enabled in Settings > Sources.
|
||||
api.audit = NewAudit(cfg.auditLog, cfg.hostRoot, cfg.auditHost, cfg.dataDir, cfg.auditFill, sink)
|
||||
go api.audit.Run(ctx)
|
||||
// Log files of other programs (Traefik, nginx…) mounted in the container,
|
||||
// declared in Settings > Sources.
|
||||
api.files = NewFiles(cfg.hostRoot, cfg.dataDir, cfg.filesFill, sink)
|
||||
go api.files.Run(ctx)
|
||||
api.selfLog = selfLog
|
||||
api.Routes(mux)
|
||||
mux.Handle("GET /", http.FileServer(http.FS(static)))
|
||||
|
||||
Reference in new issue
Block a user