Log files: find the history by date, count from the first poll

A file turned on was read from its start to find its last hour: with a
Traefik access log that holds months of lines (never rotated by
Traefik), LogStream spent minutes on old lines that were all left out,
and new lines waited behind them. Entries sent during the first poll
were also left out of the count shown in Settings.

- start of the FILES_BACKFILL history found by bisection on the dates
  of the lines; files without dates: only new lines
- up to 16 reads of 4 MB per file and per poll
- status "behind" (bytes left to read), shown as "Reading the history:
  N MB left"
- the status of a file exists before its first entries are counted

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
cedricandClaude Opus 5.5 committed 2026-10-05 12:42:09 +02:00
1 parent 194edee6de
commit 1a3b3ee23a
5 files changed
+140 -16

No files matched your search

+4 -2
View File
@@ -312,8 +312,10 @@ optional **host** (default: `/host/etc/hostname`, else `localhost`) and a switch
- **Display**: source `file`, the path in `log_file`. In the list these lines have a green left
edge, a light tint and a **FILE** badge before the app; the **Source** filter shows them alone.
- When a file is turned on, the lines of the last hour already in it are read first
(`FILES_BACKFILL`, `0` = only new lines; lines without a timestamp are then left out); a file
created later is read from its start. The inode and an offset are saved per file in
(`FILES_BACKFILL`, `0` = only new lines; lines without a timestamp are then left out). The
start of that hour is found by bisection on the dates of the lines, so a large file that holds
months of logs is not read from its start; a file created later is read from its start, up to
64 MB per second, the status showing what is left to read. The inode and an offset are saved per file in
`/data/files-state.json`: after a restart reading resumes there; after a rotation
(`access.log` renamed `access.log.1`), the end of the old file is read before the new one;
a file truncated in place (`copytruncate`) is read again from its start.