Log files: find the history by date, count from the first poll
A file turned on was read from its start to find its last hour: with a Traefik access log that holds months of lines (never rotated by Traefik), LogStream spent minutes on old lines that were all left out, and new lines waited behind them. Entries sent during the first poll were also left out of the count shown in Settings. - start of the FILES_BACKFILL history found by bisection on the dates of the lines; files without dates: only new lines - up to 16 reads of 4 MB per file and per poll - status "behind" (bytes left to read), shown as "Reading the history: N MB left" - the status of a file exists before its first entries are counted Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
194edee6de
commit
1a3b3ee23a
5 files changed
+140
-16
No files matched your search
@@ -312,8 +312,10 @@ optional **host** (default: `/host/etc/hostname`, else `localhost`) and a switch
|
||||
- **Display**: source `file`, the path in `log_file`. In the list these lines have a green left
|
||||
edge, a light tint and a **FILE** badge before the app; the **Source** filter shows them alone.
|
||||
- When a file is turned on, the lines of the last hour already in it are read first
|
||||
(`FILES_BACKFILL`, `0` = only new lines; lines without a timestamp are then left out); a file
|
||||
created later is read from its start. The inode and an offset are saved per file in
|
||||
(`FILES_BACKFILL`, `0` = only new lines; lines without a timestamp are then left out). The
|
||||
start of that hour is found by bisection on the dates of the lines, so a large file that holds
|
||||
months of logs is not read from its start; a file created later is read from its start, up to
|
||||
64 MB per second, the status showing what is left to read. The inode and an offset are saved per file in
|
||||
`/data/files-state.json`: after a restart reading resumes there; after a rotation
|
||||
(`access.log` renamed `access.log.1`), the end of the old file is read before the new one;
|
||||
a file truncated in place (`copytruncate`) is read again from its start.
|
||||
|
||||
Reference in new issue
Block a user