Before: turning on a large file (a Traefik access log never rotated) read it from its start to find its last hour; every old line was skipped, so Settings showed "Followed: 0 entries" and new lines waited for minutes behind the history. Entries of the first poll were also missing from the count.
After: the start of the last hour is found by bisection on the line dates, new lines show at once, up to 64 MB are read per second when catching up, and the status says how much history is left ("Reading the history: N MB left").
How: backfillStart in filesource.go (files without dates read only new lines), 16 reads of 4 MB per poll, behind in /api/files, status created before the first count. Test with a 32 MB access log in Cédric's Traefik format.
<!-- ccr-projects-attribution: {"github_login":"vblogio"} -->
_Requested by **Cédric**_
Before: turning on a large file (a Traefik access log never rotated) read it from its start to find its last hour; every old line was skipped, so Settings showed "Followed: 0 entries" and new lines waited for minutes behind the history. Entries of the first poll were also missing from the count.
After: the start of the last hour is found by bisection on the line dates, new lines show at once, up to 64 MB are read per second when catching up, and the status says how much history is left ("Reading the history: N MB left").
How: `backfillStart` in `filesource.go` (files without dates read only new lines), 16 reads of 4 MB per poll, `behind` in `/api/files`, status created before the first count. Test with a 32 MB access log in Cédric's Traefik format.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
A file turned on was read from its start to find its last hour: with a
Traefik access log that holds months of lines (never rotated by
Traefik), LogStream spent minutes on old lines that were all left out,
and new lines waited behind them. Entries sent during the first poll
were also left out of the count shown in Settings.
- start of the FILES_BACKFILL history found by bisection on the dates
of the lines; files without dates: only new lines
- up to 16 reads of 4 MB per file and per poll
- status "behind" (bytes left to read), shown as "Reading the history:
N MB left"
- the status of a file exists before its first entries are counted
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Requested by Cédric
Before: turning on a large file (a Traefik access log never rotated) read it from its start to find its last hour; every old line was skipped, so Settings showed "Followed: 0 entries" and new lines waited for minutes behind the history. Entries of the first poll were also missing from the count.
After: the start of the last hour is found by bisection on the line dates, new lines show at once, up to 64 MB are read per second when catching up, and the status says how much history is left ("Reading the history: N MB left").
How:
backfillStartinfilesource.go(files without dates read only new lines), 16 reads of 4 MB per poll,behindin/api/files, status created before the first count. Test with a 32 MB access log in Cédric's Traefik format.🤖 Generated with Claude Code
Pull request closed