WIP: Fichiers de logs : historique trouvé par date, compteur dès la première lecture #25

Closed
claude Bot wants to merge 0 commits from fix/files-ingest into main
pull from: fix/files-ingest

Requested by Cédric

Before: turning on a large file (a Traefik access log never rotated) read it from its start to find its last hour; every old line was skipped, so Settings showed "Followed: 0 entries" and new lines waited for minutes behind the history. Entries of the first poll were also missing from the count.

After: the start of the last hour is found by bisection on the line dates, new lines show at once, up to 64 MB are read per second when catching up, and the status says how much history is left ("Reading the history: N MB left").

How: backfillStart in filesource.go (files without dates read only new lines), 16 reads of 4 MB per poll, behind in /api/files, status created before the first count. Test with a 32 MB access log in Cédric's Traefik format.

🤖 Generated with Claude Code

<!-- ccr-projects-attribution: {"github_login":"vblogio"} --> _Requested by **Cédric**_ Before: turning on a large file (a Traefik access log never rotated) read it from its start to find its last hour; every old line was skipped, so Settings showed "Followed: 0 entries" and new lines waited for minutes behind the history. Entries of the first poll were also missing from the count. After: the start of the last hour is found by bisection on the line dates, new lines show at once, up to 64 MB are read per second when catching up, and the status says how much history is left ("Reading the history: N MB left"). How: `backfillStart` in `filesource.go` (files without dates read only new lines), 16 reads of 4 MB per poll, `behind` in `/api/files`, status created before the first count. Test with a 32 MB access log in Cédric's Traefik format. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude Bot added 1 commit 2026-10-05 12:42:17 +02:00
A file turned on was read from its start to find its last hour: with a
Traefik access log that holds months of lines (never rotated by
Traefik), LogStream spent minutes on old lines that were all left out,
and new lines waited behind them. Entries sent during the first poll
were also left out of the count shown in Settings.

- start of the FILES_BACKFILL history found by bisection on the dates
  of the lines; files without dates: only new lines
- up to 16 reads of 4 MB per file and per poll
- status "behind" (bytes left to read), shown as "Reading the history:
  N MB left"
- the status of a file exists before its first entries are counted

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
claude Bot closed this pull request 2026-10-05 12:42:45 +02:00

Pull request closed

Please reopen this pull request to perform a merge.
This pull request is marked as a work in progress.
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vLab-BZH/logstream#25