Reading API /api/v1/logs with a stable cursor, and API tokens

GET /api/v1/logs?query=<LogsQL>&since=<cursor|time|duration>&limit=<n>
returns the logs after a cursor, oldest first, with next_cursor and
has_more. The cursor is a position on _time (the reception time) plus
the number of rows already returned among those sharing that exact
time, ordered by a hash of their content, so following the cursors
gives every row once even within groups cut by a page. Rows count once
older than the lag (30 s), and nothing is returned while the disk
spool still holds logs.

API tokens (Settings > API) give read-only access to /api/v1/ as
"Authorization: Bearer lst_...", in every auth mode; only their SHA-256
is kept in /data/api_tokens.json. README (EN/FR) documents the route and
an n8n workflow keeping the cursor in $getWorkflowStaticData.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
cedricandClaude Opus 5.5 committed 2026-11-01 15:47:57 +01:00
1 parent d61384ed2c
commit c8f45044a3
12 files changed
+1327 -7

No files matched your search

+75 -1
View File
@@ -354,13 +354,82 @@ chosen in Settings), `severity`, `facility`, `host`, `host_ip`, `app`, `pid`, `s
From the command line: `curl -o logs.csv 'localhost:8080/api/export.csv?q=error&range=24h&tz=Europe/Paris'`.
## Reading API (n8n, scripts)
`GET /api/v1/logs` returns the stored logs that follow a cursor, oldest first, so that a tool
called on a schedule (n8n, a script, a SIEM connector) gets every new log once, without gaps
or duplicates, even when many logs share the same timestamp.
Access goes through an **API token**, created in *Settings > API* by an administrator and sent
as `Authorization: Bearer lst_…`. A token is read-only and only opens `/api/v1/`; it works
whatever the [authentication](#authentication) mode (`local`, `oidc`, or none). It is shown once,
when it is created: LogStream only keeps its SHA-256 hash, in `/data/api_tokens.json`.
Revoking it cuts off access at once; the list shows when and from where each token was last
used. A logged-in browser session can call the route too.
| Parameter | Meaning |
|---|---|
| `query` | LogsQL filter (default `*`), e.g. `severity:in(err,crit) app:sshd`. Pipes (`\| ...`) are refused. |
| `since` | where to start: the `next_cursor` of the previous answer, an RFC 3339 time (`2026-11-01T08:00:00Z`) or a duration back from now (`15m`, `6h`, `7d`). Default: `1h`. |
| `limit` | rows per answer, 1 to 10,000 (default 1,000). |
| `lag` | only logs received at least this long ago are returned (default `30s`, at most `1h`), so that the logs still in the queue or waiting for their DNS name are not skipped. |
```json
{
"rows": [{"_time": "2026-11-01T14:02:11.52Z", "_msg": "…", "host": "fw01", "app": "sshd", "severity": "err", …}],
"next_cursor": "eyJ0IjoiMjAyNi0xMS0wMVQxNDowMjoxMS41MjAwMDAwMDFaIiwicSI6IjJlYjE4YzNhIn0",
"has_more": false,
"until": "2026-11-01T14:05:30.1Z"
}
```
Rows have the stored fields (as in the UI's JSON view), sorted by `_time`, the reception time.
Always send back `next_cursor`, even when `rows` is empty. `has_more: true` means more logs are
ready: call again at once. `waiting: true` means logs from the [disk buffer](#configuration) are
still being sent to VictoriaLogs: nothing is returned until they are stored, so none is missed.
The cursor belongs to its `query`: changing the query keeps the position in time. Errors come
as `{"error", "code"}` with status 400 (bad parameter or LogsQL), 401 (invalid or revoked token)
or 502 (VictoriaLogs unreachable).
```bash
curl -H "Authorization: Bearer lst_…" \
"https://logs.example.org/api/v1/logs?query=severity:err&since=15m&limit=1000"
```
### Example: n8n workflow
Four nodes, the cursor kept in the workflow's static data between runs:
1. **Schedule Trigger**: every minute.
2. **Code** (*Run Once for All Items*), reads the cursor:
```js
const data = $getWorkflowStaticData('global');
return [{ json: { since: data.cursor || '15m' } }];
```
3. **HTTP Request**: method `GET`, URL `https://logs.example.org/api/v1/logs`.
*Authentication*: Generic Credential Type > Header Auth, with name `Authorization` and value
`Bearer lst_…`. *Send Query Parameters*: `query` = `severity:in(emerg,alert,crit,err)`,
`since` = `{{ $json.since }}`, `limit` = `1000`.
4. **Code** (*Run Once for All Items*), saves the cursor and turns each log into an item:
```js
const data = $getWorkflowStaticData('global');
const page = $input.first().json;
data.cursor = page.next_cursor;
return page.rows.map((row) => ({ json: row }));
```
Then connect what the logs should feed (Slack, mail, ticket, database…). n8n only keeps static
data for runs of the active workflow, not for manual test runs: while testing, each run starts
again from `15m`. When more than 1,000 logs arrive per minute, `has_more` is true and the
following runs catch up; raise `limit` or run the trigger more often if needed.
## Settings
The gear icon opens the settings, organized in tabs. Changes show at once but are only kept
with the **Save** button, under the tab labels: it takes the accent color while changes are
waiting. **Cancel** goes back to the saved settings, and closing the window with unsaved changes
asks before dropping them. Localization and Appearance are remembered per browser; color tags
and log sources are saved on the server. Purging and resetting column widths act at once.
and log sources are saved on the server. Purging, API tokens and resetting column widths act at once.
- **Localization**
- *Language*: English or French.
@@ -408,6 +477,9 @@ and log sources are saved on the server. Purging and resetting column widths act
(already set in `docker-compose.yml`). The feature is off by default: set `ALLOW_PURGE=true`
to allow it. Any admin can then purge: turn on [authentication](#authentication) if the UI
is reachable by others.
- **API**: API tokens for the [reading API](#reading-api-n8n-scripts): create one with a name
(shown once, copy it then), see when and from where each was last used, revoke it. Reserved
to administrators.
## Authentication
@@ -577,6 +649,8 @@ To update one of them:
| `hub.go` | pushes new messages to browsers (SSE) |
| `rdns.go` | cached reverse DNS lookups |
| `export.go` | streamed CSV export |
| `logsapi.go` | reading API `/api/v1/logs`: cursor, groups of logs with the same time |
| `apitokens.go` | API tokens: storage (hashed), Bearer check, `/api/tokens` |
| `docker.go` | Docker container logs (API, followers, positions, level detection) |
| `syslogserver.go` | syslog listeners opened and closed from Settings > Log sources |
| `hostlogs.go`, `journal.go` | host system logs: systemd journal reader (no `journalctl`) and `/var/log` follower |