cedricandClaude Opus 5.5
e583ab4b87
- _time is now the reception time; the device timestamp moves to msg_time. Devices with a wrong clock were indexed in the past and escaped time ranges and the host list. - Host/app lists also include values seen in displayed and live logs; clicking a host or app cell filters on it. - Severity badges err/crit and warning use the colors of the error and warning tags; default tags are now pastel (old default colors migrated). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
139 lines
6.3 KiB
Markdown
139 lines
6.3 KiB
Markdown
# Logstream
|
|
|
|
A simple syslog sink: receives logs over UDP/TCP on port 514, stores them in
|
|
[VictoriaLogs](https://docs.victoriametrics.com/victorialogs/), and serves a clean web
|
|
interface (light/dark theme, live search, live view, color tags, English/French UI).
|
|
|
|
```
|
|
devices ──514 udp/tcp──▶ logstream (Go) ──HTTP batches──▶ VictoriaLogs
|
|
▲ └── SSE (live) ──▶ browser
|
|
└──── API / UI ◀─────────┘
|
|
```
|
|
|
|
## Getting started
|
|
|
|
```bash
|
|
cp .env.example .env # optional
|
|
docker compose up -d --build
|
|
./tools/send-test-logs.sh # sends 100 test messages
|
|
```
|
|
|
|
Then open <http://localhost:8080>.
|
|
|
|
## Sending logs
|
|
|
|
- **rsyslog** (Linux): add `*.* @SERVER_IP:514` (UDP) or `*.* @@SERVER_IP:514` (TCP)
|
|
to `/etc/rsyslog.d/90-logstream.conf`, then run `systemctl restart rsyslog`.
|
|
- **Network gear, NAS, firewalls**: set the server IP and port 514 in their "remote syslog" settings.
|
|
- **Manual test**: `logger -n 127.0.0.1 -P 514 -d "hello error"` (util-linux) or
|
|
`echo "<14>test ok" | nc -u -w1 127.0.0.1 514`.
|
|
|
|
Supported formats: RFC 3164 (BSD) and RFC 5424. Over TCP, both "one message per line"
|
|
and "octet counting" (RFC 6587) framing are accepted.
|
|
|
|
## Search
|
|
|
|
**Simple mode** (default): each word is matched as a case-insensitive substring of the
|
|
message, host and app. Words are combined with AND.
|
|
|
|
| Input | Meaning |
|
|
|---|---|
|
|
| `error disk` | contains "error" **and** "disk" |
|
|
| `"disk full"` | contains the exact phrase |
|
|
| `error -timeout` | contains "error" but not "timeout" |
|
|
|
|
**LogsQL mode** (`Simple` / `LogsQL` button): the full VictoriaLogs query language, e.g.
|
|
`error AND host:web-01`, `app:~"ssh|nginx"`, or `* | stats by (host) count()`.
|
|
The live view is disabled in this mode.
|
|
|
|
Each row shows, from left to right: the **reception time** (server clock), the timestamp
|
|
found in the message itself (`msg_time`), severity, host, app and message. Click a host or an
|
|
app to filter on it.
|
|
|
|
Logs are indexed, searched and sorted by **reception time**: devices with a wrong clock
|
|
(e.g. access points whose NTP fails) still show up in the right time range. Logs stored by
|
|
versions before this change are indexed by their message timestamp; purge them from
|
|
Settings to start clean.
|
|
|
|
Severity badges `err`/`crit` and `warning` use the colors of the `error` and `warning` tags.
|
|
|
|
Shortcuts: `/` focuses the search box, `Esc` clears it. Clicking a row shows all its fields.
|
|
|
|
## Settings
|
|
|
|
The gear icon opens the settings panel:
|
|
|
|
- **Language**: English or French. The choice is remembered in the browser.
|
|
- **Date & time**: time zone (browser zone, UTC, or about 80 common zones) and the display
|
|
format of the reception time: `DD/MM/YYYY HH:MM:SS` (default, the usual French display),
|
|
with milliseconds, `YYYY-MM-DD`, 12-hour clock, ISO 8601 or Unix epoch. The time zone
|
|
applies to every date shown. Remembered in the browser.
|
|
- **Danger zone**: "Delete all logs" permanently erases every stored log (you must type
|
|
`PURGE` to confirm). Tags and settings are kept. VictoriaLogs needs `-delete.enable`
|
|
(already set in `docker-compose.yml`); set `ALLOW_PURGE=false` to disable the feature.
|
|
Anyone who can open the UI can purge: set `AUTH_USER` / `AUTH_PASS` if the UI is reachable
|
|
by others.
|
|
|
|
## Host names (reverse DNS)
|
|
|
|
When a device sends its IP address as host name (or no host name at all), Logstream looks up
|
|
its DNS name (PTR record) and stores the name in `host` and the IP in `host_ip`. Results are
|
|
cached (1 hour, 10 minutes when there is no name). Logs stored earlier with an IP are resolved
|
|
on display, and the host filter shows `name (IP)`.
|
|
|
|
The container uses Docker's DNS, which forwards to the host's resolvers. If your local names
|
|
are only known by your router or a local DNS (Pi-hole, AdGuard, Unbound…), set
|
|
`DNS_SERVER=192.168.1.1` (its address). Set `RDNS=off` to disable lookups.
|
|
- **Color tags**: each tag has a keyword, a background color (the text automatically
|
|
switches to black or white to stay readable) and options: whole word, match case,
|
|
regular expression, active. Tags are stored in `/data/tags.json` (`logstream-data` volume).
|
|
Default tags (pastel): `warning` (orange), `error` (red), `ok` (green). Default tags
|
|
still using the colors of earlier versions are switched to the pastel ones automatically.
|
|
|
|
## Configuration
|
|
|
|
| Variable | Default | Purpose |
|
|
|---|---|---|
|
|
| `SYSLOG_PORT` | `514` | syslog port published on the host |
|
|
| `HTTP_PORT` | `8080` | web UI port |
|
|
| `RETENTION` | `30d` | how long VictoriaLogs keeps logs |
|
|
| `AUTH_USER` / `AUTH_PASS` | empty | HTTP Basic authentication for the UI |
|
|
| `RDNS` | `on` | resolve IP hosts to DNS names |
|
|
| `DNS_SERVER` | empty | DNS server for reverse lookups (`ip` or `ip:port`) |
|
|
| `ALLOW_PURGE` | `true` | allow "Delete all logs" in Settings |
|
|
| `TZ` | `Europe/Paris` | time zone for RFC 3164 timestamps (which carry none) |
|
|
| `BATCH_SIZE`, `FLUSH_MS`, `QUEUE_SIZE` | `1000`, `1000`, `100000` | ingestion tuning |
|
|
|
|
## Debugging
|
|
|
|
- `docker compose logs -f logstream`: receive errors and errors sending to VictoriaLogs.
|
|
- The bottom bar shows received / stored / dropped counters and the last storage error.
|
|
- <http://localhost:9428/select/vmui>: VictoriaLogs' own UI to try LogsQL queries.
|
|
- API:
|
|
```bash
|
|
curl 'localhost:8080/api/logs?q=error&range=1h&limit=5' # the response includes the generated LogsQL query
|
|
curl localhost:8080/api/stats
|
|
curl localhost:8080/api/tags
|
|
```
|
|
- Running outside Docker (Go 1.22+): `VLOGS_URL=http://localhost:9428 DATA_DIR=./data SYSLOG_ADDR=:5514 go run .`
|
|
|
|
## Code layout
|
|
|
|
| File | Contents |
|
|
|---|---|
|
|
| `main.go` | configuration, startup, authentication |
|
|
| `syslog.go` | UDP/TCP listeners and RFC 3164 / 5424 parsing |
|
|
| `store.go` | batched inserts into VictoriaLogs and LogsQL queries |
|
|
| `query.go` | turns UI filters into LogsQL; live-view filter |
|
|
| `hub.go` | pushes new messages to browsers (SSE) |
|
|
| `rdns.go` | cached reverse DNS lookups |
|
|
| `tags.go` | color tag storage |
|
|
| `api.go` | `/api/*` HTTP routes |
|
|
| `web/` | UI (HTML, CSS, plain JavaScript, no build step), embedded in the binary; translations live in `web/app.js` (`I18N`) |
|
|
|
|
## Note
|
|
|
|
With Docker Desktop (macOS/Windows), the source IP seen by the container for UDP packets is
|
|
the Docker gateway. The `host` field still comes from the syslog header, which normally
|
|
carries the sender's real name.
|