103 lines
4.1 KiB
Markdown
103 lines
4.1 KiB
Markdown
# Logstream
|
|
|
|
A simple syslog sink: receives logs over UDP/TCP on port 514, stores them in
|
|
[VictoriaLogs](https://docs.victoriametrics.com/victorialogs/), and serves a clean web
|
|
interface (light/dark theme, live search, live view, color tags, English/French UI).
|
|
|
|
```
|
|
devices ──514 udp/tcp──▶ logstream (Go) ──HTTP batches──▶ VictoriaLogs
|
|
▲ └── SSE (live) ──▶ browser
|
|
└──── API / UI ◀─────────┘
|
|
```
|
|
|
|
## Getting started
|
|
|
|
```bash
|
|
cp .env.example .env # optional
|
|
docker compose up -d --build
|
|
./tools/send-test-logs.sh # sends 100 test messages
|
|
```
|
|
|
|
Then open <http://localhost:8080>.
|
|
|
|
## Sending logs
|
|
|
|
- **rsyslog** (Linux): add `*.* @SERVER_IP:514` (UDP) or `*.* @@SERVER_IP:514` (TCP)
|
|
to `/etc/rsyslog.d/90-logstream.conf`, then run `systemctl restart rsyslog`.
|
|
- **Network gear, NAS, firewalls**: set the server IP and port 514 in their "remote syslog" settings.
|
|
- **Manual test**: `logger -n 127.0.0.1 -P 514 -d "hello error"` (util-linux) or
|
|
`echo "<14>test ok" | nc -u -w1 127.0.0.1 514`.
|
|
|
|
Supported formats: RFC 3164 (BSD) and RFC 5424. Over TCP, both "one message per line"
|
|
and "octet counting" (RFC 6587) framing are accepted.
|
|
|
|
## Search
|
|
|
|
**Simple mode** (default): each word is matched as a case-insensitive substring of the
|
|
message, host and app. Words are combined with AND.
|
|
|
|
| Input | Meaning |
|
|
|---|---|
|
|
| `error disk` | contains "error" **and** "disk" |
|
|
| `"disk full"` | contains the exact phrase |
|
|
| `error -timeout` | contains "error" but not "timeout" |
|
|
|
|
**LogsQL mode** (`Simple` / `LogsQL` button): the full VictoriaLogs query language, e.g.
|
|
`error AND host:web-01`, `app:~"ssh|nginx"`, or `* | stats by (host) count()`.
|
|
The live view is disabled in this mode.
|
|
|
|
Shortcuts: `/` focuses the search box, `Esc` clears it. Clicking a row shows all its fields.
|
|
|
|
## Settings
|
|
|
|
The gear icon opens the settings panel:
|
|
|
|
- **Language**: English or French. The choice is remembered in the browser.
|
|
- **Color tags**: each tag has a keyword, a background color (the text automatically
|
|
switches to black or white to stay readable) and options: whole word, match case,
|
|
regular expression, active. Tags are stored in `/data/tags.json` (`logstream-data` volume).
|
|
Default tags: `warning` (orange), `error` (red), `ok` (green).
|
|
|
|
## Configuration
|
|
|
|
| Variable | Default | Purpose |
|
|
|---|---|---|
|
|
| `SYSLOG_PORT` | `514` | syslog port published on the host |
|
|
| `HTTP_PORT` | `8080` | web UI port |
|
|
| `RETENTION` | `30d` | how long VictoriaLogs keeps logs |
|
|
| `AUTH_USER` / `AUTH_PASS` | empty | HTTP Basic authentication for the UI |
|
|
| `TZ` | `Europe/Paris` | time zone for RFC 3164 timestamps (which carry none) |
|
|
| `BATCH_SIZE`, `FLUSH_MS`, `QUEUE_SIZE` | `1000`, `1000`, `100000` | ingestion tuning |
|
|
|
|
## Debugging
|
|
|
|
- `docker compose logs -f logstream`: receive errors and errors sending to VictoriaLogs.
|
|
- The bottom bar shows received / stored / dropped counters and the last storage error.
|
|
- <http://localhost:9428/select/vmui>: VictoriaLogs' own UI to try LogsQL queries.
|
|
- API:
|
|
```bash
|
|
curl 'localhost:8080/api/logs?q=error&range=1h&limit=5' # the response includes the generated LogsQL query
|
|
curl localhost:8080/api/stats
|
|
curl localhost:8080/api/tags
|
|
```
|
|
- Running outside Docker (Go 1.22+): `VLOGS_URL=http://localhost:9428 DATA_DIR=./data SYSLOG_ADDR=:5514 go run .`
|
|
|
|
## Code layout
|
|
|
|
| File | Contents |
|
|
|---|---|
|
|
| `main.go` | configuration, startup, authentication |
|
|
| `syslog.go` | UDP/TCP listeners and RFC 3164 / 5424 parsing |
|
|
| `store.go` | batched inserts into VictoriaLogs and LogsQL queries |
|
|
| `query.go` | turns UI filters into LogsQL; live-view filter |
|
|
| `hub.go` | pushes new messages to browsers (SSE) |
|
|
| `tags.go` | color tag storage |
|
|
| `api.go` | `/api/*` HTTP routes |
|
|
| `web/` | UI (HTML, CSS, plain JavaScript, no build step), embedded in the binary; translations live in `web/app.js` (`I18N`) |
|
|
|
|
## Note
|
|
|
|
With Docker Desktop (macOS/Windows), the source IP seen by the container for UDP packets is
|
|
the Docker gateway. The `host` field still comes from the syslog header, which normally
|
|
carries the sender's real name.
|