logstream/README.md

English | Français

Logstream

A simple syslog sink: receives logs over UDP/TCP on port 514, stores them in VictoriaLogs, and serves a clean web interface (light/dark theme, live search, live view, color tags, English/French UI).

devices ──514 udp/tcp──▶ logstream (Go) ──HTTP batches──▶ VictoriaLogs
                              ▲    └── SSE (live) ──▶ browser
                              └──── API / UI ◀─────────┘

Architecture

• Ingestion: syslog (UDP/TCP) and Docker container logs both go through sink() (reverse DNS on IP hosts), then the Store queue, which sends them in batches to VictoriaLogs.
• Live view: sink() also publishes each message to the Hub, which streams it to the browsers over SSE.
• Search: the HTTP API turns the UI filters into LogsQL queries sent to VictoriaLogs.
• Timeline: /api/histogram (histogram.go) counts the logs per interval and severity, aligned on the local time; the zoom bounds (from/to) also apply to the list and the export.
• State: tags and source settings live in /data (logstream-data volume); the logs themselves in the vlogs-data volume.

The editable source of the diagram is docs/architecture.excalidraw (open it on excalidraw.com).

Getting started

cp .env.example .env      # optional
docker compose up -d --build
./tools/send-test-logs.sh # sends 100 test messages

Then open http://localhost:8080.

Sending logs

• rsyslog (Linux): add *.* @SERVER_IP:514 (UDP) or *.* @@SERVER_IP:514 (TCP) to /etc/rsyslog.d/90-logstream.conf, then run systemctl restart rsyslog.
• Network gear, NAS, firewalls: set the server IP and port 514 in their "remote syslog" settings.
• Manual test: logger -n 127.0.0.1 -P 514 -d "hello error" (util-linux) or echo "<14>test ok" | nc -u -w1 127.0.0.1 514.

Settings > Sources > Syslog turns syslog reception on or off and chooses the protocols (UDP, TCP) without restarting; the choice is saved in /data/syslog.json. It also shows the listening state (and the error if the port is already in use). The port itself is published by docker-compose: change SYSLOG_PORT in .env, then run docker compose up -d.

Supported formats: RFC 3164 (BSD) and RFC 5424. Over TCP, both "one message per line" and "octet counting" (RFC 6587) framing are accepted.

Search

Simple mode (default): each word is matched as a case-insensitive substring of the message, host and app. Words are combined with AND.

Input	Meaning
error disk	contains "error" and "disk"
"disk full"	contains the exact phrase
error -timeout	contains "error" but not "timeout"

LogsQL mode (Simple / LogsQL button): the full VictoriaLogs query language, e.g. error AND host:web-01, app:~"ssh|nginx", or * | stats by (host) count(). The live view is disabled in this mode.

Each row shows, from left to right: the reception time (server clock), the timestamp found in the message itself (msg_time), severity, host, app and message. Click a host or an app to filter on it.

Logs are indexed, searched and sorted by reception time: devices with a wrong clock (e.g. access points whose NTP fails) still show up in the right time range. Logs stored by versions before this change are indexed by their message timestamp; purge them from Settings to start clean.

Severity badges err/crit and warning use the colors of the error and warning tags.

Shortcuts: / focuses the search box, Esc clears it. Clicking a row shows all its fields.

The column headers stay visible while scrolling. Drag the edge of a header (received, message time, severity, host, app) to resize the column, double-click it to go back to the automatic width; the message takes the remaining space. Widths are remembered by the browser (Settings > Interface > Reset column widths restores them all). On phones the list keeps its two-line layout without columns.

Timeline

The timeline above the list shows the volume of logs per interval, counted by reception time on the server clock (so it matches the times shown in the rows).

• Hover an interval: its bounds, total and detail per severity.
• Click a bar to zoom on that interval, or drag across several bars to zoom on the selection. The time range then shows the zoomed period ("× Reset zoom" or any other range leaves it); the list, the counters and the CSV export follow the zoom, and the live view pauses.
• In live mode the last interval grows as messages arrive, and the timeline reloads at each new interval. Nothing is refreshed while the browser tab is hidden; it catches up when shown again.

Settings > Interface > Timeline (remembered per browser): scale (linear, √ by default, log), height (S/M/L: 40/80/120 px), color (stacked by severity, intensity compared with the median of the window: calm, burst above 3×, anomaly above 10×, or none), bars or area, division (automatic, about 100 intervals, or fixed: 1 s, 10 s, 1 min, 5 min, 1 h, 1 day) and refresh (off, 5 s, 15 s, 30 s, 1 min, or at each new interval). A fixed division that would exceed 300 intervals over the range is enlarged (shown as "enlarged"). Intervals are aligned on the local time of the time zone chosen in Settings (days start at local midnight).

API: curl 'localhost:8080/api/histogram?range=24h&step=auto&tz=Europe/Paris' returns step (ms), start (ms), count, now (server clock) and the non-empty buckets (i = interval index, n = total, sev = count per severity). Every log endpoint also accepts from / to (Unix milliseconds) instead of range.

Docker container logs

Logstream also collects the logs of the Docker containers running on the machine where it is installed (DOCKER_LOGS=on, the default in docker-compose.yml). They are searched, filtered, colored and exported like syslog messages:

• host is the Docker host name, app the compose service (or the container name), and each log also carries container, container_id, image, compose_project, compose_service and stream (stdout/stderr), visible in the row details.
• The Source filter shows only syslog or only Docker logs; Docker rows have a small cube before the app name, colored like its compose project.
• The severity comes from the line itself when the application writes it: JSON ("level":"error"), logfmt (level=warn), [ERROR], or an upper-case level word at the start of the line (ERROR, WARN…). Otherwise it is info. Terminal color codes are removed.
• Settings > Sources shows one label per container (project/service) in a single field: followed containers in color, then not followed ones in grey; a click switches a label. The color identifies the compose project, and Docker app names in the log list use the same color. Stopped containers are hidden by default ("Show stopped containers" displays them, dashed, and they stay editable). A filter box and "Enable all" / "Disable all" (applied to the labels shown) help with many containers. New containers are followed automatically unless that option is turned off. Choices are saved per compose service (or container name) in /data/docker.json, so they survive re-creations.
• Logstream remembers the position read in each container (/data/docker-state.json): after a restart it resumes without losing or duplicating lines. A container seen for the first time is read from DOCKER_BACKFILL ago (1 hour by default).
• Logstream itself and the proxy below are never collected; add the label logstream.exclude=true to any other container to exclude it for good.

Security: access to the Docker socket is equivalent to root on the machine. Logstream therefore goes through docker-socket-proxy, which only lets through listing containers, reading logs, events and engine info (GET only).

Host system logs

Logstream can also collect the system logs of the machine hosting the stack, without configuring anything on the host. The source is off by default: turn it on in Settings > Sources > Host system logs (saved in /data/hostlogs.json).

• docker-compose.yml mounts /var/log and /run/log/journal read-only under /host.
• When the host runs systemd, Logstream reads the systemd journal files directly (no journalctl needed in the image): host is the machine name, app the program (SYSLOG_IDENTIFIER), severity and facility come from the journal, and the systemd unit is kept in unit. Otherwise it follows the text files of /var/log (syslog, messages, *.log), parsed like syslog lines, with the file name in log_file.
• These logs have the host source: the Source filter shows them alone.
• When the source is turned on, the last hour is read first (HOST_LOGS_BACKFILL); the position reached is saved in /data/hostlogs-state.json, so a restart neither loses nor duplicates entries.
• Permissions: the container runs as an unprivileged user and gets the adm group (gid 4), which can read the journal and /var/log on Debian and Ubuntu. On other systems, set HOST_LOGS_GID in .env to the gid of systemd-journal (getent group systemd-journal | cut -d: -f3). Settings > Sources shows a clear message when access is denied.
• Limits: journal fields compressed by journald (messages longer than 512 bytes, compressed with zstd/lz4/xz) cannot be decoded without extra libraries; they are counted in Settings and shown as "(compressed journal entry)". Rotated or compressed text files (*.1, *.gz) are not read.

CSV export

The Export button (next to the log count) downloads every stored log matching the current filters (search, time range, severity, host, app), newest first, up to EXPORT_MAX rows (100,000 by default), not only the rows on screen. Two variants:

• CSV: comma separated, UTF-8.
• CSV for Excel: semicolon separated with a UTF-8 BOM, so French Excel opens it directly with accents. Cells starting with =, +, - or @ are prefixed with ' so that a crafted log message cannot run as a formula.

Columns: received, message_time (both as YYYY-MM-DD HH:MM:SS.mmm in the time zone chosen in Settings), severity, facility, host, host_ip, app, pid, source_ip, proto, message. In LogsQL mode, only the filter part is supported (no | pipes).

From the command line: curl -o logs.csv 'localhost:8080/api/export.csv?q=error&range=24h&tz=Europe/Paris'.

Settings

The gear icon opens the settings, organized in tabs. Everything except color tags is remembered per browser.

• Localization
  • Language: English or French.
  • Date & time: time zone (browser zone, UTC, or about 80 common zones) and the display format of the reception time: DD/MM/YYYY HH:MM:SS (default, the usual French display), with milliseconds, YYYY-MM-DD, 12-hour clock, ISO 8601 or Unix epoch. The time zone applies to every date shown.
• Filters: color tags. Each tag has a keyword, a background color (the text automatically switches to black or white to stay readable) and options: whole word, match case, regular expression, active. Tags are stored on the server in /data/tags.json (logstream-data volume), so they are shared by every browser. Default tags (pastel): warning (orange), error (red), ok (green). Default tags still using the colors of earlier versions are switched to the pastel ones automatically.
• Interface
  • Theme: System (follows the computer/phone preference), Light or Dark. The sun/moon button in the header switches between light and dark.
  • Log display: font size (tiny, small, medium, large) and font: the system monospace font, or one of 12 free fonts made for dense text (JetBrains Mono, Fira Code, Source Code Pro, IBM Plex Mono, Cascadia Code, Roboto Mono, Ubuntu Mono, Inconsolata, Red Hat Mono, Noto Sans Mono, Victor Mono, DM Mono). They are loaded by the browser from Bunny Fonts, a privacy-friendly European font service; without internet access, the system font is used. Ligatures are disabled so -> or != show as typed.
• Data: "Delete all logs" permanently erases every stored log (you must type PURGE to confirm). Tags and settings are kept. VictoriaLogs needs -delete.enable (already set in docker-compose.yml); set ALLOW_PURGE=false to disable the feature. Anyone who can open the UI can purge: set AUTH_USER / AUTH_PASS if the UI is reachable by others.

Host names (reverse DNS)

When a device sends its IP address as host name (or no host name at all), Logstream looks up its DNS name (PTR record) and stores the name in host and the IP in host_ip. Results are cached (1 hour, 10 minutes when there is no name). Logs stored earlier with an IP are resolved on display, and the host filter shows name (IP).

The container uses Docker's DNS, which forwards to the host's resolvers. If your local names are only known by your router or a local DNS (Pi-hole, AdGuard, Unbound…), set DNS_SERVER=192.168.1.1 (its address). Set RDNS=off to disable lookups.

Configuration

Variable	Default	Purpose
SYSLOG_PORT	514	syslog port published on the host
HTTP_PORT	8080	web UI port
RETENTION	30d	how long VictoriaLogs keeps logs
AUTH_USER / AUTH_PASS	empty	HTTP Basic authentication for the UI
RDNS	on	resolve IP hosts to DNS names
DNS_SERVER	empty	DNS server for reverse lookups (ip or ip:port)
ALLOW_PURGE	true	allow "Delete all logs" in Settings
EXPORT_MAX	100000	maximum number of rows in a CSV export
DOCKER_LOGS	on in compose	collect the logs of the local Docker containers
DOCKER_HOST	tcp://docker-proxy:2375 in compose	Docker API address (unix:///var/run/docker.sock outside compose)
DOCKER_BACKFILL	1h	history read from a container seen for the first time
HOST_LOGS_GID	4 (adm) in compose	group given to the container to read the host logs
HOST_LOGS_BACKFILL	1h	history read when the host system logs source is turned on
HOST_LOGS_ROOT	/host	where the host directories are mounted
TZ	Europe/Paris	time zone for RFC 3164 timestamps (which carry none)
BATCH_SIZE, FLUSH_MS, QUEUE_SIZE	1000, 1000, 100000	ingestion tuning

Debugging

• docker compose logs -f logstream: receive errors and errors sending to VictoriaLogs.
• The bottom bar shows received / stored / dropped counters and the last storage error.
• http://localhost:9428/select/vmui: VictoriaLogs' own UI to try LogsQL queries.
• API:

  curl 'localhost:8080/api/logs?q=error&range=1h&limit=5'   # the response includes the generated LogsQL query
  curl localhost:8080/api/stats
  curl localhost:8080/api/tags
  

• Running outside Docker (Go 1.22+): VLOGS_URL=http://localhost:9428 DATA_DIR=./data SYSLOG_ADDR=:5514 go run .

Updating

Every image version is pinned, so a docker compose pull or a rebuild never changes a component behind your back:

Where	Image	Version
docker-compose.yml	victoriametrics/victoria-logs	v1.52.0
docker-compose.yml	tecnativa/docker-socket-proxy	v0.5.0
Dockerfile (build)	golang	1.27.1-alpine3.24
Dockerfile (runtime)	alpine	3.24.2

To update one of them:

1. Check the release notes: VictoriaLogs, docker-socket-proxy, Go, Alpine. VictoriaLogs keeps its storage format across minor versions; read the changelog before a major version change.
2. Change the version in the file above, then run docker compose up -d --build.
3. Check the bottom bar (received / stored / dropped) and Settings > Sources. To go back, restore the previous version and run the same command.

Code layout

File	Contents
main.go	configuration, startup, authentication
syslog.go	UDP/TCP listeners and RFC 3164 / 5424 parsing
store.go	batched inserts into VictoriaLogs and LogsQL queries
query.go	turns UI filters into LogsQL; live-view filter
histogram.go	timeline: division, interval alignment, /api/histogram
hub.go	pushes new messages to browsers (SSE)
rdns.go	cached reverse DNS lookups
export.go	streamed CSV export
docker.go	Docker container logs (API, followers, positions, level detection)
syslogserver.go	syslog listeners opened and closed from Settings > Sources
hostlogs.go, journal.go	host system logs: systemd journal reader (no journalctl) and /var/log follower
tags.go	color tag storage
api.go	/api/* HTTP routes
web/	UI (HTML, CSS, plain JavaScript, no build step), embedded in the binary; translations live in web/app.js (I18N)

Note

With Docker Desktop (macOS/Windows), the source IP seen by the container for UDP packets is the Docker gateway. The host field still comes from the syslog header, which normally carries the sender's real name.