Syslog receiver monitoring with Topologix

Monitoring & Observability

A syslog listener on one of your proxy agents — RFC 5424 and RFC 3164 over UDP and TCP — that counts error lines per service (or per host), learns each one’s usual error count and alerts on the deviation, exactly like the Graylog All Services source. Error types come from the MSGID (or a structured-data parameter) or from fingerprints of the lines, and the latest lines are kept for View logs. Nothing leaves the proxy but counts and a few redacted sample lines.

Requirements

• A proxy agent on proxy 1.216.0 or newer, on a network your devices can reach. The listener runs only on a proxy — never on the central server, where a UDP port would face the internet and a datagram carries no tenant. Saving the integration against an older proxy is refused with “update the proxy to 1.216.0”.
• Senders that can reach the proxy on port 1514, UDP and / or TCP (both on by default). Port 514 needs root on Linux, so the proxy does not bind it — see Port 514 below for the forward.
• No credentials. The sender allow-list is the gate; TLS syslog (RFC 5425) is not supported in this version.

1 · Set up the integration

1. Open the Integrations pageClick Integrations in the toolbar and pick Syslog receiver from the catalogue.
2. Pick the proxy and the portChoose the proxy that receives the logs (required — a proxy you can use), the port (default 1514) and the transports (UDP and TCP; turning both off is refused). Two syslog integrations on one proxy must use the same port.
3. Allowed senders (optional)A list of addresses or CIDRs (comma or newline separated). Left empty, the proxy accepts the private ranges — 10/8, 172.16/12, 192.168/16, 100.64/10, loopback, link-local, fc00::/7 — plus the subnets of its own interfaces, so routed VLANs work without a list. A sender outside the list is counted as rejected, never parsed.
4. Point your devices at itThe drawer prints an rsyslog forwarding rule with a Copy button (TCP octet-counted when TCP is on, else UDP) and the 514 → 1514 sentence; its second tab, Kernel console (netconsole), is the console-host rule described below. Check listener asks the proxy whether the port is bound, how many lines it received and from which senders.

2 · Add it to a flow

1. Select the nodeClick the node, open Metrics, then + Add and choose the syslog integration. The source runs on the integration’s proxy.
2. Choose what a “service” isService field: appname (APP-NAME in 5424, TAG in 3164 — the default; falls back to the HOSTNAME, then the sender address), hostname, or appname@hostname. Use host name for network gear — a Cisco ASA sends no HOSTNAME and its %ASA-6-302013 token becomes the TAG, so with appname every ASA message id would be its own service.
3. Errors, types and filtersError severity: ≤ err (3, the default) or ≤ warning (4). Error type: the RFC 5424 MSGID (default), a structured-data parameter <SD-ID>.<param>, or none — lines without a type are fingerprinted by the server (marked approx). Optional host filter (an address / CIDR matches the sender, anything else is a glob on HOSTNAME) and app filter (globs on APP-NAME / TAG).
4. Alarm rule and saveThe same rule block as Graylog All Services — Baseline (default) or Count threshold, ratio, minimum excess, N of M, new-type warnings, Max services (200). Set the interval and click Add Source. Changing the service field, severity, filters or type restarts that source’s counts; a threshold or alarm edit keeps them.

3 · Forwarding, framing and counts

Port 514 → 1514

Most devices can be told to send to 1514. For those that can only send to 514, forward the port on the proxy host — or let the proxy bind 514 itself:

• A REDIRECT applies to traffic arriving from other hosts; a sender on the proxy host itself should use 1514 directly.
• Port, bind address, UDP / TCP and the allowed sender CIDRs are set per proxy in Settings → Proxies → Proxy settings → Syslog (proxy ≥ 1.235.0; default port 1514). A syslog {} block left in an older proxy’s config.json is imported once on its first contact after the upgrade — then remove it from the file, as the drawer asks.

iptables (UDP and TCP)

sudo iptables -t nat -A PREROUTING -p udp --dport 514 -j REDIRECT --to-ports 1514
sudo iptables -t nat -A PREROUTING -p tcp --dport 514 -j REDIRECT --to-ports 1514

nftables

sudo nft add table ip nat
sudo nft 'add chain ip nat prerouting { type nat hook prerouting priority -100; }'
sudo nft add rule ip nat prerouting udp dport 514 redirect to :1514
sudo nft add rule ip nat prerouting tcp dport 514 redirect to :1514

Or bind 514 directly: grant the capability, then set the port to 514

# systemd unit of the proxy
[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE

Sending from rsyslog

On a Linux host, one file forwards everything. TCP with octet counting is the safest framing for multi-line messages; UDP is fine for chatty, loss-tolerant senders.

• Network gear: point the device’s syslog host at the proxy, port 1514 (or 514 with the forward), and set its logging level to at least errors. Pick Service field → hostname on the source.

/etc/rsyslog.d/90-topologix.conf

# forward to the Topologix proxy (TCP, octet-counted, RFC 5424)
*.* action(type="omfwd" target="<proxy-address>" port="1514" protocol="tcp" TCP_Framing="octet-counted" template="RSYSLOG_SyslogProtocol23Format"
         queue.type="LinkedList" queue.size="10000" action.resumeRetryCount="-1")
# or UDP:
# *.* action(type="omfwd" target="<proxy-address>" port="1514" protocol="udp" template="RSYSLOG_SyslogProtocol23Format")
# then: sudo systemctl restart rsyslog

What the listener accepts

UDP: one message per datagram (trailing LF / NUL trimmed). TCP: the framing is decided by the first byte of each connection — a digit means octet counting (LEN SP MSG, up to 64 KB a frame), anything else LF-delimited. Partial frames are buffered across reads; an oversize frame is dropped, counted as rejected, and the stream resyncs at the next LF.

• RFC 5424 (<PRI>1 TS HOST APP PROCID MSGID [SD] MSG, BOM and nil fields handled) and RFC 3164 with the common variants — double-space day, no year, ISO timestamps, a year prefix, a missing HOSTNAME (Cisco) or a missing TAG. A line without a PRI is kept as user.notice (13), so it is never an error.
• Field names follow the warehouse plan: host, program, facility, severity.
• Counting is by receive time — a sender’s clock or a missing year never moves a line into another window.
• Caps: 5 000 messages a second in total (the excess is counted as dropped), 8 KB per message, 256 TCP connections, 128 KB buffered per connection, 5-minute idle timeout, 50 error types per service (the rest → (other)), 10 sample lines per service a check, Max services per source (past it → (other)).
• Memory guard: against the proxy’s RSS budget — at 75 % no new sample lines are kept, at 90 % no new services are added (their lines count into (other)), back to normal below 60 %. Every transition is logged.
• The proxy log prints one [syslog] line a minute while lines arrive: received · dropped · rejected · senders · sources.

Counts, baselines and alerts

Each check drains the source’s own counters: per service, the error lines in the window since the previous check, every line seen (as the denominator), the types and up to 10 redacted sample lines. A service seen in the last 24 hours that logged no errors reads a measured 0. Everything after that — usual count per weekday and hour, provisional baseline, breach rule, new-type warnings, tile, card, banner, Alerts page, Catalogue Logs section — is the Graylog All Services machinery (see the Graylog guide).

• One listener feeds many sources: each has its own service field, filters, severity and threshold, and drains only its own counts.
• A listener that is not bound — now or at any moment in the window (port taken, stopped, rebinding after a port change) — makes the check read no value and syslog listener not bound on :1514 (<reason>), never a row of zeros. A failed bind is retried at most every 5 minutes.
• The first check after a source is (re)scheduled covers a window of about 0 s; counts settle from the second check.
• The card shows the traffic line (received / dropped / rejected / senders in the window) above the service table; View logs opens the kept lines, already redacted on the proxy before they were queued.
• The listener opens only once a syslog source exists on that proxy — an integration with no sources binds nothing (the drawer says so).

Kernel console (netconsole)

When a Linux guest dies, its kernel’s last words — the panic line, the call trace — never reach a log file: the disk is gone before they can be written. In the incident this was built for, vSphere’s “CPU was halted, server rebooted to restore function” was the only trace left. netconsole is the kernel’s own fix: it sends every console (printk) line over UDP to another host the moment it is printed, so a console host collects what the dying guest said — typically rsyslog writing one file per sender, /var/log/netconsole/<sender-ip>.log. The syslog receiver turns those lines into kernel error types, crash episodes and a critical finding on the VM. Needs proxy 1.223.0 or newer (the lines just before a crash ≥ 1.224.0, extended-netconsole storage ≥ 1.225.0); an older proxy still counts the lines as ordinary syslog — no kernel classes, no episodes.

1. Get the lines to the proxy — either wayA · the Server Vitals files collector on the console host tails /var/log/netconsole/*.log and ships new lines with the IP from each file name as the key (see the Server Vitals guide, Files collector (Linux)). B · rsyslog forwarding: drop the file below on the console host — it keeps the per-sender files and forwards a copy of each line, with the sender’s IP as the host name. The integration drawer prints it under rsyslog forwarding rule → Kernel console (netconsole). Both yield the same episode, the same finding and the same card.
2. Add a syslog source for itOn the flow, a syslog source on that integration with Service → Host name (serviceField: hostname — the key becomes the service, so each VM is its own row) and Error type → MSGID (typeField: msgid — the kernel class becomes the error type).
3. Optionally move the VMs to extended netconsoleExtended netconsole adds each message’s kernel log level, so the proxy uses the kernel’s own severity instead of the keyword table. The kernel’s documented parameter syntax is netconsole=[+][src-port]@[src-ip]/[dev],[tgt-port]@<tgt-ip>/[tgt-mac], the leading + turning extended mode on: netconsole=+@<src-ip>/<if>,<port>@<dst-ip>/<mac> on the kernel command line, or the same string as the module’s netconsole= option.

• The rsyslog path differs only in provenance: there is no file name, the key is the sender’s IP (keyBy="sender"), rsyslog stamps the message user.notice (13) and the proxy re-derives the severity, %rawmsg% carries the datagram verbatim (no #012 to decode), and the timestamp is the receive time.

/etc/rsyslog.d/91-netconsole-topologix.conf — on the console host (the drawer fills in your listener port)

# keep one file per sender and forward a copy of every kernel line to the Topologix proxy
template(name="TopologixNetconsole" type="string" string="<%PRI%>1 %TIMESTAMP:::date-rfc3339% %fromhost-ip% kernel - - [tpx@topologix keyBy=\"sender\" fmt=\"netconsole\"] %rawmsg%")
template(name="NetconsoleFile" type="string" string="/var/log/netconsole/%fromhost-ip%.log")
ruleset(name="netconsole") {
  action(type="omfile" dynaFile="NetconsoleFile")   # your existing per-sender file action
  action(type="omfwd" target="<proxy-address>" port="1514" protocol="tcp" TCP_Framing="octet-counted" template="TopologixNetconsole"
         queue.type="LinkedList" queue.size="10000" action.resumeRetryCount="-1")
  stop
}
# the imudp input the senders target — if it already exists, add ruleset="netconsole" to it instead
# module(load="imudp")
input(type="imudp" port="6666" ruleset="netconsole")
# then: sudo systemctl restart rsyslog

Extended netconsole on a VM (kernel command line; the module option takes the same string)

netconsole=+@<src-ip>/<if>,<port>@<dst-ip>/<mac>
# e.g. netconsole=+@172.16.38.155/eth0,6666@172.16.35.169/00:50:56:aa:bb:cc

Kernel lines, classes and crash episodes

What the proxy does with a kernel line, whichever path brought it.

• Which lines are kernel lines. A line with no syslog PRI that looks like printk (an uptime stamp [ 12098203.264695] or an extended header <level>,<seq>,<µs>,<flags>;), a message whose APP-NAME is kernel, or one carrying the structured data [tpx@topologix fmt="netconsole"] (what the files collector and the rsyslog rule send). An rsyslog ISO stamp in front of the line is understood too.
• Classification. The first line of each message is matched against the table above — the same table on the proxy and the server (a byte-for-byte mirror), and the files collector’s awk program is generated from it; the first row wins. An extended line’s own level wins over the table. Severity is only ever raised, never lowered, and the class becomes the MSGID — so a panic counts as an emerg error of type panic in the log baselines, while sysrq (notice) and netconsole-up (info) are kept but are not errors. #012 in a forwarded line becomes a newline and #011 a space (not when a digit follows), so a multi-line printk stays one message.
• Crash episodes. Per service (per VM), a panic, oops, bug, mce or oom line opens an episode that collects every following line — call trace, registers, modules — for up to 60 s, 200 lines or 16 KB (beyond that it is marked truncated). It closes at ---[ end Kernel panic or at the next netconsole up banner. The episode also carries the lines just before the crash: the proxy keeps the last 8 kernel lines of each service that joined no episode (30 s), and an opening line takes the ones from the 30 s before it — that is how sysrq: Trigger a crash, the proof that an operator crashed the box on purpose, ends up in the trace.
• Reboots. A netconsole up banner whose uptime is under 300 s is a reboot: it marks the open (or the just-closed, ≤ 15 min) crash rebooted; with no crash before it, it becomes a one-line reboot episode, so an unexplained restart stays visible. A banner at a high uptime — netconsole re-enabled on a running machine — is not a reboot.
• Extended lines are stored as [uptime] body — [12098203.264695] Kernel panic … — with the level and sequence number kept as fields. The raw header is never stored: its 14–20-digit microsecond stamp looked like a card number to the redaction rules.
• An episode is emitted once, redacted on the proxy like every sample line, with the next check of the source (≤ 100 a source a check). What it becomes on the server — the finding on the VM, the incident, the AI verdict, the Kernel trace drawer — is described in the Alerting guide (Kernel crashes).

Good to know

• Proxy 1.216.0 or newer; the server sends the listener settings only to such a proxy. The proxy’s Activity view shows the listener’s received / dropped / rejected / senders counters.
• The Test button asks the proxy whether the listener is bound. A syslog source never runs on the central server — its error reads “Syslog sources run on the proxy that receives them”.
• Still open on the roadmap: substring rules (e.g. “count lines containing link down”) and a per-device timeline.