Network flows (NetFlow / IPFIX / sFlow) in Topologix

Infrastructure & Networking

Point your network gear’s flow export at a Topologix proxy and every conversation it sees becomes something you can use: who talks to whom on which port (the Flows tab: exporters, top talkers, pair queries, CSV), dependency edges between services that no trace ever reported (dotted in the Topology, confidence 70 in Blast radius), and a Network peers list on every service page. The proxy decodes NetFlow v5 / v9, IPFIX and sFlow v5 itself, keeps raw per-minute flows on its own disk (and, if you designate one, in a ClickHouse warehouse), and ships only 5-minute resolved service pairs to the server — raw flows never reach the Topologix database.

Requirements

• A proxy agent that the exporters can reach over UDP (usually one per site / management VLAN). The listener is off until you switch it on; the proxy must have an owner (flows belong to a tenant — the global dev proxy declines them by design).
• Proxy 1.196.0 or newer (1.199.0 for the ClickHouse warehouse). Node.js 22 LTS recommended on the proxy host: node:sqlite (Node ≥ 22.13) gives the 7-day SQLite flow store; Node 20 hosts fall back to segment files (queries ≤ 24 h, ≤ 7 d with an exporter + an IP). install.sh installs Node 22 on new hosts; the Proxies row says flows: listening · sqlite or · files.
• Inbound UDP 2055 (NetFlow v5 / v9), 4739 (IPFIX) and 6343 (sFlow) open on the proxy host (any port accepts any of the protocols — the version is sniffed per datagram).
• Devices that export flows: Cisco NX-OS / IOS-XE / IOS-XR, VMware vSphere Distributed Switch, FortiGate, Palo Alto, Check Point, Aruba CX / AOS-S, Juniper — snippets below. Linking an exporter to its SNMP integration names the ifIn / ifOut interfaces on the Flows tab.
• Optional: a ClickHouse integration (the existing integration type) as the flow warehouse — 30 days of the same per-minute rows, queries over up to 30 days, and the inferred databases and services of section 7 (proxy ≥ 1.228.0, 1.230.0 recommended).

1 · Enable the listener on a proxy

1. Open Settings → Proxies and edit the proxyTick Listen for NetFlow / IPFIX / sFlow. It applies at once (no Save): the proxy picks it up on its next config poll (≤ 15 s), opens UDP 2055 / 4739 / 6343 and logs [Netflow] listener up. Only the proxy’s owner, an admin, or an admin of the owner’s organisation can switch it; the change is audited (proxy.netflow.set).
2. Optionally pick a flow warehouseWith the listener on, the same form shows Flow warehouse (ClickHouse) — the ClickHouse integrations you can see. Choose one to stream the per-minute flows into topologix_flows (see “The ClickHouse warehouse” below); None keeps them on the proxy only.
3. Open the host firewallAllow UDP 2055, 4739 and 6343 from your exporters to the proxy host (Linux: ufw allow proto udp from 10.0.0.0/8 to any port 2055,4739,6343; Windows: an inbound UDP rule for the proxy’s node.exe).
4. Point the devices at the proxyUse the proxy’s address as the collector and one of the three ports (per-vendor snippets below). Within one ship interval (5 minutes) the exporters appear on Infrastructure → Flows as unexpected — auto-registered; name them, link their SNMP integration, mark them expected.
5. Check the row chipSettings → Proxies shows flows: listening · sqlite (or · files on Node 20) and, with a warehouse, · warehouse ✓ / · warehouse lag 45 s / · warehouse ✕. The queue badge shows the ship queue and anything dropped.

2 · Exporter configuration by vendor

Replace 10.0.0.50 with the proxy’s address and the source addresses with each device’s management / loopback address. Export both directions where the platform lets you (ingress on every interface is enough on a router; a firewall’s own flow table is already bidirectional). Sampling is fine — bytes are scaled by the rate the device reports and shown as an estimate.

Cisco Nexus (NX-OS) — Flexible NetFlow v9

NX-OS needs the feature, a record, an exporter and a monitor applied per interface (and a sampler if you want 1-in-N). The exporter usually leaves through the management VRF.

NX-OS

feature netflow
flow record TPX-RECORD
  match ipv4 source address
  match ipv4 destination address
  match ip protocol
  match transport source-port
  match transport destination-port
  collect counter bytes
  collect counter packets
  collect transport tcp flags
flow exporter TPX-EXPORT
  destination 10.0.0.50 use-vrf management
  transport udp 2055
  source mgmt0
  version 9
sampler TPX-1IN1000
  mode 1 out-of 1000
flow monitor TPX-MONITOR
  record TPX-RECORD
  exporter TPX-EXPORT
interface Ethernet1/1
  ip flow monitor TPX-MONITOR input sampler TPX-1IN1000

Cisco ASR 1000 / ISR (IOS-XE) and ASR 9000 (IOS-XR)

IOS-XE uses Flexible NetFlow like NX-OS; send the sampler table (option sampler-table) so the proxy learns the rate from options data. For IPFIX use export-protocol ipfix and port 4739. IOS-XR uses exporter / monitor / sampler maps.

IOS-XE

flow record TPX-RECORD
 match ipv4 protocol
 match ipv4 source address
 match ipv4 destination address
 match transport source-port
 match transport destination-port
 collect interface input
 collect interface output
 collect transport tcp flags
 collect counter bytes long
 collect counter packets long
flow exporter TPX-EXPORT
 destination 10.0.0.50
 source Loopback0
 transport udp 2055
 export-protocol netflow-v9
 option sampler-table
flow monitor TPX-MONITOR
 exporter TPX-EXPORT
 record TPX-RECORD
 cache timeout active 60
sampler TPX-1IN1000
 mode random 1 out-of 1000
interface GigabitEthernet0/0/0
 ip flow monitor TPX-MONITOR sampler TPX-1IN1000 input

IOS-XR (ASR 9000)

flow exporter-map TPX-EXPORT
 version v9
  options sampler-table timeout 60
 !
 transport udp 2055
 source Loopback0
 destination 10.0.0.50
flow monitor-map TPX-MONITOR
 record ipv4
 exporter TPX-EXPORT
 cache timeout active 60
sampler-map TPX-1IN1000
 random 1 out-of 1000
interface TenGigE0/0/0/0
 flow ipv4 monitor TPX-MONITOR sampler TPX-1IN1000 ingress

VMware vSphere Distributed Switch — IPFIX

The vDS exports IPFIX for the VM traffic it switches (east-west traffic that never touches a physical router). Two places: the switch-level collector, then NetFlow enabled per distributed port group.

1. Switch settingsvSphere Client → Networking → the distributed switch → Configure → Settings → NetFlow → Edit: Collector IP address = the proxy, Collector port = 4739, Observation domain ID = a number unique per vDS (it becomes the exporter’s domain: <switch IP>/<domain>), Switch IP address = an address that identifies this vDS in the flows (it is what the Flows tab shows), active flow export timeout 60 s, idle 15 s, sampling rate 0 (every packet) or N, Process internal flows only off.
2. Port groupsEach distributed port group → Configure → Policies → Edit → Monitoring → NetFlow: Enabled (or per uplink / per port). Only enabled port groups export.

• One exporter for every host — and two copies of each VM↔VM flow. With a Switch IP address set, all hosts of the vDS export under that one address, so the Flows tab shows a single exporter (for example 172.16.90.90/0). A conversation between VMs on two different hosts is exported by the sending host (VM port → uplink) and the receiving host (uplink → VM port): two records, identical except for ifIn / ifOut. Added up, VM↔VM traffic would count twice — a VM vCenter shows at 120 MB/s would show ≈ 240 MB/s.
• The proxy (1.233.0+) counts such a flow once: per conversation and minute it keeps each (ifIn, ifOut) pair apart and reports the largest pair instead of the sum. It turns this on by itself when it sees it — see Two-copy dedupe in section 4 — and the exporter row then shows two-copy dedupe · saved N %. Without a Switch IP each host is its own exporter and the usual MAX across exporters already keeps one copy.

FortiGate (FortiOS) — NetFlow v9

A global collector, then netflow-sampler on each interface (both = ingress and egress). FortiOS 7.2+ nests collectors under config collectors; older builds set collector-ip directly on system netflow. With VDOMs, use config system vdom-netflow inside the VDOM. A busy FortiGate sets up thousands of sessions a second — the dev FortiGate-901G-01 runs ≈ 5 000 setups/s (the Setup rate column of the Firewalls table on Infrastructure → Network shows yours): raise the proxy’s aggregation caps for such a site (see “Capacity” below).

FortiOS 7.2+

config system netflow
    set active-flow-timeout 60
    set inactive-flow-timeout 15
    set template-tx-timeout 60
    config collectors
        edit 1
            set collector-ip "10.0.0.50"
            set collector-port 2055
            set source-ip "10.0.0.1"
        next
    end
end
config system interface
    edit "port1"
        set netflow-sampler both
    next
end

Palo Alto Networks (PAN-OS) — NetFlow v9

A NetFlow server profile, attached to each interface whose traffic should be exported. PAN-OS exports unsampled NetFlow v9.

1. Server profileDevice → Server Profiles → NetFlow → Add: a name, Template Refresh Rate (30 minutes / 20 packets is fine), Active Timeout 1 minute, leave PAN-OS Field Types off (standard fields only), then under Servers add the proxy with port 2055.
2. InterfacesNetwork → Interfaces → Ethernet → the interface → NetFlow Profile = the profile (also on sub-interfaces / aggregate groups). Commit.

Check Point (Gaia) — NetFlow export

Gaia Portal → System Management → NetFlow Export → Add: the collector IP, port 2055 (or 4739 with IPFIX), the export format, the source address. Or in clish:

Gaia clish

add netflow collector ip 10.0.0.50 port 2055 srcaddr 10.0.0.1 export-format Netflow_V9
save config

Aruba CX and AOS-S (ProCurve) — sFlow

Aruba switches export sFlow: sampled packet headers, decoded on the proxy (VLAN tags, IPv4 fragments and IPv6 extension headers handled); the sampling rate rides in every sample, so bytes are scaled correctly. The agent address becomes the exporter.

AOS-CX

sflow
sflow collector 10.0.0.50 port 6343 vrf mgmt
sflow agent-ip 10.0.0.2
sflow sampling 4096
sflow polling 30
interface 1/1/1
    sflow

AOS-S

sflow 1 destination 10.0.0.50 6343
sflow 1 sampling 1-48 4096
sflow 1 polling 1-48 30

Juniper — sFlow (EX / QFX) or inline J-Flow IPFIX (MX)

Switches: sFlow. Routers: inline J-Flow exports IPFIX from the PFE — sampling instance on the FPC, an IPFIX template, the flow server, sampling on the interfaces.

sFlow (EX / QFX)

set protocols sflow collector 10.0.0.50 udp-port 6343
set protocols sflow agent-id 10.0.0.3
set protocols sflow source-ip 10.0.0.3
set protocols sflow sample-rate ingress 2048
set protocols sflow polling-interval 20
set protocols sflow interfaces ge-0/0/0

Inline J-Flow IPFIX (MX)

set chassis fpc 0 sampling-instance TPX
set services flow-monitoring version-ipfix template TPX-V4 ipv4-template
set services flow-monitoring version-ipfix template TPX-V4 flow-active-timeout 60
set forwarding-options sampling instance TPX input rate 1000
set forwarding-options sampling instance TPX family inet output flow-server 10.0.0.50 port 4739
set forwarding-options sampling instance TPX family inet output flow-server 10.0.0.50 version-ipfix template TPX-V4
set forwarding-options sampling instance TPX family inet output inline-jflow source-address 10.0.0.4
set interfaces xe-0/0/0 unit 0 family inet sampling input

3 · Ports, allow-list and local settings

Ports and who may send

The defaults are 2055 (NetFlow), 4739 (IPFIX), 6343 (sFlow), bound dual-stack (::; IPv4 arrives IPv4-mapped). Any port decodes any protocol. A port that is taken is logged and the other two still listen. Without an allow-list every sender is accepted; with one, datagrams from anyone else are dropped and counted (droppedExporter). An entry that does not parse is ignored and logged — a list of typos only admits nobody rather than everybody. Caps per proxy: 10 000 datagrams/s per exporter, 50 000 in total, 256 exporters (proxy 1.232.0; before: 5 000 / 10 000 / 64, fixed); beyond them datagrams are dropped and counted (droppedRate / droppedExporter, with the caps in force as ppsCaps in the admin status). The datagram caps are server env NETFLOW_PPS_PER_EXPORTER (1 000 – 100 000) / NETFLOW_PPS_TOTAL (5 000 – 500 000), or per proxy in Settings → Proxies → Proxy settings → Flows (datagram caps); an in-cluster flow agent sends one flow per datagram, so a busy cluster beside the vDS may need a higher total.

• Everything else is set per proxy in Settings → Proxies → Proxy settings → Flows (proxy ≥ 1.235.0): ports (0 = off), bind address, exporter allow-list, server ports, ship size / interval, local store size / retention, per-minute cut, aggregation and datagram caps, NetFlow memory budget and warehouse queue. Each row shows the effective value and its source; bounds and the full list are in Proxy requirements and sizing.
• Upgrading from an older proxy: on its first contact after 1.235.0 the netflow {} keys its config.json still carries are imported once as that proxy's settings (nothing changes that day); the drawer then lists the leftover keys — remove them from the file.
• Server ports marks extra ports as the SERVER side (e.g. an application on 7318) for the client / server decision. Known exporters (with a per-exporter sampling rate for a device that does not announce it — the rows otherwise say sampling unknown and bytes are an under-estimate) can only be removed in the drawer; imported entries keep working.
• netflow.dataDir stays in config.json. A local netflow.clickhouse {} block is still honoured and wins over the warehouse designated in the UI, but the drawer flags it — prefer a ClickHouse integration.

proxy config.json — only what stays local (proxy ≥ 1.235.0)

{
  "serverUrl": "https://topologix.example.com",
  "authKey": "…",
  "netflow": { "dataDir": "/var/lib/topologix/netflow" }
}

4 · What lands where

Raw flows stay near the network

The proxy folds records into 60-second buckets per (protocol, client, server, server port) and decides which side is the server (a TCP SYN-only record, a port under 1024 or a well-known high port on exactly one side, an ephemeral port on the other). What is stored:

• A payload re-sent after a timeout, or a partial bucket re-shipped with fuller totals, replaces its rows — it never doubles bytes.
• Two exporters that see the same conversation (a switch and the firewall behind it) are never added up: every read takes the busiest exporter’s figures per pair (MAX across exporters, never SUM) — on the proxy, in ClickHouse and in the Topologix database alike.
• Two copies under one exporter (a vDS with a Switch IP: the sending and the receiving host) are folded on the proxy before anything is stored — the two-copy dedupe below — so the local store, the warehouse and the resolved pairs all see one copy.

The flows as a map

The Flows tab’s pair query and the Flows section of Infrastructure → Forensics carry a Table | Map toggle (remembered per browser; a phone starts on the table). Map draws the same rows in the Topology’s visual language: the queried address — or the entity’s addresses — in the middle, its clients to the left and its servers to the right (a peer that is both appears once, on the side where it moved more bytes). Each resolved end is its Topology card — service, pod, VM, Kubernetes node, database, external — with its name and address; an unresolved address is an external card with This is…, and 4 or more unresolved addresses of one /24 fold into a subnet node (162.125.69.0/24 · 6 hosts · 1.2 MB) until you click it. Edges run client → server, labelled port · bytes · flows (several ports between the same pair share one edge), 1–6 px wide by bytes on a log scale, dashed through a sampled exporter, and red-tinted when one pair carries more than 50 % of the address’s bytes; hover an edge for the rows behind it (fwd / rev, exporter, if in → out, first / last seen). A service seen only in the stored pairs (Forensics → Peers) is drawn the same way from GET /api/netflow/pairs.

• Up to 40 peers a side are drawn, the busiest nearest the middle; the rest fold into +N more peers · bytes — 40 more per click.
• Click a card for Focus here (that address’s flows over the same window — one more hop, up to 3, each bounded like the first query at 1,000 rows), Open in Forensics, Show in table (back to the table at its busiest row) and, for a bare address, This is actually…. The breadcrumb above the map (10.83.233.13 › 162.125.69.19) walks back; Esc closes the card.
• Under the map the same footer as under the table says which proxies answered, from which source and store — and when no proxy listens, or none answered, the map says that instead of “no flows”.

Two-copy dedupe (one exporter, two hosts)

Within one exporter the proxy normally adds every record of a conversation — right for a router or a firewall, which sees each packet once. A vSphere vDS with a Switch IP is different: every host exports under the same address, and a flow between VMs on two hosts arrives from both, differing only in the interface pair. From proxy 1.233.0 each exporter has a dedupe mode, set in its drawer on the Flows tab (Edit → Two-copy dedupe):

• What you see: the exporter row on the Flows tab shows two-copy dedupe · saved 48 % while it applies (the bytes dropped as second copies in the last minute); hover for duplicates detected: 63 % of keys — the share that made Auto turn it on. Under the 20 % bar it shows duplicates 10 % · not deduped. Admin → System → Background workers (the network-flows block) shows the same chip per proxy, in the Exporters column, and the proxy reports dedupe per exporter (netflow-exporters) and summed in its status.
• Flows are counted per record within each pair, so a flow seen by both hosts counts once. Interface columns (if in → out) show the winning pair, oriented client → server.
• NIC teaming: a host that spreads one conversation over two uplinks shows two pairs on its side. The largest pair still wins, so such a conversation is under-counted (by the smaller uplink’s share), never doubled.
• Memory: at most 4 pairs are kept per conversation (a 5th pair is added to the largest); an exporter in Auto or On costs 12 B more per conversation in the open minute, plus 36 B for each conversation that arrives on a second pair (108 B from a third pair on). aggBytes in the status includes it. Measured (npm run perf:netflow scenario F: 60 000 records/s, 60 % of the flows sent twice, 300 000 keys): peak RSS 320 MB against 314 MB without the dedupe, ArrayBuffers 35 MB against 24 MB, ≈ 5 points more of one core.

5 · How an address becomes a service

The resolution ladder

Both ends of every pair are named from what Topologix already knows about your infrastructure (the VMware / Kubernetes / database inventory). First hit wins, and ambiguity is never guessed — it falls through to the next step:

• “This is actually…” on an ext:<ip> chip (Flows tab, Topology, Catalogue gaps) aliases the address to a service — candidates from the placement are offered. Aliases apply in the dependency sweep, so the next sweep folds the edge into the service and flags the old one stale; the Flows tab names the address immediately.
• A VM / node end is not an external: it raises no “Externals with a private address” catalogue gap, a change on the VM counts the services that talk to it in flows as depending on it, and talking to it never silences it. Name what listens on it by mapping a service to the VM — once exactly one service is placed there, step 5 names the service instead.
• Pairs between two public addresses (transit) and self-loops are dropped and counted.

How addresses are named

An address the ladder leaves at step 8 is still an ext:<ip> end — its key never changes, so no edge is redrawn — but it is labelled with what your other inventories say it is. The Flows table chip, the flow map card and the Topology node then show the name and a small source chip (Zabbix, catalogue, reverse DNS…) instead of External 10.80.3.14, with an icon for the kind (server, VM, ESXi host, device, BMC, storage, DB server). Where the label comes from, best first:

• Several sources at the best rank: the first is shown, the others ride along as candidates (+N; the card’s popover lists them). The same name from two sources counts once.
• Every source is read in the owner’s scope only — another organisation’s Zabbix or catalogue never names your addresses. The sources are re-read in the background with the IP index (netflowIndexTtlSeconds), never while a flow is being stored.
• Database servers nobody monitors: when the label’s Zabbix host carries a database template (PostgreSQL, MSSQL, MySQL, Oracle, MongoDB, Redis) or its name looks like one (db, pg, sql, mysql, oracle, mongo, redis), and the flow’s server port is that engine’s well-known port, the end reads PostgreSQL · prod-core-db-backup (not monitored), and its popover offers Monitor this database — the integration form prefilled with the type, address and port.
• Monitored databases configured by host name: step 1 also matches an instance whose host is a DNS name. A name of an integration polled directly is resolved by the server (kept 1 h, failures 10 min); a name of an integration polled through a proxy is resolved by the flow proxies on-prem, like trace externals (step 6). A flow to that address and port then resolves to the instance, exactly as if the address were configured.
• A flow end that resolves to a monitored instance lists the instance’s databases (and their schema counts, from the Databases catalogue — up to 20 + more) in the flow map card’s popover; you need access to the instance’s integration.
• Turn reverse DNS off with Admin → Background workers → Network flows: name addresses by reverse DNS (0). Admin → Background workers → Network flows shows how many addresses each owner’s index names and the reverse-DNS cache (named, without a name, pending, the last batch).
• A label is not a service: the chip keeps This is actually…, and an alias you set wins over any label.

Confidence: 100 / 90 / 70

Blast radius ranks every edge: 100 when two kinds of evidence agree (a trace AND a flow, or two trace sources), 90 for a single trace source, 70 for flows alone, 50 for a stale edge. A flow proves bytes moved between two addresses; a trace proves one service called another — so a flow-only edge is shown dotted in the Topology, with calls: not measured (never 0) and the bytes instead. Edges need at least netflowEdgeMinFlows (3) flows in the lookback.

Sampled bytes are an estimate

A sampled exporter reports 1 in N packets; the proxy scales bytes and packets by N — from the record, the options template, the sFlow sample, or your samplingRate override. Such answers carry an estimate tag (and a ~ on edge labels). With no rate known at all, bytes are unscaled and marked sampling unknown — an under-estimate. Flow counts are never scaled.

6 · The ClickHouse warehouse

Longer history and bigger queries

Designate a ClickHouse integration on the proxy (Settings → Proxies → edit → Flow warehouse, or PUT /api/proxy-nodes/:id/netflow {"warehouseIntegrationId": "…"} — it must be a ClickHouse integration you can see; audited). The server hands the proxy its URL and credential with the config (a sealed / unreadable credential is withheld, never sent blank). From then on every minute close streams every aggregated row into <database>.topologix_flows (database: the integration’s, else default) — full fidelity from proxy 1.227.0: the local store keeps only its per-minute top localTopNPerMinute, the warehouse gets all of it (an older proxy sends the warehouse the same top-N cut; Admin → Background workers → Network flows marks each proxy full fidelity or top-N only).

• Never in the listener’s way: rows are encoded once and queued in memory — consistency first, the queue is sized to ride out an outage: by default ten times aggMaxKeysTotal (≈ 10 minutes of a full aggregation table), at least 300 000 and at most 10 000 000 rows — 3 000 000 at the default caps, 10 000 000 for a site at aggMaxKeysTotal 1 000 000; set Warehouse queue (rows) on the proxy (100 000 – 20 000 000) to choose it yourself; past it the oldest are dropped and counted; inserts go out every 10 s and at each minute close, ≤ 50 000 rows / 8 MiB per request (JSONCompactEachRow). ClickHouse down → the same batch is retried with a 1 → 60 s backoff; a rejected batch (4xx) is dropped and counted.
• Queries (the Flows tab, POST /api/netflow/query) are answered from ClickHouse while it is healthy — an insert succeeded in the last 5 minutes, or a SELECT 1 probe (cached 60 s) answers — with the same semantics as the local store (MAX across exporters) and windows up to 30 days. Otherwise the proxy answers from its local store and says why; the Flows tab header shows warehouse · lag N s or local store, and a fallback reason per proxy. Every query is parameterised ({name:Type} placeholders) and filtered to the asking proxy’s own rows (proxy_id), so a shared ClickHouse never mixes tenants.
• Health: the Proxies chip (warehouse ✓ / lag N s / ✕) comes from the proxy’s own header every 15 s; Admin → Background workers → Network flows shows rows inserted, waiting, dropped, failures, the DDL state and the last error per proxy, plus two warning chips: warehouse queue dropped N rows (cumulative since the proxy started; hover for the first and latest drop) and heap too small — needs X MB with a Set heap to X MB button (Proxy settings → Runtime, then Restart proxy).
• Heap fit: whenever the bound is applied the proxy checks it against its V8 heap limit (rows × 136 B plus aggMaxKeysTotal × 300 B within 75 % of the limit). A bound that does not fit is not a crash: the proxy buffers what fits, logs one warning with the --max-old-space-size that would fit, and reports it. Set it in Settings → Proxies → Proxy settings → Runtime → heap size and press Restart proxy (proxy ≥ 1.236.0): the proxy relaunches itself with that --max-old-space-size, no service-file edits. The commented-out NODE_OPTIONS line install.sh / install.ps1 leave is only for an operator who must pin it at the service level; an operator-set NODE_OPTIONS heap is reported in the drawer and should otherwise be removed.
• A changed target (URL, database, user, password) swaps the sink live — the waiting rows carry over. Removing the warehouse keeps the local store as the only copy.

No ClickHouse yet? Use the on-prem kit

Topologix ships a single-node setup as a download: topologix-clickhouse-flows-kit.zip (also inside every proxy release zip, under kits/, and in Admin → Versions). It holds a Docker Compose file for ClickHouse 26.8 LTS (HTTP 8123 bound to the address you choose, healthcheck, data in ./data), a topologix_flows user limited to the flows database (4 GB and 120 s per query), and an init.sql that pre-creates flows.topologix_flows exactly as the proxy would, with a 30-day TTL. Unzip it into a directory on a Linux host with Docker, set CH_BIND and CH_PASSWORD in .env, run docker compose up -d, then add the ClickHouse integration (via the proxy, base URL http://<host>:8123, database flows, user topologix_flows) and pick it as the proxy’s Flow warehouse.

• Sizing: rows per day ≈ aggregated keys per minute × 1 440, about 30–40 B each on disk. 50 000 keys/min ≈ 2.5 GB/day ≈ 80 GB for 30 days; 300 000 keys/min ≈ 15 GB/day ≈ 450 GB. The kit README recommends 4 vCPU, 16 GB RAM and SSD; the measured need is lower (ClickHouse stayed under 2 GB RAM at ≈ 390 000 keys/min). Provision twice the 30-day estimate.
• Verify: SELECT count(), min(minute), max(minute) FROM flows.topologix_flows grows within minutes, and the Proxies chip shows warehouse ✓. Only the proxy host needs to reach port 8123. Backups are optional (the data ages out with the retention); when the history matters, the kit’s backup.sh runs ClickHouse BACKUP daily to S3 or a local disk.
• The kit’s README.md has the full steps, the disk-usage query and the firewall notes.

The table and its privileges

The proxy runs CREATE TABLE IF NOT EXISTS once per process. If the user may not create tables, the proxy logs it once, reports DDL denied and inserts anyway — so an administrator can create the table and grant only INSERT + SELECT. After its first insert the proxy reads the table’s TTL from system.tables and, when it differs from the configured retention, runs one ALTER TABLE … MODIFY TTL minute + INTERVAL N DAY (without the ALTER MODIFY TTL grant it logs the statement once for an administrator and leaves the table alone; a TTL of another shape is never touched). On a ClickHouse too old for the IPv6 type the proxy retries once with String address columns.

Pre-create it (retention 30 days) and grant what the kit grants

CREATE TABLE IF NOT EXISTS flows.topologix_flows (
  minute DateTime('UTC'),
  proxy_id LowCardinality(String),
  exporter_key LowCardinality(String),
  proto UInt8,
  client_ip IPv6,
  server_ip IPv6,
  server_port UInt16,
  if_in UInt32,
  if_out UInt32,
  bytes_fwd Float64,
  bytes_rev Float64,
  packets Float64,
  flows UInt32,
  flags UInt8
) ENGINE = MergeTree
PARTITION BY toYYYYMMDD(minute)
ORDER BY (exporter_key, minute, server_ip, server_port)
TTL minute + INTERVAL 30 DAY
SETTINGS ttl_only_drop_parts = 1;

CREATE USER topologix_flows IDENTIFIED WITH sha256_password BY '…';
-- what the kit (deploy/clickhouse-flows/users.d/topologix.xml) grants: the proxy may create the
-- table and keep its TTL at the Admin retention; SELECT, INSERT alone also works (the proxy then
-- logs the CREATE / ALTER once for an administrator)
GRANT SELECT, INSERT, CREATE TABLE, ALTER MODIFY TTL ON flows.* TO topologix_flows;

7 · Inferred databases and services

What the flows say is running

With a ClickHouse warehouse holding every aggregated flow, Topologix looks for the private addresses that answer on a database, queue or API port — the Postgres a team forgot to mention, the Kafka broker nobody added. Once an hour, per organisation (each organisation at its own offset in the hour, one warehouse query at a time), the server asks each of its warehouse proxies which private (address, port, protocol) answered over the last 24 hours — clients, bytes each way, flows, active minutes, days and exporters, MAX across exporters as everywhere else — scores each endpoint and keeps the result as an inferred catalogue object of the organisation (at most 500 new or refreshed rows per run, 2 000 kept per organisation).

• Skipped: Kubernetes ClusterIPs, pod addresses and nodePorts (service plumbing, not a database), public addresses, and anything a monitored instance already covers.
• Each candidate is named from what Topologix knows (the VM, node or host the address belongs to) plus a reverse-DNS lookup (≤ 200 per organisation per run, cached 24 hours).
• Tuning (Admin → Background workers → Network flows): inferredEndpointsIntervalSeconds (3 600; 600 – 86 400), inferredEndpointsLookbackHours (24; 1 – 168), inferredEndpointsPerOrgCap (2 000; 500 on SQLite). Re-scan now: an organisation owner or admin can call POST /api/netflow/inferred/run (once per 5 minutes); a super admin can run every organisation with POST /api/admin/netflow/inferred/run.

How confident it is

A score from 0 to 100 is built from plain evidence; every row lists the reasons it scored what it did.

• Low below 55, medium 55 – 79, high 80 and above — and high only with a name match: traffic alone never reaches high.
• Database and queue kinds appear in Catalogue → Gaps under “Seen in network flows, not monitored”; the service kind (API ports, unknown ports) shows only on the Databases page section and in the catalogue list, never as a gap.

Monitor this · Not a database

Each gap row shows the confidence, an evidence line (clients, bytes, the busiest clients) and two actions. Monitor this opens the integration form as a draft prefilled from the flows — engine, host and port (MongoDB and Kafka have no integration type yet, so you pick one). Not a database (or Ignore for a queue) dismisses the row; the choice is remembered across runs and can be restored from the Databases page (tabs inferred · monitored · dismissed). Dismissing and restoring take an organisation owner or admin and are audited (catalog.inferred.dismiss / restore).

• Promotion: once a monitored database instance answers on the same address and port (the first step of the resolution ladder above), the next run marks the row monitored and links it to that instance; the maps then resolve the node to the real instance. Monitored rows stay 30 days as history (“became real on …”); inferred rows unseen for 30 days are pruned.
• On the maps: the busiest clients of each inferred endpoint feed the same single network-flows dependency source (confidence stays 70), and the endpoint is drawn dashed with an “inferred · medium” badge on the Dependency and Topology maps. The catalogue list marks them with a dashed inferred badge and a filter to hide them.
• Never licensed: inferred rows never count toward infrastructure assets, APM services or catalogue items — only what you monitor does.

Requirements and the proxy upgrade

Inferred databases need a ClickHouse flow warehouse on the proxy (section 6 — no ClickHouse yet? use the on-prem kit, topologix-clickhouse-flows-kit.zip, described there) and a proxy that can answer the services query: 1.228.0 at least, 1.230.0 recommended (full-fidelity warehouse from 1.227.0, the outage-safe queue from 1.229.0, and from 1.230.0 a full queue no longer throttles syslog either). A proxy below 1.228.0 is skipped and reported per proxy in Admin → Background workers → Network flows; the local store alone is never scanned (it keeps only the top rows per minute).

• Upgrading: Settings → Proxies → the row’s Update (or re-run install.sh / install.ps1 on the host). Nothing else changes: the warehouse table, the listener settings and the local store carry over, and the first hourly run after the upgrade fills the Gaps section. For a Croco-class site (aggMaxKeysTotal 1 000 000) set the heap and RAM from the sizing table in section 8 when you upgrade.

8 · Capacity and retention

What one proxy handles

Measured with npm run perf:netflow (the listener in-process, 2 000 clients × 200 servers): ≤ 6 % of one core at 7 700 records/s (≈ 4.5 % after the phase-4a rework), ≈ 14 % at 60 000 records/s, event-loop p99 ≈ 13 ms, zero UDP loss. Memory is the limit, not CPU: an aggregation key (one conversation in the open minute) costs ≈ 57–90 B, so the default caps — aggMaxKeysPerExporter 100 000, aggMaxKeysTotal 300 000, across everything resident — hold about 27 MB; records past a cap fold into the exporter’s other row (bytes kept, pair lost, counted as capOverflow).

• RSS budget — the proxy-wide rssBudgetMB (2048 MB by default since proxy 1.204.0; per proxy in Settings → Proxies → Proxy settings → Runtime, or PROXY_RSS_BUDGET_MB on the server; see Proxy requirements and sizing): above 75 % the caps halve, above 90 % no new keys are taken (existing ones keep counting), below 60 % they are restored — one log line per change. The Flows section’s NetFlow memory budget still overrides it for the NetFlow guard alone, but is deprecated.
• Sizing a busy firewall: new keys per minute ≈ new sessions per minute. The dev FortiGate-901G-01 sets up ≈ 5 000 sessions/s; for a site like that set aggMaxKeysTotal ≈ 1 000 000 (and the per-exporter cap ≈ 500 000) — ≈ 90 MB — in Settings → Proxies → Proxy settings → Flows, or for every proxy with the server env NETFLOW_AGG_MAX_KEYS_TOTAL / NETFLOW_AGG_MAX_KEYS_PER_EXPORTER (and PROXY_RSS_BUDGET_MB for the budget; NETFLOW_RSS_BUDGET_MB still works for NetFlow alone).
• The warehouse queue only fills while ClickHouse is slow or unreachable: ≈ 136 B of heap per waiting row, so ≈ 410 MB at the default 3 000 000-row bound, ≈ 1.36 GB at 10 000 000 (an aggMaxKeysTotal of 1 000 000 or more) and ≈ 2.7 GB at the 20 000 000-row maximum — leave that headroom in the V8 heap (Runtime → heap size + Restart proxy, which relaunches with --max-old-space-size; the admin status says when it is too small) and in the host’s RAM. RAM is the cheaper side of the trade: a full queue drops the oldest rows for good.
• The RSS guard never counts the warehouse queue (proxy 1.229.0+): rssBudgetMB is the budget for everything except the outage queue, and the guard compares RSS minus the queue’s share (waiting rows × 136 B × 2.1 — measured: 2 000 000 rows ≈ 260 MB of heap but +533 MB of RSS) with it, holding the highest queue depth of the last 10 minutes because RSS falls only lazily after a drain. A ClickHouse outage that fills the queue therefore never halves aggregation; aggregation that outgrows the budget on its own still does. The whole-process budget with the queue full is rssBudgetMB + bound × 136 B × 2.1 — ≈ 2 866 MB for an untuned site (2048 MB, 3 000 000 rows), ≈ 4 772 MB for a 10 000 000-row bound — reported per proxy as warehouseQueue.effectiveRssBudgetMB; size the host’s RAM for that number. From proxy 1.230.0 the syslog listener’s guard and the CPU-profile refusal subtract the same queue share, so a full queue throttles nothing; the Activity panel keeps showing the total RSS, with the queue’s share noted under the bar.

Measured sizing for a Croco-class site

Measured with proxy 1.230.0 and a ClickHouse warehouse, using a synthetic stable population of conversations and a 3–4-minute ClickHouse outage. The full-queue rows are extrapolated linearly from the measured cost per queued row (≈ 230 B of RSS).

• During the outages the guard never tripped, the caps stayed whole and nothing was dropped. The queue drained in under 35 s after ClickHouse came back, and every minute arrived: the ClickHouse row count equalled the proxy’s inserted count.
• aggMaxKeysTotal covers the open minute and the 5-minute ship bucket, so a cap of 1 000 000 holds about 400 000 – 500 000 distinct keys a minute before new conversations fold into other. Set it to about twice your keys per minute.
• The ClickHouse kit’s README.md (topologix-clickhouse-flows-kit.zip) has the same table with the method.

The local store: SQLite or segment files

On Node ≥ 22.13 the proxy uses node:sqlite (flows.db, WAL, queries on a worker thread, windows up to 7 days; ≈ 300 MB/day at 3 exporters × 1 000 rows/min, 4 GB cap). On Node 20 it writes gzipped 80-byte rows in 5-minute segment files (≈ 85 MB/day, 2 GB cap) and queries are limited to 24 hours (7 days with an exporter and an IP). Upgrading a host to Node 22 switches it to SQLite on the next start (the old segment files are removed). A free-space guard pauses writes below 64 MB free.

Retention knobs

Everything that bounds disk and memory, and where to change it:

9 · Pod-level flows (Kubernetes)

A vDS or a switch sees a Kubernetes cluster’s traffic from the outside. Whether it can name the pod depends on how the CNI forwards pod traffic. The Flows tab tells you which case you are in.

Two Calico modes

Calico, like most CNIs, runs in one of two modes, and they look very different on the wire:

• Ask the cluster: kubectl get ippools.crd.projectcalico.org -o yaml | grep -E 'cidr|natOutgoing|ipipMode|vxlanMode'. ipipMode: Always / CrossSubnet, or a vxlanMode other than Never, means overlay; natOutgoing: true means pod → outside leaves as the node IP. To find the CNI at all: kubectl get pods -n kube-system -o wide | grep -iE 'calico|flannel|cilium|antrea|weave'.
• The hint banner: the Flows tab counts, per vDS exporter, the ends it resolved to a pod. If it sees pod ends it says “pod IPs are visible — routed CNI”, and there is nothing more to do. If it sees none while your inventory has pod IPs it says “pod traffic is masqueraded — deploy the in-cluster exporter”. Node ↔ node IPIP or VXLAN in the vDS flows adds “encapsulated node↔node traffic seen”.

The in-cluster exporter kit

For the overlay + SNAT case Topologix ships a download: topologix-k8s-flow-exporter-kit.zip (also inside every proxy release zip, under kits/, and in Admin → Versions). It runs the netobserv eBPF agent (quay.io/netobserv/netobserv-ebpf-agent:v2.0.1-community) as a privileged, host-network DaemonSet that exports IPFIX over UDP to the proxy’s listener on port 4739. The agent hooks the Calico pod veths (INTERFACES=/^cali/), so it sees every pod conversation before NAT and before the tunnel: pod → db:5432 instead of node → db:5432. The proxy needs no change: the agent’s template (5-tuple, 8-byte counters, flow times, MACs, a variable-length interface name) decodes with the standard IPFIX decoder.

• Unzip the kit on a machine with kubectl access, set the proxy address in kustomization.yaml (TARGET_HOST), then run kubectl apply -k . in that directory. Open UDP 4739 from every node IP to the proxy host.
• Each node is its own exporter (source IP = the node IP, observation domain 1). Within 5 minutes the nodes appear on the Flows tab, with the Scope column set from the node IP, for example “K8S Front · node prod-front-k8sdc2-worker03”. The Scope select in the drawer (Auto / vDS / In-cluster) overrides it.
• Sampling: the agent sends no options template, so on their own its rows would be sampling unknown. Topologix takes an in-cluster exporter as 1-in-1, and the kit sets SAMPLING=1 to match. If you raise SAMPLING to cut CPU (10 is the usual step), set the same rate as the exporter’s sampling override in the drawer.
• The agent sends no pod names over IPFIX. netobserv’s Kubernetes enrichment lives in flowlogs-pipeline, which the kit does not use. Topologix names the pod from its own inventory, as it does for a routed CNI. Enterprise elements such as Antrea’s pod-name fields (PEN 56506) are currently skipped by the proxy.
• Other CNIs: for Antrea use INTERFACES=/^antrea/ or Antrea’s own Flow Exporter; for Cilium use INTERFACES=/^lxc/ or Hubble with an IPFIX exporter. The agent needs kernel 5.8 or newer. On recent kernels CAP_BPF, CAP_PERFMON and CAP_NET_ADMIN can replace privileged. The kit’s README has the CNI checks, the verify steps and the fallbacks.

What each layer shows, and why nothing is counted twice

With the kit deployed, the same conversation can reach the proxy twice: once from the vDS (outside the node) and once from the agent (inside it). When both the VMware switch and the Topologix K8s exporter report a cluster, the post-NAT node copies are dropped: at ingest, per 5-minute bucket, before anything is stored, so the stored pairs, the Network peers panel and the dependency edges all keep one copy. The rules:

• A node row is dropped only when it has a pod-level counterpart on that node, in that bucket, to the same address and port. The node’s own traffic (kubelet → API server, filebeat → logstash, host-network pods), a NodePort (client → node:30080 is client → pod:8080 inside: another port) and every node of a cluster without an in-cluster exporter are kept.
• Order does not matter: when the pod flow arrives after the switch copy was stored (proxies re-send a bucket every minute until it closes), the stored node pair is removed, or reduced to the part the pod flows do not cover. Admin → Background workers → Network flows shows “post-NAT copies dropped (K8s exporter covers): N” on the proxy row; the ingest log line says natCopies=….
• The dependency sweep keeps a window rule as a safety net (pairs stored before the upgrade, or after a server restart): a vDS row from a node is the NAT shadow while that node’s in-cluster exporter was seen in the last 15 minutes and the same node’s pods reach the same destination and port. The same rule applies to the busiest clients the flow warehouse reports for an inferred or monitored database. The sweep log counts shadow=… and tunnel=….
• The ClickHouse warehouse and the proxy’s local store keep every exporter’s raw rows (the forensic record); the Flow query shows both copies there, each under its exporter.
• Within one layer the usual rule applies: the MAX across exporters per pair. The two veths of a same-node conversation therefore never add up, and the agent’s DEDUPER=firstCome already drops most of those duplicates.

Limits to plan for

Pod-level flows multiply exporters, datagrams and distinct conversations. Check these before you roll the kit out to a large cluster:

• Exporters per proxy: every node counts. A 33-node cluster plus the vDS fits under the per-proxy exporter cap (64 before proxy 1.232.0, 256 from it). Senders past the cap are dropped and counted as droppedExporter.
• Datagrams: the agent sends one flow record per IPFIX message, so its datagram rate equals its record rate: roughly the node’s active conversations ÷ CACHE_ACTIVE_TIMEOUT (5 s), plus the ended ones. The proxy accepts up to 5 000 datagrams/s per exporter and 10 000 in total; anything above is dropped and counted (droppedRate). On a busy cluster raise CACHE_ACTIVE_TIMEOUT to 15–30 s. The proxy aggregates per minute anyway, so this costs only timing detail, not data.
• Aggregation keys: pod-level ends multiply the number of distinct conversations. Raise aggMaxKeysTotal (server env NETFLOW_AGG_MAX_KEYS_TOTAL; 1 000 000 for a Croco-class site, see section 8) and budget ClickHouse disk for the extra rows.
• CPU on the nodes: with SAMPLING=1 the agent counts every packet, typically a few percent of a core on a busy node. The kit requests 100m / 128 Mi and limits the agent to 500m / 512 Mi.
• Pod IP churn: the resolver names pods from the pod inventory, which the Kubernetes metrics poll refreshes. A pod IP reused within minutes may briefly name the previous pod. ClickHouse keeps the raw IPs.
• Several clusters on one pod CIDR (Calico’s default 192.168.0.0/16 in each): an in-cluster exporter resolves pod addresses in its own cluster only — the cluster of the node it runs on, or the one you choose in the exporter drawer’s Kubernetes cluster select (for an exporter behind SNAT; the drawer suggests the cluster whose nodes and pods its flows show). For any other exporter, an address alive in two clusters is named by the other end of the flow when that end is a node or pod of exactly one cluster (shown as pod placement, cluster inferred from the other end, a weaker answer); when the other end names no cluster, or both, the address stays unresolved.

Good to know

• The server logs [netflow] db-cost: ingest proxy=… exporters=… raw=… pairs=… q=… per payload (a statement budget per batch), and the dependency sweep adds net=<pairs>/<edges> to its own db-cost line.
• Trying it without a device: node dev/mock-netflow-exporter.mjs --host <proxy> --rate 300 --protocols v5,v9,ipfix,sflow sends all four protocols; node dev/netflow-load.mjs generates load (--client-cidr 192.168.48.0/24 / --server-cidr draw the addresses from a pod or service CIDR).
• Naming: everything flow-related is netflow_* in the database and API — flows already means monitor canvases.