Proxy requirements and sizing in Topologix

Infrastructure & Networking

A proxy agent is one Node.js process that runs the checks and metric polls assigned to it, and optionally listens for NetFlow / IPFIX / sFlow, SNMP traps and OTLP. Most of its cost is per poll (CPU for parsing what a target answered) and per listener (memory for what it aggregates). This page gives the measured cost of each, three starting sizes, and how to check a running proxy against them in Settings → Proxies → Activity.

Requirements

• Node.js 22 LTS (Node 20 works; NetFlow then keeps its 7-day store in segment files instead of SQLite). install.sh installs Node 22 on a new host.
• Baseline host: 2 vCPU, 2 GB RAM, 10 GB disk — enough for a proxy with up to ~100 sources and no listeners.
• Outbound HTTPS to the Topologix server (the proxy polls it; nothing connects in). Inbound UDP only for the listeners you enable (NetFlow 2055 / IPFIX 4739 / sFlow 6343, traps 1162 — forward 162 to it — and the OTLP relay port).

What adds cost

Measured on the proxy code shipped in 1.204.0; "a core" is one CPU core fully busy.

Metric sources

Each poll costs CPU on the main thread for the request and for parsing the answer; the interval decides how often.

• VMware vCenter (proxy ≥ 1.204.0): one shared session, counter map and keep-alive connection pool per vCenter — no login, no counter-catalogue download and no TLS handshake per poll. A cluster poll costs ≈ 3.3 ms of CPU per 50 VMs (npm run bench:vmware: 60-VM cluster, steady state; the same harness measured ≈ 44 ms before 1.204.0, and a 123-source production proxy measured 99 ms per poll including TLS). The QueryPerf reply of 50 VMs parses in ≈ 0.75 ms (was ≈ 18 ms). Poll VMware sources every 30 s or slower: vCenter's realtime statistics are 20-second samples and every poll reads the last 5 minutes, so a 10 s poll mostly re-reads the same samples.
• VMware rounds (proxy ≥ 1.205.0, server ≥ 1.260.0): the sources of one vCenter integration are served by one coalesced round — one inventory + performance read per vCenter at the fastest source interval (each cluster, alarms list or VM refreshed at the fastest interval of the sources that need it), with a 15-sample ring per VM / host that only ever downloads the samples it does not have yet (≈ 6 KB of RAM per VM). Every source still builds its own values on its own schedule, from the same 5-minute window as before. Measured with npm run perf:vmware (100 sources at 10 / 110 / 600 s, 3 clusters × 60 VMs, real HTTPS): vCenter calls 780 → 344 / min, QueryPerf 183 → 56 / min, samples downloaded 459 k → 28 k / min, response bytes 16 → 2.7 MB / min, proxy CPU 2.1 → 1.2 % of a core. In Activity a round shows as round:<integration> among the slowest sources, and the Redaction card's VMware row lists rounds, entities, ring MB, the last round's time, fallbacks and the XML worker (bodies ≥ 256 KB parsed off the main thread).
• SNMP: ≈ 10 ms of CPU per poll (an interface walk plus device health).
• HTTP / TCP checks, Prometheus, HTTP JSON: negligible (well under 1 ms of parsing for typical answers).
• Several sources on the same vCenter / host share its session; the number of sources matters more than the number of targets.

Redaction

Every result is scanned for secrets before it leaves the proxy. Measured at 830 polls/min: ≈ 0.5 % of a core. Large results and OTLP bodies go to worker threads (redactionWorker, on by default): ≈ 70 MB of RAM per worker, two workers on hosts with ≥ 4 cores, one otherwise.

NetFlow / IPFIX / sFlow listener

Memory, not CPU, is the limit.

• Aggregation: ≈ 60–90 bytes per key (one conversation in the open minute) — the default caps (300 000 keys in total, 100 000 per exporter) hold ≈ 25 MB.
• CPU: ≤ 20 % of a core at 2 600 packets/s (production), ≈ 14 % at 60 000 records/s in the bench.
• Disk: the 7-day local store — up to 4 GB (SQLite, Node 22) or 2 GB (segment files, Node 20) by default (netflow.localMaxMB).
• The ClickHouse sink's queue while the warehouse is unreachable: ≤ 30 MB.
• The RSS guard works against the proxy's RSS budget: above 75 % the caps halve, above 90 % no new keys are taken, below 60 % they are restored.

OTLP relay and SNMP traps

The OTLP relay accepts bodies up to 8 MiB and redacts each before it is queued (large bodies on a redaction worker); its cost follows what the applications push. The trap listener is tiny (a bounded queue in front of the redaction).

Three sizes to start from

Pick the row that covers the busiest thing the proxy does; then check the Activity panel after a day.

Sizing table

RSS budget = the rssBudgetMB to set (below); leave the rest of the RAM to the OS and the page cache.

• The Medium row is a real one: a production office proxy with 123 vCenter sources, 54 ESXi hosts, SNMP, NetFlow and the redaction workers.
• Splitting is usually better than growing: one proxy per site or per vCenter keeps a slow target from delaying everything else on the same event loop.

Reading the Activity panel

Settings → Proxies (or Admin → Proxies) → Activity on the proxy's row.

What to look at

The Right now card is the live state, the sparklines the last 24 h.

• RSS bar — resident memory against the RSS budget ("budget 2.0 GB (default)", or where your setting came from). Green below 75 %, orange from 75 %, red from 90 %. Sitting in the orange for hours means the proxy needs more RAM, a larger budget, or less work.
• CPU % — of one core, over the last minute. A steady 50 % or more on a proxy with few listeners means polls are the cost: open CPU profile to see which source type.
• Event-loop lag p99 — how late timers fire. Under ~30 ms is healthy; hundreds of ms means something blocks the main thread (a profile names it).
• CPU profile — 30 / 60 / 120 s of sampling; the bars show where the main thread's time went (poll:vmware, redaction, netflow …) with CPU ms per poll of each type.
• Poll load — the five slowest sources by their last duration; a slow target does not cost CPU while it waits, but it holds a connection and delays its own results. A VMware source's duration is its build from the round (≈ 1 ms); the round itself is the round:<integration> row.

Settings that bound memory

config.json is bootstrap only (proxy ≥ 1.235.0)

From proxy 1.235.0 the proxy's config.json (next to the agent) carries only what it needs to reach the server and the few things that must stay on the machine. Everything else is set per proxy in Settings → Proxies → Proxy settings; each row shows the effective value and its source (set in the UI, imported from config.json, server environment, or default).

• Bootstrap keys (stay local, never imported): serverUrl, authKey, rejectUnauthorized, allowInsecureServerUrl, netflow.dataDir, netflow.clickhouse (honoured, but flagged — prefer a ClickHouse integration), apiRelay.tls (the relay certificate and key stay on the machine), plus the TOPOLOGIX_* / REDACTION_MEMO environment.
• A proxy older than 1.235.0 keeps the old rule (a local value wins over the server's); the drawer says so per row.

proxy config.json — the whole file

{
  "serverUrl": "https://topologix.example.com",
  "authKey": "…",
  "rejectUnauthorized": true,
  "netflow": { "dataDir": "/var/lib/topologix/netflow" }
}

Settings → Proxies → Proxy settings

The sections and the settings they hold (bounds are the proxy's own clamps; a value outside them is refused on save).

• Flows — ports listeners.netflow.ports.netflow / .ipfix / .sflow (0 … 65 535, 0 = off; defaults 2055 / 4739 / 6343); bind address listeners.netflow.address; exporter allow-list listeners.netflow.allowExporters (CIDRs, ≤ 256); server ports listeners.netflow.serverPorts (≤ 64); ship size topN (1 … 5 000 rows per exporter batch) and interval shipIntervalSec (10 … 3 600 s); local store size localMaxMB (1 … 65 536 MB) and retention localRetentionDays (1 … 7); per-minute cut localTopNPerMinute (10 … 20 000); aggregation caps aggMaxKeysPerExporter (1 000 … 2 000 000) / aggMaxKeysTotal (1 000 … 5 000 000); datagram caps ppsPerExporter (1 000 … 100 000) / ppsTotal (5 000 … 500 000); NetFlow memory budget rssBudgetMB (256 … 65 536 MB, deprecated — use Runtime); warehouse queue warehouseMaxQueueRows (100 000 … 20 000 000 rows); known exporters exporters (remove only, ≤ 200). All under listeners.netflow.; the listener switch itself is the existing Flows toggle.
• Syslog — listeners.syslog.forceOn, .port (1 … 65 535, default 1514), .address, .udp, .tcp, .allowedCidrs (≤ 256).
• OTLP — listeners.otel.forceOn, .port (1 … 65 535, default 4318), .host.
• SNMP traps — listeners.snmpTraps.forceOn, .port (1 … 65 535, default 1162), .address, .community (≤ 64 characters).
• Changes + Catalogue API relay — listeners.apiRelay.port (1 … 65 535, default 4320), .host; the on/off switch is the existing relay toggle (Expose the Changes + Catalogue API on this proxy; the catalogue inventory needs proxy ≥ 1.262.0); apiRelay.tls stays in config.json.
• Runtime — memory budget runtime.rssBudgetMB (256 … 65 536 MB, default 2048; the Activity bar, the CPU-profile refusal above 90 % and the NetFlow guard use it; server-wide default: env PROXY_RSS_BUDGET_MB); heap size runtime.heapMB (512 … 131 072 MB, proxy ≥ 1.236.0, applied on Restart proxy); redaction switches runtime.redactionMemo and runtime.redactionWorker (on / off; off keeps every scan on the main thread — saves ≈ 70 MB per worker; switching it off takes effect on the next restart).
• Listener changes (ports, bind address) rebind live; a port the machine refuses shows an error on that row and the proxy keeps running.

Migration from config.json

On the first contact after upgrading to 1.235.0 the proxy reports what its config.json still carries. The server imports those values once as that proxy's settings, so behaviour is unchanged on upgrade day (they show as "imported from config.json"). From then on the local values are ignored (one log line per key) and the drawer lists the leftover keys — remove them from the file. A setting you reset in the drawer is never re-imported. netflow.enabled / apiRelay.enabled import into the existing switches.

Heap size and Restart proxy

Set Runtime → heap size and press Restart proxy: the proxy relaunches itself with --max-old-space-size — no service-file edits. An operator-set NODE_OPTIONS heap is reported in the drawer and should be removed. Restart proxy is audited (proxy.restart) and rate-limited to one per 2 minutes per proxy.

Good to know

• null in the Activity panel ("—") means not measured, never zero.
• Numbers on this page come from npm run bench:vmware, npm run perf:netflow, the redaction benches and the first production CPU profile (2026-09-27); they are per core of a current x86 / Apple-silicon server.