Alerting in Topologix

Monitoring & Observability

An alert rule lives on a flow (flow menu → Alert rules…): it watches monitors, metric sources and process lanes of that flow and sends through the flow's alert-channel integrations (Slack, Teams, e-mail, webhook, PagerDuty, Telegram, Twilio, ServiceNow). Every alert is evaluated and recorded in the rule history; what reaches a channel is decided afterwards by one shared suppression gate (ack, maintenance, change, schedule, flap, parent-down) and by the rule's own routing, reminders and digest settings. Nothing is ever silently dropped: a held alert is in the history with a chip that says who held it.

Requirements

• Edit access to the flow (owner or admin) to create or change rules, silence nodes and acknowledge with a hold; view-only users can read the history and acknowledge without silencing.
• At least one alert-channel integration visible to the flow (Settings → Integrations). A rule can only send through channels the flow can use — checked when the rule is saved and again at every send.
• Organisation admins (or a user without an organisation, for their own estate) for maintenance windows on everything or on whole catalogue services.

Create a rule

1. Op
2. Pi
3. Ch
4. Sa

Held, not lost — suppressions

Every send asks one question first: does a live suppression cover this alert? If yes, the alert is recorded with the suppression's id and not sent; the in-app toast still appears, greyed, and never jumps the canvas. Workflow alert triggers still fire (they see who held it). A suppression either holds notifications (evaluated and recorded) or, for maintenance, does not evaluate at all (no event; N-of-M and hysteresis state do not advance). Recoveries are never held by flap mutes.

How the chips read

The rule history, the toast and the change page show the same chip on a held alert:

Dependencies (parent down)

A node's alerts wait while one of its parents is in error — the child's alert is recorded as held and delivered only if it is still failing after the parent recovers. Parents come from:

• the node panel / rule form Depends on (explicit nodes);
• Alert rules → Silences → Canvas arrows: "A → B" means A calls B, so B is A's parent (one hop);
• Infrastructure: a service node whose runs-on chain reaches a VM or host node on the canvas (and a VM node's host); an open Hypervisors host down finding on such a host holds its children too.
• Cycles are refused when you save (the picker hides choices that would close one). Children that were already alerting are not held.

Where to see what is held

The rule history (Alert rules → Rules & history) groups held rows by their holder; Alert rules → Silences lists the maintenance windows, acknowledgements and change windows touching the flow; the node panel shows the parent holds on that node; GET /api/suppressions?active=1 returns the live rows you may see. Every maintenance create / edit / end and every flap mute is in the Activity log.

Fewer false alarms — how a trigger decides

One evaluation order for every metric trigger: implausible samples are dropped → the verdict (a scalar, an expression or a composite) → the N-of-M ring → hysteresis → the state flip → severity and damping → the rule's schedule → cooldown → the send (suppressions, routing, digest). Reminders never re-enter evaluation.

Hysteresis and N-of-M

On a threshold trigger (and on a source's health thresholds):

• Clear value: "alert above 90, clear when below 85" — a breach holds while the value stays inside the band (at 85 it still holds). Stops a value hovering at the threshold from alerting on every sample.
• N of M: "3 of the last 5 samples" — the trigger breaches only when at least N of the last M (≤ 30) samples breach. A missing sample counts as not breached; it never wipes the ring.
• Both warm up from the stored samples after a restart, so a restart neither re-alerts nor forgets a breach.

Expressions and composite conditions

A threshold may judge an expression over named metrics (errors / requests * 100 > 5; + − * /, parentheses, min, max, abs) or a composite of conditions joined by ALL / ANY. A missing value, a division by zero or a non-finite result is not measured — it never fires. Limits: 200 characters, 32 tokens, 8 variables, 8 conditions; a composite with two or more conditions drops the clear value. The editor previews the expression against the node's latest sample. Derived values have no rollup history (no charts, no seasonal baseline).

Plausibility bounds

On a metric source (under the thresholds): per metric a min and / or max; a sample outside them is ignored everywhere (health, alerts, rollups) and the card says "1 implausible sample ignored". Use it for sensor glitches (a 6,000 °C reading) rather than raising thresholds.

Log error baselines

A Graylog All Services source on the Baseline alarm rule judges every service against what is usual for it at this weekday and hour — a provisional baseline for the first three weeks, then the seasonal one (see the Graylog guide). Its breaches reach the toasts, the Log errors above baseline banner and the Alerts page without any rule. To send them to channels, tick Fire when a service's log errors deviate from its baseline (and on recovery) — the metric-baseline trigger:

• One alert per service, with its own recovery, as one alert per item does (above 20 at once a single summary alert stands for the rest). The recovery comes when the breach resolves: back below ¾ of the line the row is clearing, and it resolves after the source's clean-check count (10 by default); a re-breach while clearing keeps the same alert open, without a new toast. The message is the breach sentence — transfer: 40 errors in 60 s, 8× the usual 5 (alarm above 15) for Tue 14:00 (TimeoutException ×30, …) — the value is the count and the threshold the alarm line.
• Also on new error types (on by default): one warning per type the service has not logged for 7 days — payments: new error type TimeoutException (×140). No recovery follows. Sampled types (fingerprints, no error type field) are confirmed over two checks first and pause on a noisy service (the source's Noisy service threshold, 25 error families in 24 h by default).
• Every log source can drive it: Graylog All Services, the Syslog receiver, OpenTelemetry Logs — all services, Aria Operations for Logs Per service, vCenter Host logs (ESXi hosts and vSAN), Sentry Errors per project, Elastic APM Errors per service and Elasticsearch / OpenSearch logs Errors per service — the same rule, banner, Alerts row and template variables; each guide lists what one check costs. Each of these forms has an optional RECAP incident correlation picker (shown when a RECAP integration is visible): the card's rows then get a RECAP badge, matched by service and check time.
• Template variables for these alerts: {{baseline}} (the usual count per check), {{ratio}} (the source's ratio), {{window}} (what one check counts, e.g. 60 s), {{topErrors}} (TimeoutException ×140, IOException ×20) and {{provisional}} (provisional or seasonal); {{service}} is the service. They are empty on every other alert.

Value-change notices

Trigger Value changed on a metric key: a notice (severity info) whenever the value differs from the previous sample — firmware versions, HA roles, a config hash. Ignore the first value suppresses the notice at the first sample after a restart. Notices are rendered as notices everywhere (Slack "Notice", e-mail [Notice], PagerDuty severity info with a unique key); ServiceNow and Twilio skip them.

Warn before error (damping)

A monitor with Warn first N (2–50): the first failures turn the node orange; the alert goes out at error severity only when the N-th consecutive failure arrives. Rules with notify on warning also get the early warning (and its recovery).

No data

On a metric source or a monitor: If no data for … min, treat as error (red, fires the source-error / monitor triggers like a real failure), warning (orange, nothing sent) or unknown (grey, never an alert). The card says "No data since 14:02 — treated as …"; data coming back recovers it on the next poll. Sources 1 min – 1 day, monitors 1 min – 7 days; a Don't evaluate window skips the check; a restart never declares silence before the delay has passed.

Who hears it, when, how often

Reminders and escalation

Remind while still failing every ≥ 5 min, at most N times (default 10): "Still failing after 45 min — …" with the same alert key (PagerDuty / ServiceNow update the open incident). A rule with reminders is told once when the problem starts; rules without them keep the old "fire while breached every cooldown". Escalate if still failing after ≥ 5 min: one "Escalated: …" to the chosen channels, which also hear the recovery. Reminders pause while the alert is acknowledged, in maintenance or a change, or off duty — no event row is written for a held reminder — and stop on recovery. Open alerts live in the database, so a restart loses no reminder.

Routing by severity

Route by severity: error channels, warning and notice channels. A warning falls back to the error channels, a notice to warning then error, then the rule's channels. A recovery goes where its error went.

Message templates

Custom message: a title and body with variables — {{node}}, {{rule}}, {{severity}}, {{value}}, {{threshold}}, {{downtime}}, {{deepLink}}, {{time}}, {{reason}}, {{suppressedBy}}, {{topErrors}} … (30 in all), with fallbacks {{service|unknown}}. The editor warns about unknown variables and previews the result. The history keeps the built-in text.

Digests

Batch into a digest (1, 5, 10, 15, 30 or 60 min): the rule's alerts are queued per channel and sent as one "N alerts in the last 5 min" message when the window closes (the flush runs every hygiene pass, ~60 s). Recoveries, reminders, escalations and tests are never batched. Suppression is re-checked for every item at flush — a silenced item is dropped ("dropped (silenced)"). The history shows "digested → sent 14:05 (7 items)". Sent digest rows are kept Alerting: digest history retention (7 days, 2 on SQLite).

VPN tunnels (Cisco, FortiGate, PAN-OS)

Built-in, per tunnel: VPN tunnel down (error) on FortiGate / PAN-OS (the tunnel list) and on Cisco IOS / IOS-XE routers polled by SNMP — there the tunnel table is compared read to read, because Cisco deletes a down tunnel's row: a peer missing in 2 consecutive reads is down, a failed read never counts. Cisco adds VPN tunnel flapped (warning — re-established between two polls: active time reset, index replaced or counters restarted), IKE (phase 1) down (warning) and VPN mass tunnel drop (error — 3 peers or 50 % down within 15 min). Every knob is on the SNMP source form (VPN tunnel alerts); see the SNMP guide. DMVPN peers and crypto sessions read over SSH on a schedule (the SSH integration's VPN tunnels source mode) raise the same down / flapped / mass-drop alerts.

One alert per item

On metric source in error: One alert per item sends one alert per breaching row (a disk, an interface, a queue) and its own recovery, instead of one per source; above 20 items at once a single summary alert stands for the rest.

On-duty hours

Days, from / to (overnight ranges belong to the day they start), a zone (default the organisation zone, DST-correct). Outside: hold notifications (recorded as "Held · outside schedule") or don't evaluate.

Alerting quality

Alert rules → Quality (this flow) and Admin → System → Alerting quality (every flow you own), 7 or 30 days. A dash means nothing was measured in the window — never zero.

What it shows

• Time to recover — every run of errors of a subject paired with the recovery that ends it: median and p90, how many are open now, by catalogue service (single-service metric sources, latency alarms) and by monitor / source / rule, with links to the node.
• Noisiest rules — alerts per rule split into delivered / held / digested, plus reminders, notices and recoveries.
• Flapping — subjects with at least half the flap threshold of failures on one day, the automatic mutes in the window and the ones active now.
• Suppression — held alerts by kind (acknowledged, maintenance, change window, outside schedule, flapping mute, held by a parent) and the held share = held ÷ (held + delivered).
• Digests — batches sent, alerts in them, alerts per batch, dropped as silenced, waiting.

API

GET /api/alerts/quality?days=7|30&flowId=<id> (any access to the flow) or without flowId for every flow you own (super admins: &ownerId=). Four database statements, cached 60 s.

The Alerts page and the wallboard

The Alerts button in the middle of the top bar (with its live errors · warnings pill) opens one page with everything open right now across every flow you can see, plus the hardware findings and catalogue services your organisation's integrations can see. ⌘K → Alerts gets there too. It updates live: whenever something opens, recovers or is held, the page and the pill refetch within a few seconds (and every minute while the tab is visible).

What it lists

Rows judged against a baseline

• A log errors row says its number against the usual one for this weekday and hour: transfer · "Log errors 160 vs ~18 usual for Wed 14:00 · 8.9×", with chips — seasonal · 9 wk (the weekday×hour baseline) or provisional · 2 h of data (the hourly mean while it learns) — the top error type and "+2 types", and a 24-hour sparkline with the usual value as a band. Open for: "since 14:02 · checked 30 s ago".
• After the spike the row is clearing: "Log errors 9 vs ~12 usual · was 41 at 13:52", the chip clearing 4/10 clean checks, Ack hidden (it resolves by itself), Open for "recovering since 14:21". An acknowledged row is dimmed: "acked by Ann 12 min ago".
• A seasonal (unusual for the hour) row reads the same way: "p95 1.45 s vs 420 ms usual for Wed 14:00 · 3.5×".
• The same subject on several flows (a switch on two network flows) is one row: "seen in 2 flows · VK-Core-AT network, Office network", acknowledged together with Ack both.
• A vCenter alarm on a VM shows the VM's CPU (or memory) — "VM CPU 97 % · vCenter alarm "Virtual machine CPU usage"" — with its cluster, host and when vCenter raised it. The figure comes from the Hypervisors collector's live read on the server that runs it; when it has no reading (or another server instance answers) the row shows no number rather than a guess.
• A node still called Metrics Panel is named after its source (Graylog · 3 services), and a group lists its members' values: "checkout 743 ms · transfer 612 ms · free-spin 540 ms".

Incidents — measured impact on the services that matter

• An incident is a measured, meaningful impact on a service that matters — its throughput fell, its error rate or latency rose, a user-facing check is down, a production database it depends on is down, or a Business KPI left its weekday × hour pattern. Alerts are its suspected causes; an impact with no cause is still an incident, and it resolves when the impact is back to its baseline. An alert group is not an incident (problems that started together — below).
• What is watched. Catalogue services whose tier is one of the organisation's impact tiers (critical and important by default — Settings → Organisation → Incidents; an untiered or standard service is never judged), a monitor marked user-facing (or one on a tiered service, or on a public URL when the organisation counts those), a database instance a tiered service depends on (two call hops), and a metric source's Business KPI block (the "online players" case). The footer says what the watch covers ("Impact watch covers 4 of 6 tiered services · checkout has no throughput series") or "No tiered services — incidents need critical or important tiers in the Catalogue".
• The incident card sits above the alert groups: "INCIDENT · OUTAGE" (red) or "INCIDENT · DEGRADED" (amber), the tier, the service, and ONE number — "throughput −35 % · 12/s vs 18/s usual for Wed 14:00 · since 19:01 (28 min)". Expanded: each signal with the baseline that judged it and its peak, the services also impacted and the callers affected (victims, never causes), the suspected causes ranked by rule — or the no-cause notice that says what was searched — and the timeline. Actions: Explain, Acknowledge (ownership — never its end), Open service, and for owners / admins Resolve (its subject + signal kinds are not judged again for 60 min), Not an incident (24 h) and Attach a cause.

Incidents — lifecycle, causes, notifications

• Confirm and resolve. A throughput / error / latency / KPI signal opens after 2 passes with a new 5-minute bucket; its start is back-dated to the first bucket that tripped. Every signal back inside its clear band → Recovering (2 of 3); 3 clean passes → Resolved (at the first clean one). A signal whose input goes silent while the service is still watched is held ("not measured since 19:14") — missing data never resolves an incident. A return within 30 min reopens the same incident. The counts are editable in Admin → Alerts (with the effective value and its source; ALERTS_IMPACT=off|shadow|on overrides the mode only, shown read-only), each organisation can override the mode in Settings → Incidents (shadow records incidents but shows none).
• Suspected causes. Open alert groups, problems in no group, changes and (with a RECAP integration) the RECAP incident matched to the service at the onset are ranked by how they relate — on the service 1.0, a dependency it calls 0.9 / 0.7 (one / two hops), what it runs on 0.8, a change to it 0.9 (to a dependency 0.6), RECAP 0.8 — times how close they started (from 45 min before to 10 min after; later ones are symptoms; a RECAP incident is timed by its first sighting, so one first seen hours earlier is not a cause). ≥ 0.8 (or pinned) = identified. A RECAP cause carries an R badge and opens in RECAP; the lookup runs once per new incident, in the background, and a slow or unreachable RECAP simply adds nothing.
• Notifications (Settings → Organisation → Incidents → Notifications, off until an owner / admin turns them on): the alert channels the organisation's owner can use (own + shared with the organisation), the least severity and tier, reminders every N minutes while nobody has acknowledged it (at most M), an escalation to other channels after N unacknowledged minutes, and the minimum gap between two messages of one incident (300 s). Messages: opened ("INCIDENT · OUTAGE — checkout (critical): throughput −35 % (12/s vs 18/s usual for Wed 14:00) since 19:01. Suspected cause: cascade from esxi142 (0.8)." or "Cause unknown — …"), escalated, cause identified (once), reminders, the escalation, reopened and resolved ("Resolved 19:42 · 41 min · peak −62 % at 19:14") — all under one identity per incident, so PagerDuty and ServiceNow close what they opened. Recoveries are always owed: an incident already announced still sends its recovery (to the channels its alarms went to) after notifications are turned off or the organisation switches incidents to shadow; with incidents off, open incidents pause and resolve — with their recovery — once incidents are on again. A maintenance window or an active change on the service holds the message (recorded, not sent); once it ends an incident still open is sent once, "[held during change 'Spring 6.1 rollout']". Send test posts a test message; the last 20 notification rows show what was sent, held or released and how each channel answered.
• Explain on an incident asks the monitoring bot what caused it (the incident, its signals, its victims, its ranked causes and the automatic verdict when there is one; ≤ 4 model turns). The answer is stored on the incident and shown again for 6 hours at no cost; it counts against the same daily budget as alert-group explanations (Explanations per organisation a day, Admin → Alerts, default 24). With a stored automatic verdict and no stored explanation, Explain first shows the verdict — no model call — and Deep-dive with the monitoring bot runs the bot.
• Automatic AI. Once an incident has settled (5 min after it was confirmed and 3 min after its suspected causes last changed; at the latest 15 min), the organisation's analysis AI role (Organisation → AI models) reads it — the impact, the signals, the ranked candidates with their relation and timing, the symptoms, the callers, what was searched, earlier incidents on the same service; redacted with the organisation's rule set, at most 12 KB — and answers in a fixed form, kept short (at most 1,200 tokens — Admin → Alerts → Verdict output budget): the candidate that explains the impact or none, a confidence, a ranking, a headline, the likely cause (≤ 40 words), up to four evidence lines quoted verbatim and the next check (≤ 25 words). An answer the budget cuts off — or one that comes back without its likely cause or evidence — is never shown as a verdict: it is asked again at once with twice the budget (up to 2,500 tokens; both asks count toward the caps), and if that is cut too the card says "AI answer was cut off twice — Analyse now to retry" instead of a verdict. A part the model did not give (no evidence, no next check) is left out of the panel, never shown empty. The card shows it as "AI · cause: Cascade from esxi142 · 0.82 — …" (or "AI · no listed cause · 0.35 — … Next: …"). An agreeing verdict at 0.7 or more makes the cause identified (a timeline row; the notification "Cause identified" follows); a disagreeing one is shown beside the rule's top and never reorders the list. An incident someone pinned a cause on or marked Not an incident is never analysed automatically (a resolved one gets at most one look: within 30 minutes of its end, or later when it lasted 15 minutes or more). Alert groups get the same once settled ("AI · common cause · 0.82 — …", or "burst, not common cause"), except when the rule is already confident (never for an unclear class, a shared platform or a plain alert group — those wait for the AI, see Alert groups), the group is small or warnings only, every member is held or acknowledged, or the group is a cause of an incident whose verdict is in (a cause group waits while its incident is still being analysed). Each card says why it was not analysed ("AI · not analysed — the rule is confident (cascade 0.90)", "— hourly cap reached … next slot 14:40", "— the platform AI pack is 90 % used — the rest is kept for Explain and the monitoring bot"); owners and admins can Analyse now (it skips the waiting, never the caps or the pack reserve). At most 3 runs per incident or group, one retry 10 minutes after a provider error, and a re-run only on a real change — a quarter or more of the members (an incident's services and candidates) new since the verdict, or the rule's class (top cause) changed — and at least 60 minutes after the last verdict (Admin → Alerts → Re-run no sooner than, 15–1440 min; a cut answer never restarts that clock). The server log's [alerts] verdict-queue line says why each re-run fired ("members +40 % (5 → 7) · class same · verdict 72 min old").
• AI caps (Admin → Alerts, each with its effective value and source): 6 analyses per organisation an hour, 30 a day (an organisation can switch the automatic AI off or set its own 0–120 a day in Settings → Incidents → Automatic AI — 0 = the rules only), 60 an hour for every organisation together, and on the platform key automatic spend stops at 90 % of the AI pack (your own provider is counted, never pack-gated). Automatic AI on / off there is the kill switch; the server's ALERTS_INCIDENT_AI=off|on is an emergency override, shown read-only. Settings → Incidents → Automatic AI shows the meter (this hour, today, explanations, the pack this month), the analysis model and the last 20 runs. Each analysis is one AI action (about 3,000 tokens in, 250 out).
• History and numbers. GET /api/alerts/impacts lists the organisation's incidents (30 days, with a summary: count, outages, no cause, dismissed, median minutes, MTTA from the confirmation, MTTR). The daily digest leads with them ("2 incidents yesterday (41 min, 12 min) — 1 open now"), the monitoring bot lists them first, the wallboard shows a strip of the live ones at the top, and a workflow can start on Incident (impact) — opened, escalated or resolved — with the incident as its trigger context. Resolved incidents are kept for Incidents: retention (90 days, SQLite 30).
• Rolling it out. Incidents and the automatic AI are on by default; notifications stay off until an owner / admin turns them on. To watch first, set Admin → Alerts → Impact incidents to shadow (incidents are computed and stored — no incident cards, notifications or incident AI; alert groups are not affected) and the AI kill switch to off (it stops the alert groups' automatic AI too) — or, before the first start of a new version, the server's ALERTS_IMPACT=shadow and ALERTS_INCIDENT_AI=off (emergency overrides, shown read-only in Admin → Alerts until removed). Then: tier the services that matter in the Catalogue, read the shadow incidents (GET /api/alerts/impacts answers the owner / admin in shadow) for false opens — night-time traffic dips first — and the server log's [alerts] db-cost: impact-watch / group-watch lines for the database cost; switch to on with notifications off; then notifications on a dedicated channel; then the automatic AI, watching the meter in Settings → Incidents → Automatic AI.

Business KPI incidents and the automatic investigation

• The switch. On a service in the Catalogue (the ownership row), Business KPI marks a service whose health is a business number — checkout, payments, sign-ups. A Business KPI service is watched by the incident watcher whatever its tier (an untiered one counts as critical; its tier, and so change-approval criticality, is not changed — they stay separate settings). Editors of the service's owner switch it (the same rights as the ownership row); it is audited as an ownership edit. A metric source's Business KPI block (the "online players" case) works as before.
• What opens a KPI incident. Everything that opens an incident on a tiered service (throughput, error rate, latency, a monitor, a database it depends on — the table above) and any open problem on the service itself: an error alert (a rule, a source or a service-health card whose service is the KPI service) or an unusual one — a vs usual (seasonal) rule that says the value is out of its weekday × hour baseline — and a warning. Each is a significant signal ("checkout problem: Orders per minute — orders 12/min < 40"; "checkout out of its usual range: …"), so a critical KPI service opens an error incident. By default it opens on the first pass (no waiting for 2 passes); Settings → Organisation → Incidents → Open the incident on the first pass turns that off (then the alert must be open for the instance's passes × 5 min). A stale, held, acknowledged-hold or clearing alert never counts; the alert clearing clears the signal and the incident resolves as usual.
• Auto-investigate KPI incidents. As soon as a KPI incident is confirmed the monitoring bot explains it — the same investigation as Explain (the incident, its signals, its suspected causes, the evidence pack, ≤ 4 model turns) — in the background, never delaying the incident or its first notification. It runs as the organisation's owner on the organisation-shared integrations, with stored data only (no live reads), and it is recorded on the incident: the card shows AI investigated automatically · 19:04, the likely cause and Show the investigation; the timeline gets an Explained (automatic) row. Only a viewer who sees every service and suspected cause of the incident sees the answer — anyone else reads "AI analysis is not available for your view" (the rule the automatic verdicts follow). Explain on that incident then shows the stored answer at no cost. It runs once per incident, and once more when the incident escalates (warning → error, or to an outage) unless Investigate again on escalation is off.
• Budget. Each run is one explanation in the organisation's daily budget (Explanations per organisation a day, 24) and runs only while the automatic AI caps have room (6 an hour, 30 a day per organisation, 60 an hour for the instance) and the platform AI pack is below its reserve (90 %). A refused run is recorded once in Settings → Incidents → Automatic AI → last runs (reason: explain-budget, cap-day, pack-reserve, unavailable …) and tried again 15 minutes later while the incident is live; an incident confirmed more than 2 hours ago is not investigated for the first time (Explain is there).
• Switches. Admin → Alerts → Auto-investigate KPI incidents (on) and Investigate again on escalation (on), each with its effective value and source; the Automatic AI kill switch (and ALERTS_INCIDENT_AI=off) stops it too. Per organisation: Settings → Incidents → Auto-investigate KPI incidents (on), shown with what decides it ("off · this organisation's automatic AI is off").
• Notifications. When the investigation lands after the incident was announced, every configured incident channel gets ONE more message — "AI analysis — checkout: the orders-db connection pool is exhausted …" — under the incident's identity (the cooldown, maintenance holds and the severity / tier gates apply as for every incident message). Every later message of the incident carries an AI analysis row. The text is the answer's likely cause, redacted with the organisation's redaction rules and cut to 300 characters; when the rules cannot be read the row is left out.

Alert groups and "when things started"

• Under the headline, When things started · last 24 h shows 48 half-hour bars — how many of the open problems started in each half hour, errors (red) over warnings (amber). Click a bar to list only the problems that started then (the bar gets a ring; Show all or a second click clears it); it combines with the other filters. The wallboard shows the same strip, read-only; a phone leaves it out.
• History: new failure or standing condition? Every open problem carries what its own state changes say about the last 7 days (Admin → Alerts → History window, 3–30 days): how often it fired, on how many days, and how often it flipped in the last 24 hours. A problem that flipped 10 or more times in 24 hours (Flapping: flips in 24 hours, 4–100) gets the chip flapping · 20 flips in 24 h; one that fired within an hour of the current time of day on at least 4 of those days (Recurs daily: fired around this hour on, 2–30 days, never more than the window; judged in the organisation's timezone) gets recurs daily 06–07 · 6 of the last 7 days, the hours in your clock. Its tooltip says the rest ("Fired 26× on 6 of the last 7 days · 20 flips in 24 h — a standing condition, not a new failure"). The history is read with one small query for the problems that do not have it yet and kept 10 minutes (at most 500 problems and 20,000 hour buckets a read; the server log's [alerts] db-cost: open-problems-history line).
• Problems that started within 15 minutes of each other and are related fold into an alert group card above the list. Related means: on the same canvas node; about the same service; a service whose reason is the other card; a VM, host, k8s node, pod or service within two hops of each other on the Infrastructure map (a k8s node that is a VM, or a host that is a device, counts as one machine); a cluster and its hosts / VMs; a VM alarm and the host it names. A single problem is never a group; stale latches and services without data never take part. An alert group is not an incident: in Topologix an incident is a measured impact on a service that matters; an alert group is problems that started together.
• Each group gets a class — a rule, never a model, the first that holds wins — shown as the card's eyebrow with the rule confidence in its tooltip. CASCADE: one member is upstream of all the others (the VMs on a host, the pods on those VMs and the services on those pods are downstream of it) — "Cascade from esxi142", why "Earliest problem and upstream of the rest: 1 VM and 3 services started failing within 4 minutes." (when the host reported after its VM: "… though worker04 reported first"); only a cascade names a suspected origin. COMMON CAUSE: at least half of the members (two or more, on two or more services, nodes or entities — or three or more of one service) share one signature — the same new error type, top log-error type, finding, vCenter alarm, rule, metric or message: "COMMON CAUSE · new error type org.springframework.web.servlet… · Graylog · All Services", why "The same new error type appeared in 10 services within a minute — one shared change or dependency, not a chain." Nine pods of one service logging the same new error is a common cause (a dependency or a change they share), not their node. SHARED PLATFORM: three or more members of two or more services run on one host, VM, k8s node, datastore or vSphere cluster that reports nothing itself, and no signature is shared by most of them — "3 VMs that run on esxi142 started failing within 2 minutes; esxi142 itself reports nothing — check the host." (one service's pods on one node is no evidence of the node.) BURST: three or more different problems from one canvas node (one log source) within 2 minutes with nothing else shared — "likely one event at the source (a restart, a backlog, a re-learn), not 10 separate failures." Otherwise ALERT GROUP: "Started together, no direction between them: transfer first, then 1 service within 3 minutes."
• Standing conditions never start a group. A problem that is flapping or recurs daily at this hour (its history chip, above) is a standing condition, not a new failure: alert groups form from the other problems only. A standing condition may join a group one of them starts — its chip says "flapping" on the card, the why line adds "webauth-server flaps (20 flips in 24 h) — a standing condition, never its origin." — and it is never a cascade's suspected origin. Standing conditions alone form no group; each stays a single row that says "not grouped: flapping — a standing condition". One source, one member: a card that only mirrors a hardware finding on its own VM or host — a service whose health is that finding, or a database source's latched VM metric ("Latched warn on Prod PG · prod-intg-resource-pg01" — the VM numbers the database check reads from vCenter) — is the finding's card, not a second problem: it is folded into it as a note ("also latched Prod PG") and its own row says "mirrors the finding on prod-intg-resource-pg01". A VM and a service on it are only a cascade when the service shows something else (errors, latency, traffic). A common cause needs a specific signature: the same new error type, top log-error type, message, vCenter alarm or finding names one thing; the same kind of threshold — latency, CPU, memory, an alert rule, a metric — does not. Such a generic signature counts only when its problems share a dependency or a platform (on the Infrastructure map) or when each of them is unusual for this hour against its seasonal baseline (no baseline = not unusual). Otherwise it is no common cause, and the problems group only on a real link — two services on one "All Services" log source share a collector, not a cause: each row then says why ("not grouped: Latency breach alone — no shared dependency, platform or unusual breach", "not grouped: it only shares a log source with webauth-server and bonus-config — nothing else links them").
• Unclear? The AI decides first. A class is clear when its rule is at least 0.8 confident and no other rule comes within 0.15 of it. An unclear class — and every shared platform or plain alert group — is never shown before the automatic AI (below) has had its say: the card reads ALERT GROUP with an Analyzing chip, the rule's provisional class in small text ("rule: shared platform 0.80") and what is being analysed ("9 problems · 1 service · 2 log samples · 0 changes · queued #2 of 5"); such groups go to the front of the AI queue (the caps and the pack reserve still hold). When the AI answers, the card changes in place — no reload: its class names the card and the line gives its reason, "AI · 0.70 · sportsbook backend returning 503, not a worker02 issue (rule said shared platform)"; the class is stored with the verdict, so it stays put. When the AI cannot run (a cap, the pack, no provider, switched off, warnings only, first seen too long before the automatic analysis looked at it …) the card shows the rule's class with why — what happened and when, in your clock: "rule-based — AI not run: held by the hourly cap since 10:12 (6 an hour for this organisation) — next slot 10:58", "… automatic analysis first looked at it at 15:12, 20 h after the watcher first saw it (19:01) — it only runs within 30 min of first sight" (the 30 minutes count from when the alert-group watcher first saw the group, never from when its problems started — a group first seen after a deploy is still analysed), "… the rule is confident (cascade 0.90) — decided 10:12". A clear class shows at once; the AI's confirmation is added when it runs ("AI · 0.85 · … (confirms the rule)"), and an AI that disagrees still replaces it. The daily digest, its mail and the monitoring bot say the same ("Analyzing (rule: shared platform 0.80)").
• The deep analysis has the last word. Explain on an alert group runs the monitoring bot with its tools (logs, events, metrics, changes); it now ends with a structured conclusion — the class (cascade, common cause, shared platform, burst, related or separate events), whether it is one incident, a confidence, a one-line reason and the members it found to be standing conditions — while its written answer stays in the panel. When that conclusion names another class than the card shows, the card takes it: the eyebrow becomes that class and the line reads "AI (deep) · common cause — not a real cascade, the same memory pressure behind both"; the automatic one-shot verdict is retired for that group (not re-run while the conclusion stands, never shown beside it) — one opinion on the card. When it concludes separate events (or not one incident), the group is dissolved: no card, each problem a single row saying "AI: separate events · analysed 12:16", and the same problems (or some of them) are not grouped again for 24 hours (Admin → Alerts → Dissolved group stays apart for, 1–168 h); a new problem joining them is new evidence. The conclusion covers only the problems the bot analysed — a group that has grown since is judged afresh. Admin → Alerts → Deep analysis overrides the verdict (on by default) turns this off: the conclusion is still stored, the card keeps the automatic verdict. Each conclusion is one extra model request of the explanation (it counts toward nothing else). The one-shot verdict itself now reads each member's history and seasonal comparison and is told that a member that recurs daily or flaps is a standing condition — it does not confirm a class that rests on one.
• The header reads "<CLASS> · <name> <context> · N problems · since 14:01 (28 min)" (a burst adds how fast: "4 problems in 90 s"); a group of new error types adds whether the type is still being logged — "still occurring · last seen 10:42", or "stopped 19:36 · recovers 20:36 unless seen again" (a sampled type: after 6 h at least, and it does not reopen). Expanded: the why line, a signature line when members share one ("10 of 10 share: NEW TYPE first seen 13:58 in checkout · error type …" — the badge says the error type is new to these services' logs, its tooltip "first seen in these services' logs at 13:58 — not seen in the previous 7 days"; the type in monospace, the whole text in its tooltip), then the members — each with its kind, when it started and its number ("Log errors 160 vs ~18 usual · 8.9×"). Ack all N acknowledges every member you can (clearing rows resolve by themselves); the third button opens the origin (a cascade or a plain group), the log source (a burst, or a common cause from one source), else the first member. The members stay in the list below, each marked "part of the <class> alert group above". On a phone the card keeps the header, the members, the signature line and the buttons.
• Expanded, an alert group also says what Topologix found about it — once, when its members change, never on your page load: up to two log samples of the shared error type ("first seen 13:58 in checkout"), the changes around it ("Change: 'Spring 6.1 rollout' active 13:50–14:30 · holds 4 members", a link to the change), vCenter events on its VMs / hosts ("vCenter: vMotion worker04 at 14:00") and its history: "Flared together 3 times in the week before · last Tue 14:02 · up to 4 of these problems" — the moments in the 7 days before the group started when two or more of its members went bad within 15 minutes of each other (sources, monitors, latency alarms, process lanes and hardware findings keep that history; rule alerts and services do not). You only see what is about members you can see. The link /?view=alerts&group=<id> (the daily digest's alert-group links) opens one group — expanded and outlined, or, when it is a suspected cause of an incident, that incident with the group shown in it.

How alerts are grouped (evidence)

• Admin → Alerts → Alert grouping chooses how problems become alert groups: Off (the default — every problem its own row), Legacy (the rules above), Evidence or Shadow (legacy on the page, the evidence engine run beside it and compared in the diagnostic). With Evidence, problems form a group only over a typed relation between them — starting at the same time is never enough: the same entity (one VM, one host, one Catalogue service), a reason pointer, placement (one runs on the other — VM on host, pod on node, database on instance), calls (the service call graph), a canvas "Depends on", coverage, vendor, a change on them, a hardware finding, a shared platform reached by three or more members (a datastore: four or more — many unrelated VMs share one), or the same specific symptom (one new error type, one alarm, one message — never just "latency" or "CPU"). Generic messages and alarm names make no group: a vCenter stock usage alarm ("Host CPU usage", "Virtual machine memory usage", "Datastore usage on disk", "Network usage" …), a message shorter than 25 characters, or one that is only a generic failure phrase once numbers and filler words are gone ("timeout", "connection refused", "HTTP 503 Service Unavailable from upstream" …) is never a shared symptom; a specific message needs three or more problems. Placement and shared platforms reach Placement hops = 2 layers down by default (service → pod → node/VM); 3 also reaches the host, but a 3-hop link alone then clears the 60 % threshold, so raise it only with care. Time only weakens a relation (full weight up to a third of Onset decay, 15 min by default, 0.6 at it, nothing at twice it); an edge counts at Edge threshold (60 %) or more.
• Every group card says joined because … — one line per relation that holds it ("checkout → calls → payments-db (90 %, seen 2 h ago)", "PDC covers dc01 (declared)", "api declares vendor Cloudflare (Catalogue)") — and is named after what joined it: "Change: deploy checkout 2.3", "Vendor: Cloudflare", "Large VM snapshot · worker04". Problems that started together but share no relation are not grouped; the page lists them under also started within N minutes — no relation found, with what was tried ("calls: neither is a service · coverage: PDC → VM dc01, not failing · vendor: no failing service declares cloudflare") so the missing declaration is visible.
• Coverage — "This source covers …". A source that watches something it is not itself (an Active Directory check on a domain controller, an OpenTelemetry or log source for one service, a status page for a vendor) is joined with the failing thing it covers. The infra sweep finds most of it by itself: an Active Directory DC source's host → its VM (the Databases page's matcher: pinned, address or DNS — never a bare name, never an ambiguous match), an OpenTelemetry / Graylog / Elasticsearch / Aria / syslog source's service, and a status page's vendor (its integration's Vendor field, else the page's address — www.cloudflarestatus.com → cloudflare). Anything else you declare on the source: metric source → This source covers — pick a Catalogue service or an asset (a VM, a host, a k8s node …) from the search, or type a vendor, then Save coverage. Only the flow's editors may declare it, and an asset must have been discovered by an integration the flow may use (another organisation's VM is refused). What the sweep found is listed read-only beside it; a declared entry wins over it. Sweep rows not seen again for 7 days are no longer read.
• Vendors — Catalogue → service → Ownership → Vendors. List the outside vendors a service depends on (cloudflare, aws, stripe … — up to 10). When a status page of one of them reports an incident and services that declare it fail within the window, they form one common cause named after the vendor ("Vendor: Cloudflare"); a failing service that does not declare it stays out (and the side note says so).
• Operator buttons. Not one event on a group card (the organisation's owners and admins) splits it at once and keeps those problems apart for Dissolved group stays apart for (24 h by default); These belong together on the side note joins problems the engine kept apart for the same time. Both are stored as grouping labels (audited alerts.group.label) and used to measure the engine (npm run alerts:replay).
• The automatic analysis may say "separate". The AI verdict on a group now also reads the group's evidence lines and may answer separate events (the members only started together) or name the members that are a separate event (separate_members; the rest stays one group). At 0.7 or more, on an evidence group, it is applied at once: the whole group is kept apart, or the named members are split off, for Dissolved group stays apart for. The page lists it — "AI: separate events — PDC, CDN Network Status · until 14:02", or on the remaining card "AI split off web from api, edge-proxy — separate events" — and the organisation's owners and admins can Undo it (the problems may group again on the next read; the verdict is not re-run unless the group's evidence changes). Admin → Alerts → The automatic analysis may split a group (on by default) turns the cut off: the AI then only says so. A "separate" answer that cut nothing (below 0.7, the switch off, or a legacy group) keeps the rule's class and names on the card and says so on its class line — "AI · AI thinks these are separate events — …" — the card is never relabelled "related". A viewer who does not see every problem a cut names sees none of it ("not grouped: kept apart for now"). The verdict runs once per evidence: its re-run gate keys on the group's relations and members, so a re-formed group is judged once, within the usual AI caps.
• Explain may propose a split. When the monitoring bot's answer finds several separate events among a group's members, its conclusion carries a proposal — "These look like separate events: checkout | payments, cart — proposed by AI (not applied)". Nothing changes until an owner or admin presses Apply for 24 h: the parts are kept apart for 24 hours (an operator label, audited, with Undo); the parts themselves may still group.
• Cost: the evidence fold is pure — its relations are read with at most four statements per organisation scope (shared by its viewers; each viewer's changes are read for them alone) (calls, changes, the canvas, coverage + vendors), cached 5 minutes; a coverage or vendor change drops the cache. The knobs (edge threshold, onset decay, hops, group size, hysteresis passes, "Show the evidence on the card") are in Admin → Alerts → Alert grouping with their effective value and source.

Explain — the monitoring bot on one row or one alert group

• Explain (beside Ack on a table row, the outlined button on an alert-group card, or e on a focused row) asks the monitoring bot to diagnose that problem right there: a drawer opens under it, shows the bot's tool calls as it makes them (resolve_entity("esxi142"), search_events(…)) and its text as it streams, then a short answer in three parts — Likely cause, Evidence, Next check (≤ 120 words) — with "<provider> · <model> · N tool calls · explained N min ago". An alert group is explained as a whole: the bot sees its class, title and why line, the first problem (a cascade's suspected origin) and every member, and is asked whether the class is right — and, for a cascade, whether the origin really is upstream of the rest.
• Who can use it: anyone who sees the alert on the page — the same visibility as the row itself (a problem or alert group you cannot see answers "not found"). The bot always reads stored data (open problems, events, changes, logs and metrics already collected); live reads of devices, databases and integrations (the bot's live tools) run only when you can edit the flow — a view-only user gets the stored-data answer and the drawer says "Stored data only — live reads need edit access to the flow."
• It uses your organisation's Assistant AI role (Organisation → AI models). With no provider assigned the drawer says so in one sentence and nothing is sent anywhere; a model without tool use answers from the evidence pack in one pass.
• An answer is cached for 10 minutes per problem (a new value or a new episode asks again; the drawer then says "cached"), and one explanation runs at a time per person. Rows without a flow — a vCenter alarm, a hardware finding, a catalogue service — are explained on the organisation scope of their integration (stored data; live reads only when you can edit that integration); a row Topologix cannot tie to any integration or owner says so and points you to where it lives. The wallboard and the Cards view have no Explain.

Kernel crashes (netconsole)

When a VM’s kernel panics, its console lines reach Topologix through netconsole and the Syslog receiver (see the Syslog receiver guide, Kernel console (netconsole), and the Server Vitals guide, Files collector). Each crash episode is stored, tied to the VM and turned into a Hardware row here:

• Which VM. The episode’s key (the IP from the file name, the banner’s host, or the sender’s address) is matched to a VM’s guest IPs, an ESXi host’s or a device’s address, then a VM / host / device name or DNS name, then a Server Vitals host, then the source’s host filter when it names one host. A key nothing matches still gets its finding, on an entity named after the key (netconsole:172.16.38.155) under your first hypervisor integration — never dropped. Without any hypervisor integration the crash shows as an ordinary state-change row of the syslog source instead (for 24 h), with the same Kernel trace.
• One alert group, the crash first. The three kinds take part in alert groups, and two hardware rows on the same entity are related even when the Infrastructure map does not know it (a synthetic netconsole: entity). A crash row’s “since” is the time of the panic itself (not when the finding was first written), so a panic at 14:06:51 and vCenter’s HA reset 40 s later make one alert group that opens with the crash: “Started together, no direction between them: saltmaster03 first, then 1 VM within 1 minute.” (one machine is never upstream of itself, so it is not a cascade).
• Kernel trace on the row opens the drawer under it: the VM, a class chip, when it started and “rebooted after N s” (or just “rebooted”, or “no reboot seen”); the AI verdict card — cause, triggered by, a confidence bar, a summary, the suspect module or symbol and the next check; the trace in monospace (≤ 200 lines) with a “N lines before the crash” divider after the lines that preceded it and the verdict’s evidence lines highlighted; the provenance line (file · key · by name / banner / sender · source); Copy (needs an https page); and Explain on the row.
• The verdict card has three other states, each with its stored sentence: unavailable (the organisation has no model in the analysis role), skipped (the hourly cap was reached) and error (the model call failed); with no verdict stored yet it says “Not judged yet”.
• The AI verdict. Once per episode, right after it is stored, one structured call to the organisation’s analysis AI role (Organisation → AI models) reads the kernel’s own lines — redacted with the organisation’s rule set on the AI path, at most 16 KB — and answers through a fixed form: a cause (kernel-panic, oops, bug, oom-kill, hardware-error, hung-task, soft-lockup, manual-sysrq, unexplained-reboot, other), triggered by (software, hardware, operator, unknown), a confidence, a ≤ 60-word summary, up to six evidence lines quoted verbatim, the suspect and a next check. The server keeps only evidence lines that really are in the trace. At most 20 verdicts per organisation per hour. The finding never waits for it: the verdict joins the finding’s sentence on the next collector pass (“Verdict: manual-sysrq · operator · 0.95 — …”), and Explain starts from it.
• The Catalogue. A service’s Logs section shows a Kernel crashes strip — its episodes of the last 90 days, at most 20, hidden when there are none; Open shows the same trace and verdict in a dialog (without Explain, which belongs to an Alerts row).
• The monitoring bot finds a crash by the VM’s name: its event search includes a kernel source (severity critical for a panic or machine check, else warning, titled with the panic line and the verdict’s cause).
• API: GET /api/kernel-episodes?entity=&class=&limit= (newest first, no lines), GET /api/kernel-episodes/:id (with the lines), GET /api/catalog/service/:key/kernel-episodes (90 days, ≤ 20) — all in your catalogue scope, 404 outside it. Episodes are kept for Kernel crash episodes: retention (90 days, SQLite 30) after they started.

Tiles, grouping, filters

• Six buckets — Errors · Warnings · Held · Flapping · Unknown / no data · Stale latches — each problem in exactly one: a stale latch first, then flapping (3 or more failures in the last hour, or a flap mute), then held (maintenance, acknowledgement, change window, a snoozed finding), then by severity. Under the headline, the counts are the filters ("19 errors · 26 warnings · 7 above baseline · 3 new error types · 2 stale latches"); Unknown is not among them — the footer says "N services without a measured input · M log sources still learning (first baseline after 3 h)" and links to it. The wallboard keeps the six tiles. The pill in the top bar counts errors · warnings only.
• Table (the default) or Cards — the toggle is remembered per browser. The table groups rows by node (a service or a hardware entity by itself): a node with two or more open problems is one line — API · 3 subjects: traceCount, p95Ms, avgMs, its worst severity, since the oldest — that opens into a row per problem. Columns: Subject · Flow › node · What · Since · Changed · Held, with Ack ▾ / Open / Hold… on each row; sort by clicking Subject, What (severity), Since or Changed. On a phone the table scrolls sideways inside its card.
• By severity (a section per tile), By kind (Services · Monitors · Hardware · Sources · Processes) or By flow — remembered per browser. A flow filter and a search narrow the page; the tiles follow the flow filter, the top-bar pill always counts everything.
• A card shows where it is (flow › node, the service, the hardware), since when, the hold (by whom, until when) and its reasons; a reason about another card on the page is a link to it.

Ack · Open · Hold

• Ack ▾ (30 min / 2 h / 8 h / until it recovers / with a note): an alert rule's card acknowledges its alert event; a monitor / source / latency / process / service card gets an acknowledgement hold on that subject; a latched breach is acknowledged; a hardware finding is snoozed (for the chosen time; "until it recovers" = one day). A service has no recovery signal, so "until it recovers" holds it for 24 hours. View-only access acknowledges without holding anything.
• Open jumps to the node on its flow (and the metric source row), the service on the Catalogue, or the finding on Hypervisors.
• Hold… opens the maintenance window dialog pre-scoped to the node, the service or the hardware entity.

Wallboard (a TV)

Wallboard on the page — or the link /?view=alerts&wall=1 (add &flow=<id> for one flow) — shows a strip of the live incidents at the top (name · the impact · how long · the suspected cause), the tiles and a compact table of the ten worst (then newest) error / warning / flapping rows, grouped by node, in large type, with no top bar and no buttons, a clock and "updated … ago". It refreshes every 30 s and on every live update, the browser tab title carries the counts ((2 · 5) Alerts — Topologix), and a reload stays on the wallboard. Esc or Exit leaves it.

API

GET /api/alerts/open?flowId=&fresh=1 — the open problems in your scope (a flow outside it → 404), counts per bucket and your flows; GET /api/alerts/open/counts — the counts only; POST /api/alerts/ack {subjectKey, durationSeconds? | untilRecovery?, message?} (a subject you cannot see → 404). The socket event alerts:dirty carries no data — it only says "refetch". Cost: at most 4 database statements for your scope (cached 60 s) and at most 8 for the list (cached 15 s, dropped when something changes); fresh=1 skips the caches at most once per 5 s. GET /api/alerts/incidents?flowId= — the alert groups of the same scope as groups (id, since, severity, size, kind, kindConfidence, title, headline, why, signature, platform, origin with its context, members; incidents carries the same array for one more client release) and rules.classes, the class table; at most 3 statements of its own (the Infrastructure map slice), cached 60 s per user and re-folded on every alerts:dirty. The monitoring bot's get_open_problems answer carries the same groups in short (groups: kind, title, why). POST /api/alerts/explain {subjectKey | incidentId, requestId?} (incidentId = an alert group's id) — Explain: 200 with the answer (markdown, provider, tool calls, tokens, cached); a problem or alert group you cannot see → 404; a row with no flow, or one already running for you → 409; no Assistant provider → 503 with the reason. Tool activity and partial text stream on the flow's socket room as bot:tool / bot:partial tagged with the requestId. Incidents: GET /api/alerts/incidents leads with impacts; GET /api/alerts/impacts?status=&from=&to=&limit= (history + summary), GET /api/alerts/impacts/:id (with its timeline), GET /api/alerts/impacts/:id/ai (the stored explanation and the automatic verdict), ack / resolve / dismiss / cause routes under it, POST /api/alerts/impacts/:id/classify and POST /api/alerts/incidents/:episodeId/classify (Analyse now: 202 queued, 409 running or already analysed on the same evidence, 429 at a cap or the pack reserve, 403 off / no provider), POST /api/alerts/explain {impactId}; GET/PUT /api/organizations/:id/impact, /impact-notify (+ POST …/impact-notify/test) and /incident-ai for owners and admins; GET/PUT /api/admin/alerts-settings for super admins. The deep link /?view=alerts&impact=<id> opens one incident, /?view=alerts&group=<id> one alert group.

Tuning (Admin → System → Background workers)

Keys

Floors and caps

Reminders and escalation ≥ 5 min; ≤ 100 reminders; digest windows 1–60 min; ≤ 20 channels per list; ≤ 14 on-duty ranges; N-of-M M ≤ 30; one alert per item ≤ 20 at once. An invalid schedule is dropped when saved (the rule is then always on duty — never silently silent).

Good to know

• Known gaps. One-alert-per-item alerts have no reminders (reminders are per rule and node), and after a restart every still-breaching item alerts once again.
• A rule without reminders re-alerts once after a server restart for a subject that is still failing (rules with reminders do not — their open alert is in the database).
• A flap mute that expires while the server is down sends no "Unmuted" notice.
• A parent-down hold needs the parent to alert: a parent node without alert rules holds its children only through a Hypervisors host-down finding. Infrastructure parents rely on VM / host nodes placed from the Hypervisors or infrastructure views.
• Expressions and composite conditions have no rollup history, so no charts and no seasonal baseline for derived values; a windowed trigger still judges its raw samples (a single implausible spike cannot fire a window, since every sample must breach).
• Recoveries are always delivered; only the maintenance mode Don't evaluate stops evaluation — polling itself is never paused.
• The Alerts page reads recorded state only: a monitor, source or process that has not changed state since state changes were first recorded is not listed until it next changes; outside on-duty hours holds (they have no window row) and no data marks are not shown as held / unknown; info-level hardware findings are left out.