Skip to content

Verify a Health/Alarm Matcher by Counting Bound Instances, Not by Reading Config

The Problem

A monitoring matcher (a Netdata health config alarm: block, a Grafana/Loki alert rule label selector, a Docker healthcheck matcher, etc.) can be syntactically valid, deployed, and "active" by every static check — and still be watching zero real instances. Reading the config file, running a linter, or seeing changed=0 on a config-management run all say "this is fine." None of them tell you whether the matcher's selector actually binds to anything live.

Symptom: An alert that should have fired during a real incident never does. When you go looking afterward, the config looks completely correct — right syntax, right file, right service restarted after the change. Nothing in the deploy log or the file diff explains why it was silent.

Why It Happens

Static checks (config syntax validation, Ansible changed=false/idempotency, "the service reloaded without error") only confirm the matcher parses. They say nothing about whether its selector — a unit name pattern, a label match, a container name glob — actually matches any of the instances running on the host right now. A matcher can be perfectly well-formed and still bind to nothing if:

  • The thing it's supposed to match was renamed, restructured, or never matched the pattern in the first place (e.g. a systemd unit suffix that changed between OS versions).
  • The label/selector was copied from a template or another host and never adjusted for this host's actual unit/container names.
  • The matcher was written against a planned state (what the service names were expected to be) rather than the deployed state.

Two confirmed occurrences on MajorInfrastructure, both invisible to config review:

  1. Container-Alerting-Gaps — Grafana alert rules for docker_container_down and docker_container_unhealthy were deployed with label selectors that matched zero running container instances on the target hosts. The rules were valid YAML, provisioned without error, and showed as "active" in the Grafana UI — but had nothing to alert on.
  2. netdata-updater matcher — a Netdata health alarm used a .service-suffixed unit name to match the updater's systemd unit. The suffix didn't match how the unit was actually registered on the host: the matcher bound to 0 of 102 eligible units fleet-wide. The alarm config deployed cleanly on every run and never once logged an error.
  3. tttpod fail2ban jail — a jail had been enabled and configured since provisioning, but the log file it watched had sat at 0 bytes the entire time. fail2ban-client status showed enabled = true, no error anywhere — the jail was simply never fed a single line to match.
  4. recidive backend mismatch, 7 of 9 fleet hosts (2026-08-19) — the recidive jail's [DEFAULT] backend = systemd pointed it at the journal, while every jail's actual Ban lines were written to /var/log/fail2ban.log (logtarget). Only the two hosts already on backend = polling worked. The tell wasn't a zero — fail2ban-client status recidive read Total banned: 1, which looks alive at a glance. What actually exposed it was comparing two sources directly: 218 Ban lines in the file log vs. 0 in the journal over the same 24h window. Fixed by pinning backend = polling + explicit logpath as a per-jail override on all 9 hosts; live-verified afterward (not from the diff) — filter counts went from 0 to real per-host numbers (majorlinux 103, dcaprod 37, teelia 32, tttpod 26, majortoot 10), and permabans that should have already existed landed immediately.

All four share the same failure shape: the deploy pipeline reported success because success, for a config-management tool, means "the file is correctly in place" — not "the thing the file describes is doing anything." The recidive case adds a variant worth flagging on its own: an implausibly low live count (1) is just as much a red flag as a hard zero — it still looks like the monitor is working, right up until you compare it against what should be there."

The Fix

After deploying or changing any matcher, verify it by counting how many live instances it is currently bound to, using the monitoring tool's own introspection — not by re-reading the config you just wrote.

Netdata

# List alarms currently bound on a host, with the instance/chart they're attached to
curl -s http://localhost:19999/api/v1/alarms?all | python3 -c \
  "import json,sys; d=json.load(sys.stdin); print(len(d.get('alarms', {})))"

# Or, for a specific alarm name, confirm it's attached to the expected number of
# instances/units across the fleet rather than just checking one host
ansible all -m shell -a \
  "curl -s http://localhost:19999/api/v1/alarms?all | grep -c '\"name\":\"<alarm_name>\"'"

A count of 0 (or far fewer than the number of eligible units/containers) means the matcher isn't bound, even though the health config file deployed without error.

Grafana / Loki alert rules

# Check the rule's evaluation state and how many series/instances it's currently
# evaluating against — not just that the rule exists
curl -s -u "$GRAFANA_USER:$GRAFANA_PASS" \
  http://localhost:3000/api/v1/provisioning/alert-rules | \
  python3 -c "import json,sys; [print(r['title'], r['uid']) for r in json.load(sys.stdin)]"

# Then check the rule's live evaluation result (alerting/normal/no data) —
# "no data" on a rule that should have live series is the same failure class
# as the Netdata 0-instance case

A rule stuck permanently in "No Data" for a matcher that should have eligible targets is the Grafana-side version of the same problem — the rule is valid but bound to nothing.

Docker container health matchers

# Confirm the matcher's name/label pattern actually matches running containers,
# don't just confirm the healthcheck stanza exists in docker-compose.yml
docker ps --filter "name=<pattern>" --format '{{.Names}}: {{.Status}}' | wc -l

fail2ban jails

# A jail showing enabled=true proves nothing about whether it's matching real
# log lines. Compare the jail's own ban count against the raw log directly.
fail2ban-client status <jail>

# Cross-check against the actual log/journal source the jail's backend reads from
grep -c 'Ban ' /var/log/fail2ban.log          # if backend = polling / logpath
journalctl -u fail2ban --since -24h | grep -c 'Ban '   # if backend = systemd

If the jail's backend/logpath points at journald but bans are actually landing in the file log (or vice versa), the jail matches nothing — and Total banned: 1 can look alive at a glance while the real count sits in the hundreds. A 0 or implausibly low count from fail2ban-client status, compared against a much higher raw count in the log source the jail should be reading, is the same failure signature as a Netdata alarm bound to nothing.

When to Apply This Check

Run a live bound-instance count any time you:

  • Deploy a new alerting rule, health check, or Netdata alarm config.
  • Rename a systemd unit, container, or service that an existing matcher targets.
  • Copy a matcher from one host/role to another (label selectors and unit-name patterns rarely transfer cleanly — verify on the new target, don't assume the template was right).
  • See an Ansible run report changed=0 on a monitoring role — that's evidence the file matches what's already deployed, not evidence the thing being monitored still exists in the form the matcher expects.

Takeaway

"The config is correct and deployed" and "the thing it's monitoring is actually being watched" are two different claims, and only the second one matters during a real incident. Config review, linting, and idempotent changed=0 deploys all verify the first claim. Only a live count against the monitoring tool's own API or CLI verifies the second. Treat a 0-instance (or unexpectedly low) bound count as a hard failure requiring investigation, the same way you'd treat a syntax error — it's just a failure that static tooling can't see.

Added 2026-08-23 (Memory-GC promotion, two occurrences in one week) — a sibling failure mode to the bound-instance check above. Both are "the check passed but proved nothing" failures; this one is about what the check ran against rather than what it counted.

The rule: When verifying a fix, exercise the real artifact in its real environment. A re-implemented check can disagree with the thing it models, and a fresh clone or copy may not contain the change under test — a green or red result from a stand-in proves nothing about the original.

Two occurrences in the same week:

  1. 2026-08-20 — journal.py regex test. A verification script re-implemented the section parser's regex separately rather than importing and calling the real journal.py function. The re-implementation passed; it was subtly different from the actual code path Marcus's sessions run.
  2. 2026-08-22 — MajorDocs pre-commit hook test. A hook fix was verified by testing against a fresh git clone rather than the working repo the hook actually runs in — the clone didn't carry the local state the bug depended on, so the test came back clean regardless of whether the fix worked.

Both share the shape of the bound-instance problem above: the check ran cleanly and reported success, but the thing that mattered — does the real code, in its real environment, actually do the right thing — was never touched. Config validity isn't behavior; a passing re-implementation or a clean-room clone isn't the artifact under test.

See Also