Skip to content

Grafana Says RESOLVED but Nothing Was Fixed: Stale Alerts After a DatasourceError

A Grafana alert that has been firing for days sends a [RESOLVED] email overnight. Nobody patched anything. The problem it was reporting is still there.

What actually happened is that the rule stopped being able to evaluate, and Grafana cleaned up the alerts it could no longer confirm. The all-clear is a side effect of the check breaking, and it arrives looking exactly like a real one.

Symptoms

  • One [FIRING:1] DatasourceError email for the rule, with no matching resolve.
  • Roughly one evaluation interval later, a [RESOLVED] email for every instance the rule had firing.
  • No change was made to the thing being monitored.
  • The data source itself looks healthy: low CPU, no errors of its own, other rules on the same data source evaluating normally.

If you only read the second email, you conclude the problem went away.

Why Grafana resolves alerts it can't check

A Grafana alert rule produces one alert instance per series its query returns. Each evaluation, the state manager matches the returned series against the instances it already has.

When the query fails outright, the rule goes to its execErrState. With execErrState: Error, Grafana raises a separate DatasourceError alert. That part is correct and useful.

The existing instances, though, are not refreshed by a failed evaluation. After they go unseen for long enough they are treated as stale: Grafana drops them and sends a resolved notification. The log line is explicit about it:

logger=ngalert.state.manager rule_uid=<uid> level=info msg="Detected stale state entry" state=Alerting

So the sequence is: evaluation fails, DatasourceError fires, the real alerts age out, [RESOLVED] goes out. This is not specific to any one rule or data source.

Confirm it

Look in Grafana's own log for the rule, not at the data source:

docker logs --since 24h grafana 2>&1 | grep 'Failed to evaluate rule' \
  | grep -o 'rule_uid=[a-z0-9-]*' | sort | uniq -c

A rule that appears here on every interval is blind. The error text tells you why. In the case that prompted this article:

error="... failed to execute query [A]: Get \"http://loki:3100/loki/api/v1/query_range?...
  &step=62837ms\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

Then check whether the instances were dropped rather than cleared:

docker logs --since 24h grafana 2>&1 | grep 'Detected stale state entry' | grep 'state=Alerting'

If both are present, treat every [RESOLVED] from that rule since the first failure as unverified.

The cause in this case: a range query doing an instant query's job

The rule answered "what did the latest weekly scan report?" with LogQL like this:

last_over_time({syslog_identifier="scan-job"} | logfmt | unwrap critical_count [8d]) by (host, image)

The [8d] lookback is needed, because the scan only runs weekly. The mistake was in how the rule ran it:

relativeTimeRange: { from: 691200, to: 0 }   # 8 days
model:
  queryType: range

A range query evaluates the expression at every step across the window. Over 8 days Grafana chose a step of about 63 seconds, which is roughly 11,000 evaluation points, each one looking back 8 days. A downstream reduce: last then threw away all but the final point.

It worked for a few days and then started timing out on every evaluation. Loki was idle throughout; the stream held under 200 lines.

Asking Loki directly made the difference obvious:

Query Result
Instant, [8d] lookback answered immediately, same series
Range over the last 10 minutes answered immediately, same series
Range over 8 days (what the rule ran) no answer after 120 seconds

Fix

Keep the lookback in the expression and evaluate it once:

relativeTimeRange: { from: 600, to: 0 }
model:
  queryType: instant
  expr: 'last_over_time({syslog_identifier="scan-job"} | logfmt | unwrap critical_count [8d]) by (host, image)'

An instant query evaluates at the to end of relativeTimeRange. That matters for week-over-week rules that compare two windows: to read "the week before", move to, not the lookback.

# this week
relativeTimeRange: { from: 600, to: 0 }
# previous week: evaluated 8 days ago, looking back 8 days from there
relativeTimeRange: { from: 691800, to: 691200 }

Before deploying, run both forms against the data source and compare the instant value with the last point of the range result. They should match.

After the change, the next evaluation should complete in seconds. Expect two emails: a resolve for the DatasourceError, and the original alerts firing again. The second one is the correct state returning, not a new incident.

A second way to get there: duplicate series from a label that comes and goes

Same rule, different failure, a month later. The week-over-week Lynis rule fired DatasourceError on two hosts in one night, and neither host's score had changed.

Query A (this week) returned two series for the same host: one with a unit="lynis-audit.service" label and one without. Query C (last week) returned one. Grafana's math step ($B - $D) can't pair two left-hand series with one right-hand series, the expression errors, and a rule with exec_err_state: Error turns that into DatasourceError.

Why the label flickered: the score line is written by logger from a short-lived script. Whether journald records _SYSTEMD_UNIT for that line is a race against the process exiting, so some weeks the line carries the unit label and some weeks it doesn't. Alloy turns the field into a Loki label, so each variant is its own stream. Any week a host's variant differs from the previous week's, the 8-day window holds both.

Confirm it by running query A by hand at the failing evaluation time and counting series per host:

max_over_time({syslog_identifier="lynis-audit"} | logfmt | unwrap hardening_index [8d])

Two results for one host, differing only by unit, is this bug.

Fix: aggregate away every label you don't compare on, in both queries:

max by (host) (max_over_time({syslog_identifier="lynis-audit"} | logfmt | unwrap hardening_index [8d]))

Replay both queries at the failing times before deploying: exactly one series per host, same values as before.

Takeaways

  • A resolve is only as trustworthy as the evaluation behind it. A [RESOLVED] that follows an unresolved DatasourceError on the same rule is not evidence of anything.
  • Don't route DatasourceError somewhere quieter than the alerts it protects. It was the only true signal here, and it was one email against a dozen resolves.
  • If a rule reduces a range query with last, it probably wants an instant query. The [range] inside a LogQL or PromQL *_over_time function already supplies the history.
  • A rule that evaluates rarely fails rarely. Hourly or weekly rules can be broken for a long time before anyone connects the dots, so check Failed to evaluate rule counts when reviewing alert mail, not just the mail itself.
  • Comparing two queries? Aggregate to the labels you compare on. Any label that can appear or vanish between evaluations (journald's unit, a pod name, a version tag) splits one host into two series, and the math step errors instead of comparing. max by (host) costs nothing and removes the whole class.