LogQL Returns Almost Nothing: Anchored Regex, Not a Broken Pipeline¶
Symptom¶
A severity-scoped query across a 7.5-hour window over nine hosts returns one result:
Meanwhile you can see, by hand on the box, a service emitting warnings every single minute for that entire window. The obvious conclusion — the log pipeline is dropping severity, the level label isn't populated, the relabel rule is broken — is wrong.
The cause: =~ is fully anchored¶
In both LogQL and PromQL, a regex label matcher is implicitly wrapped in ^(?:...)$. There is no substring matching.
{level=~"warn"} does NOT match level="warning"
{level=~"critical"} does NOT match level="crit"
{level=~"err"} does NOT match level="error"
The query above therefore only ever matched the literal value error. Every warning and crit line in the window was silently excluded — silently, because a matcher that matches nothing is not an error. It returns success with fewer rows.
Same window, only the matcher corrected:
| Query | Streams |
|---|---|
{level=~"error\|critical\|warn"} |
1 |
{level=~"error\|crit\|warning"} |
9 |
A service whose warnings supposedly "never reached Loki" returned 200 matches once the values were spelled correctly.
journald priority keywords are not the words you expect¶
If severity comes from journald via Alloy's __journal_priority_keyword, the values are syslog keywords, which are abbreviated in ways that do not match casual English:
| Priority | Keyword |
|---|---|
| 0 | emerg |
| 1 | alert |
| 2 | crit |
| 3 | err (Alloy/Grafana commonly normalise to error) |
| 4 | warning |
| 5 | notice |
| 6 | info |
| 7 | debug |
Note the three that bite: crit not critical, warning not warn, and err/error depending on your pipeline. Never guess — enumerate.
Check the label's real values first¶
Before concluding anything about ingestion, ask Loki what values actually exist:
curl -s http://localhost:3100/loki/api/v1/label/level/values
# {"status":"success","data":["crit","debug","error","info","notice","warning"]}
Scope it to a window, since label values are time-bounded and a value absent today may exist during the period you care about:
S=$(( $(date +%s) - 86400 ))000000000; E=$(date +%s)000000000
curl -sG http://localhost:3100/loki/api/v1/label/level/values \
--data-urlencode "start=$S" --data-urlencode "end=$E"
And list the labels themselves if you are unsure one exists at all:
Thirty seconds of this would have prevented the misdiagnosis entirely.
Writing severity matchers that hold up¶
# Explicit, anchored-safe — preferred
{level=~"error|crit|warning"}
# If your pipeline may emit either form
{level=~"err|error|crit|critical|warning|warn"}
# Substring semantics, when you genuinely want them: anchor it yourself
{level=~".*warn.*"}
The last form matches warn and warning both, but it is a blunt instrument — it will also match a hypothetical no-warnings value. Enumerating the real values is safer.
The same rule governs every regex matcher, not just level:
{unit=~"nginx"} # misses nginx.service
{unit=~"nginx.*"} # matches
{host=~"major"} # misses majorhome, majorlab, majorlinux
{host=~"major.*"} # matches
Note also that . is a regex metacharacter: {unit=~"nginx.service"} happens to work but would equally match nginxXservice. Escape it — nginx\\.service — when precision matters.
Confirm the pipeline separately before blaming it¶
If a query still looks wrong after the matcher is verified, prove ingestion independently rather than inferring it:
# What Loki holds for a host, no severity filter at all
curl -sG http://localhost:3100/loki/api/v1/query_range \
--data-urlencode 'query={host="hostA"}' \
--data-urlencode "start=$S" --data-urlencode "end=$E" --data-urlencode 'limit=10'
# Compare against the source of truth on the box
ssh hostA 'journalctl --since "..." --until "..." -p warning --no-pager | wc -l'
When the two agree, ingestion is fine and the fault is in the query. In the incident that produced this article, Loki reported 0 warn+ entries for a host over a window where journalctl -p warning also returned "No entries" — agreement that simultaneously exonerated the pipeline and disproved the "label is unpopulated" theory.
Sanity check: is the count plausible?¶
A severity query returning 1 result across nine hosts and 7.5 hours should trigger suspicion on its own. Real fleets are noisier than that. Treat an implausibly quiet result as a query bug until proven otherwise, exactly as you would treat an implausibly loud one.
The general form of the mistake:
An empty or near-empty result is far more often a bad matcher than a broken pipeline.
Broken ingestion tends to fail loudly and completely — no streams for a host at any level, a dead agent, a dropped target. Selective silence that spares exactly one value is the fingerprint of a matcher problem.
Search your own notes before filing a defect¶
The misdiagnosis behind this article was filed as a fleet-wide defect against a working system. The project's own documentation already recorded, a month earlier, that the relabel worked — and listed the exact valid values that would have shown the regex was wrong. The evidence was in the same file the false task got written into.
If you keep a knowledge base, query it before declaring something broken. The cheapest debugging step is usually reading what you already wrote down.