Superseded 2026-09-19 — this article documented three threshold-tuning passes against a definition that was already known to be wrong

This alarm was tuned five times in five months (2026-04-15, 05-08, 07-16, 07-20, 09-07) before anyone actually fixed it. The 05-08 pass below correctly diagnosed the root cause in plain terms — "301 redirects are excluded from the definition of successful, and the definition itself is wrong for this workload" — and then lowered the alarm's thresholds instead of fixing the definition. 07-20 raised the evaluation gate instead. Both are real, working mitigations, and the diagnostic steps in §🔬 below are still the right way to confirm this failure mode. But the actual fix, found and deployed 2026-09-07, is one line:

-   lookup: sum -1m unaligned of success
+   lookup: sum -1m unaligned of success,redirect

The tell, if you hit this again on a different host: the same alarm coming back with a different trigger each time (a scraper, then GPTBot, then GPTBot again, then ShapBot) means the alarm's definition is wrong, not that the triggers are unusual. Reach for the definition, not another threshold. §✅ below now leads with this fix; the original threshold/gate approach is kept afterward as a fallback for sites where redirects genuinely aren't a normal, correct response (see §🧭 When NOT to apply this).

Tuning Netdata `web_log_1m_successful` for Redirect-Heavy WordPress Sites¶

🛑 Problem¶

Netdata's stock web_log_1m_successful alarm fires CRITICAL on a perfectly healthy WordPress site whenever a crawler hammers legacy URLs. Example email/notification:

[CRITICAL] web_log_1m_successful = 54.1%
Ratio of successful HTTP requests over the last minute (1xx, 2xx, 304, 401, 429)

Meanwhile the front page returns HTTP 200, no 5xx errors are logged, and only a handful of 4xx noise hits appear. So why the alert?

🔬 Root Cause¶

The metric counts as "successful" only the response code classes:

1xx, 2xx, 304, 401, 429

301 redirects are NOT counted as successful by the stock definition. They land in the redirect dimension and pull the success ratio down.

WordPress sites generate large volumes of 301s as a normal part of life:

Redirect source	Why a 301
`/?p=NNNN` legacy shortlinks	Canonical URL rewrite to slug
Stale post slugs after permalink edits	Old → new path
`/feed` → `/feed/`	Trailing-slash normalization
`http://` → `https://`	TLS upgrade
`domain.com` ↔ `www.domain.com`	Host canonicalization
Proxy CONNECT probes (e.g. `www.instagram.com:443`)	Apache returns 301 to canonical host

When a feed scraper or vulnerability crawler walks a long list of legacy /?p= URLs, every single hit is a 301. A short burst can push the stock ratio below the warn/crit floor within a single minute — even though the server is functioning perfectly and every one of those 301s is the correct response. A real 2026-09-07 burst on dcanalysts.net: 2,104 requests that day (~20% of all traffic) from ShapBot/0.1.0, peaking at 847 req/min, of which 504 were 301s and 343 were 2xx. Under the stock definition that's a 40.5% success ratio; the site never stopped working for a single one of those requests.

Verifying the cause¶

Pull the last few thousand lines of the access log and split by status code:

sudo tail -5000 /var/log/apache2/access.log | awk '{print $9}' | sort | uniq -c | sort -rn

If you see something like:

…the math is 196 / (196+162+5) ≈ 54%, which matches the alarm value almost exactly. The alert is correct by its (stock) definition; the definition is wrong for this workload.

Cross-check the source IPs:

sudo tail -2000 /var/log/apache2/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10

If a single IP or user-agent dominates (hundreds of requests in minutes) and most of its hits are 301 to legacy URLs, you have your culprit.

✅ Solution¶

Fix the alarm's definition first. If your site is genuinely redirect-heavy by design (legacy shortlinks, trailing-slash normalization, host canonicalization — anything where a 301 is the correct response, not a degraded one), count redirects as successful. If it isn't — see §🧭 When NOT to apply this — use the threshold/gate approach in §2b instead, and also consider blocking the abusive scraper (§4).

1. Fix it at the definition (preferred — verified live 2026-09-07)¶

Create /etc/netdata/health.d/web_log_override.conf (host-scoped; see the danger callout below on filename) with a complete copy of the web_log_1m_successful template, changing only the lookup line:

 template: web_log_1m_successful
       on: web_log.type_requests
    class: Workload
     type: Web Server
component: Web log
-   lookup: sum -1m unaligned of success
+   lookup: sum -1m unaligned of success,redirect
     calc: $this * 100 / $web_log_1m_requests
    units: %
    every: 10s
     warn: ($web_log_1m_requests > 500) ? ($this < (($status >= $WARNING ) ? ( 95 ) : ( 85 )) ) : ( 0 )
     crit: ($web_log_1m_requests > 500) ? ($this < (($status == $CRITICAL) ? ( 85 ) : ( 75 )) ) : ( 0 )
    delay: up 2m down 15m multiplier 1.5 max 1h
  summary: Web log successful
     info: Ratio of successful HTTP requests over the last minute (1xx, 2xx, 304, 401, 429, redirects)
       to: webmaster

Two things worth noticing about this diff versus the old threshold-tuning approach:

The warn/crit thresholds go back to stock (95/85, 85/75). Once redirects count as successful, a redirect-heavy site reads close to 100% under normal load — replaying the 2026-09-07 ShapBot burst through the new formula gives (504 + 343) / 847 = 100%, versus 40.5% under the old one. You no longer need a loosened floor to tolerate normal traffic, because the metric now actually reflects reality.
The request-count gate (> 500) is kept, from the 2026-07-20 pass. That part of the earlier tuning wasn't wrong — it's still useful for suppressing ratio noise at very low traffic — it just wasn't sufficient on its own, because the underlying definition was still counting a correct response as a failure.

Deploy via Ansible as a host-scoped template (MajorAnsible pattern: templates/health_web_log_<host>.conf.j2, one host or host-group per file, gated in the play so it only applies where the site is genuinely redirect-heavy by design) rather than a hand-created file — see §5.

Do NOT create /etc/netdata/health.d/web_log.conf

A file in health.d/ with the same name as a stock health file shadows that entire stock file, not just the templates you redefine. Creating web_log.conf containing only the block you want to tune silently deletes the other eight web_log alarms — including web_log_1m_internal_errors, the 5xx outage alarm. You end up blind to real outages while believing you only changed one thing.

Verified on Netdata v2.10.4 (dcaprod, 2026-07-20): stock yields 10 web_log alarms; a trimmed same-named web_log.conf cut that to 2. Re-verified 2026-09-07 after the definition fix: still 10 alarms, all distinct names — the file-shadowing trap did not recur.

Use a differently named file — /etc/netdata/health.d/web_log_override.conf (or, better, a host-scoped Ansible template — see §5). Files in health.d/ load after stock and override only same-named templates, leaving the siblings live. This also means a Netdata upgrade's new stock alarms still reach you, which a full-file copy would mask.

Then reload Netdata health:

sudo netdatacli reload-health

Confirm the new definition and thresholds are active:

curl -s http://localhost:19999/api/v1/alarms?all \
  | jq -r '.alarms | to_entries[] | select(.value.name == "web_log_1m_successful") | .value.warn,.value.crit'

Outage coverage is unchanged, all still at stock sensitivity: web_log_1m_internal_errors (5xx) and web_log_1m_bad_requests (4xx) are untouched by this change, and web_log_1m_redirects still fires on a genuine redirect loop — the one failure mode that folding redirects into "successful" doesn't hide, because that's a separate alarm watching the same dimension from the other direction.

2. If redirects aren't the whole story: threshold + gate tuning (superseded as the primary fix, still valid as a fallback)¶

This is the approach documented here from 2026-05-08 through 2026-07-20, before the definition fix existed. It works — it's just treating the symptom, and needs redoing on every version bump since stock thresholds vary by Netdata release. Use it on a host where §🧭 applies (redirects genuinely aren't a healthy-by-design pattern) but you still want more headroom than stock, without changing what counts as "successful."

Loosen the ratio floor:

    warn: ($web_log_1m_requests > 120) ? ($this < (($status >= $WARNING ) ? ( 95 ) : ( 85 )) ) : ( 0 )
    crit: ($web_log_1m_requests > 120) ? ($this < (($status == $CRITICAL) ? ( 85 ) : ( 75 )) ) : ( 0 )
+    warn: ($web_log_1m_requests > 120) ? ($this < (($status >= $WARNING ) ? ( 50 ) : ( 40 )) ) : ( 0 )
+    crit: ($web_log_1m_requests > 120) ? ($this < (($status == $CRITICAL) ? ( 30 ) : ( 20 )) ) : ( 0 )

Verify your own stock values before trusting any documented threshold

Stock thresholds vary by Netdata version. Read your own stock file first:

awk '/template: web_log_1m_successful/,/^$/' /usr/lib/netdata/conf.d/health.d/web_log.conf

2b. Raise the request-count gate above realistic scanner-burst volume¶

On a low-traffic host (a personal blog might average ~9 requests/minute), any scanner that fires 120+ hits in a single minute clears the default gate and evaluates the ratio. Raise the gate above realistic burst volume so these ratio alarms simply don't evaluate unless traffic is genuinely high:

-    warn: ($web_log_1m_requests > 120) ? ($this < (($status >= $WARNING ) ? ( 50 ) : ( 40 )) ) : ( 0 )
+    warn: ($web_log_1m_requests > 500) ? ($this < (($status >= $WARNING ) ? ( 60 ) : ( 50 )) ) : ( 0 )

Pick the gate from your own data:

curl -s "http://localhost:19999/api/v1/data?chart=web_log_apache.requests_by_type&after=-5400&points=90&group=sum&gtime=60&format=json" \
  | jq -r '.data[] | [.[0], (.[1:] | add)] | @tsv' | sort -k2 -rn | head

Set the gate comfortably above the burst peak (e.g. median 9, burst 195 → gate 500). Apply the same gate to the sibling web_log_1m_redirects alarm if it's noisy for the same reason. Leave bad_requests (4xx) and internal_errors (5xx) at stock — those stay your real outage signals regardless of which approach you take above.

3. Lean on the right alarms for real outages¶

Two other web_log alarms remain stock and are the correct outage signals, whichever approach you took above:

Alarm	Catches	Default thresholds
`web_log_1m_internal_errors`	5xx ratio	warn 2% / crit 5%
`web_log_1m_bad_requests`	4xx (excl. 401, 429)	warn 30%

Verify both are active and CLEAR after your change:

curl -s http://localhost:19999/api/v1/alarms?all \
  | jq -r '.alarms | to_entries[] | select(.value.name | test("web_log")) | "\(.value.status) | \(.value.name)"'

4. Block the abusive scraper (independent of which fix above you use)¶

Identify the dominant offender from step 1's IP list. For a one-off, ban it permanently via the recidive jail (assuming bantime = -1 is set in jail.local):

sudo fail2ban-client set recidive banip 74.7.242.61
sudo fail2ban-client status recidive

For a known bot user-agent (ShapBot, GPTBot, etc.) that will just come back on a different IP, block by user-agent/robots.txt instead — see the fleet's ai_crawler_policy Ansible role, which renders both the Apache deny list and robots.txt from a single ai_crawler_agents list so the two can't drift apart. Full detail: dcaprod § 2026-09-07, AI crawler policy runbook.

If bantime is finite on your host, edit /etc/fail2ban/jail.local:

[recidive]
bantime = -1
findtime = 86400
maxretry = 3

5. Make it survive a rebuild (config management)¶

A file dropped in /etc/netdata/health.d/ is wiped when you reprovision the host — codify it. The fleet's current pattern (2026-09-07 onward): a host-scoped Ansible template, templates/health_web_log_<host>.conf.j2, deployed only to the hosts where the tuning applies (netdata.yml --limit <host>) — not a fleet-wide file, since a busy webserver on a different host may legitimately need the stock definition and floor. This supersedes the two patterns below, which are kept here because the reasoning (why a full-file copy is dangerous) still matters if you're working outside Ansible:

Full-file copy — manage /etc/netdata/health.d/web_log.conf as a complete copy of the stock template with your edits. Simple, but it shadows the entire stock file, so you silently miss new stock alarms on a Netdata upgrade — and if the copy is ever trimmed to just the tuned blocks, every omitted alarm disappears, including the 5xx one. Not recommended; see the danger callout in §1.
Partial override file (manual, non-Ansible hosts) — ship only the retuned template: block to a separately named file such as /etc/netdata/health.d/web_log_override.conf. Same-named templates deterministically override stock while untouched siblings (bad_requests, internal_errors) keep coming from stock. Each override block must be a complete, valid template (every field), not just the changed lines, or Netdata will reject it.

🧪 Verification¶

After applying either fix:

# 1. Active alarms — should be empty (or only your real ones)
curl -s http://localhost:19999/api/v1/alarms?active | jq '.alarms'

# 2. Recidive ban list includes the IP, if you banned one
sudo fail2ban-client status recidive

# 3. Live ratio during a redirect burst — with the definition fix, should read close to 100%;
#    with threshold-only tuning, should stay above your new warn floor
watch -n 5 'curl -s http://localhost:19999/api/v1/data?chart=web_log_apache.requests_by_type\&after=-60\&points=1\&format=json | jq'

🧭 When NOT to apply the definition fix (§1)¶

If your site is an API or SPA backend that should have a 200-dominated traffic mix, a burst of 301s is worth knowing about — a redirect ratio spike there is more likely a misconfiguration or loop than normal traffic. Diagnose what's actually returning 301 rather than folding it into "successful."
If 5xx errors are climbing in tandem with the success-ratio drop, either fix here will mask a real outage if you stop watching the other alarms. Always check web_log_1m_internal_errors first, whichever approach you take.
Genuinely redirect-heavy-by-design sites (WordPress with legacy shortlinks, trailing-slash canonicalization, host canonicalization) are the case §1 is for.

📚 References¶

Netdata stock template: /usr/lib/netdata/conf.d/health.d/web_log.conf
Definition fix, host-scoped: MajorAnsible templates/health_web_log_<host>.conf.j2
Manual override (non-Ansible): /etc/netdata/health.d/web_log_override.conf — deliberately not web_log.conf, which would shadow the whole stock file (see §1 danger callout)
Netdata web_log Go module dimensions: success, redirect, bad, error, other
Full incident history and the "tuned five times in five months" lesson: dcaprod § 2026-09-07
Runbook: dcaprod-netdata-web-log-tuning-runbook
Related: Custom Fail2ban Jail: Apache Directory Scanning
Related: Tuning Netdata Web Log Alerts — general redirect-alarm walkthrough

Tuning Netdata web_log_1m_successful for Redirect-Heavy WordPress Sites¶