Skip to content

Netdata → n8n Enriched Alert Emails

Status: Retired 2026-10-10 — kept as reference (was live across the fleet from 2026-03-21)

Retired 2026-10-10 — this pipeline is switched off

Netdata now sends only its plain native alert email. SEND_CUSTOM is "NO" on every agent host and the Netdata Enriched Alerts n8n workflow is deactivated, because the enriched mail duplicated the plain mail (55 of 68 enriched mails had a plain twin within 10 minutes) and the plain format is the one kept. Everything below describes how it worked and how to bring it back.

To re-enable: set SEND_CUSTOM="YES" in the Netdata notification config on the hosts (deployed via Ansible, which restarts the agent), then reactivate the workflow in n8n (a CLI change needs an n8n restart to take effect).

Replaces Netdata's plain-text alert emails with rich HTML emails that include a plain-English explanation, a suggested remediation command, and a direct link to the relevant MajorDocs article.

How It Works

Netdata alarm fires
  → custom_sender() in health_alarm_notify.conf
    → POST JSON payload to n8n webhook
      → Code node enriches with suggestion + wiki link
        → Send Email node sends HTML email via SMTP
          → Respond node returns 200 OK

n8n Workflow

Name: Netdata Enriched Alerts
URL: https://n8n.majorshouse.com
Webhook endpoint: POST https://n8n.majorshouse.com/webhook/netdata-alert
Workflow ID: a1b2c3d4-aaaa-bbbb-cccc-000000000001

Nodes

  1. Netdata Webhook — receives POST from Netdata's custom_sender()
  2. Enrich Alert — Code node; matches alarm/chart/family to enrichment table, builds HTML email body in $json.emailBody
  3. Send Enriched Email — sends via SMTP port 465 (SMTP account 2), from netdata@majorshouse.com to marcus@majorshouse.com
  4. Respond OK — returns ok with HTTP 200 to Netdata

Enrichment Keys

The Code node matches each key against the alarm, chart, or family field, case-insensitively and on token boundaries (matchesKey()), so net matches net_packets but not netdata or cabinet. First match wins, so specific keys still come before generic ones:

Key Title Wiki Article
reboot Reboot Required ansible-unattended-upgrades-fleet (majorlab: docker-caddy-selinux-post-reboot-recovery)
selinux SELinux Denial Alert netdata-selinux-avc-chart
disk_space Disk Space Alert diagnosing-disk-space-alerts
ram Memory Alert managing-linux-services-systemd-ansible
cpu CPU Alert managing-linux-services-systemd-ansible
load Load Average Alert managing-linux-services-systemd-ansible
net Network Alert tailscale-homelab-remote-access
docker Docker Container Alert debugging-broken-docker-containers
web_log Web Log Alert tuning-netdata-web-log-alerts
health Docker Health Alarm netdata-docker-health-alarm-tuning
mdstat RAID Array Alert mdadm-usb-hub-disconnect-recovery
systemd Systemd Service Alert managing-linux-services-systemd-ansible
(no match) Server Alert netdata-new-server-setup

The reboot key catches Netdata's system_post_update_reboot_status alarm ("System requires reboot after package updates"). It fires before a reboot, not after one, so its suggestion is a plan: see which packages need it (sudo dnf needs-restarting -r on Fedora, cat /var/run/reboot-required.pkgs on Debian/Ubuntu), check for in-flight work, then ansible-playbook reboot.yml -l <host>. Before this key existed (added 2026-09-23), the alarm fell through to the generic Server Alert text.

The selinux key covers the Fedora hosts' AVC denial alarm, which was the most common alert in the workflow's history (25 of 74 runs, 2026-09-11 to 09-24) and had previously fallen through to the generic Server Alert. Its first command groups denials by permission, process and SELinux domain straight from /var/log/audit/audit.log. It deliberately uses grep rather than ausearch, which has returned no matches on majorlab while the log held denials. The systemd key pulls the real unit name out of the chart id (…unit_restic-backup_service_state → restic-backup.service), so the email says systemctl status restic-backup.service instead of a placeholder.

Suggestion layout

Single-line suggestions are a plain suggestion string. Multi-step ones declare intro, steps (each an optional text and an optional cmd) and note, and render as a numbered list with each command in its own monospace block, so long commands wrap inside the block instead of mid-sentence. reboot, selinux and systemd use this. Every inline <code> in the email also gets an inline monospace style, because many mail clients drop <style> blocks. The returned suggestion field stays a flat string for the tests and any downstream use.

Test against real alerts, not just made-up ones. roles/n8n_alerts/files/test_enrich_alert.js runs before every deploy. Cases built from real payloads caught what synthetic ones missed. The workflow's history can be decoded read-only inside the container: execution_data is stored in flatted format, and n8n bundles the flatted package.

Host names: Netdata FQDN vs inventory name

Netdata reports each host's own hostname -f, which is not always the Ansible inventory name — majorlab.majorshouse.com, home.majorshouse.com (majorhome), temptedparadise.com (teelia), majorpi.majorshouse.com. The Code node maps these to inventory names (invHost) before any per-host check. Without that map, a check like dockerHosts.includes(hostname) silently never matches, and the host quietly gets the generic advice instead. That happened here until 2026-09-23. When adding a host-specific branch, key it on invHost, and add the host to the alias map if its FQDN differs.

Netdata Configuration

Config File Locations

Server Path
majorhome, majormail, majordiscord, tttpod, teelia /etc/netdata/health_alarm_notify.conf
majorlinux, majortoot, dca /usr/lib/netdata/conf.d/health_alarm_notify.conf

Required Settings

DEFAULT_RECIPIENT_CUSTOM="n8n"
role_recipients_custom[sysadmin]="${DEFAULT_RECIPIENT_CUSTOM}"

custom_sender() Function

custom_sender() {
    local to="${1}"
    local payload
    payload=$(jq -n \
        --arg hostname "${host}" \
        --arg alarm "${name}" \
        --arg chart "${chart}" \
        --arg family "${family}" \
        --arg status "${status}" \
        --arg old_status "${old_status}" \
        --arg value "${value_string}" \
        --arg units "${units}" \
        --arg info "${info}" \
        --arg alert_url "${goto_url}" \
        --arg severity "${severity}" \
        --arg raised_for "${raised_for}" \
        --arg total_warnings "${total_warnings}" \
        --arg total_critical "${total_critical}" \
        '{hostname:$hostname,alarm:$alarm,chart:$chart,family:$family,status:$status,old_status:$old_status,value:$value,units:$units,info:$info,alert_url:$alert_url,severity:$severity,raised_for:$raised_for,total_warnings:$total_warnings,total_critical:$total_critical}')
    local httpcode
    httpcode=$(docurl -s -o /dev/null -w "%{http_code}" \
        -X POST \
        -H "Content-Type: application/json" \
        -d "${payload}" \
        "https://n8n.majorshouse.com/webhook/netdata-alert")
    if [ "${httpcode}" = "200" ]; then
        info "sent enriched notification to n8n for ${status} of ${host}.${name}"
        sent=$((sent + 1))
    else
        error "failed to send notification to n8n, HTTP code: ${httpcode}"
    fi
}

jq required

The custom_sender() function requires jq to be installed. Verify with which jq on each server.

Deploying to a New Server

# 1. Find the config file
find /etc/netdata /usr/lib/netdata -name health_alarm_notify.conf 2>/dev/null

# 2. Edit it — add the two lines and the custom_sender() function above

# 3. Test connectivity from the server
curl -s -o /dev/null -w "%{http_code}" \
  -X POST https://n8n.majorshouse.com/webhook/netdata-alert \
  -H "Content-Type: application/json" \
  -d '{"hostname":"test","alarm":"disk_space._","status":"WARNING"}'
# Expected: 200

# 4. Restart Netdata
systemctl restart netdata

# 5. Send a test alarm
/usr/libexec/netdata/plugins.d/alarm-notify.sh test custom

Troubleshooting

Emails not arriving — check n8n execution log:
Go to https://n8n.majorshouse.com → open "Netdata Enriched Alerts" → Executions tab. Look for error status entries.

Email body empty:
The Send Email node's HTML field must be ={{ $json.emailBody }}. Shell variable expansion can silently strip $json if the workflow is patched via inline SSH commands — always use a Python script file.

000 curl response from a server:
Usually a timeout, not a DNS or connection failure. Re-test with --max-time 30.

custom_sender() syntax error in Netdata logs:
Bash heredocs don't work inside sourced config files. Use jq -n --arg ... as shown above — no heredocs.

n8n N8N_TRUST_PROXY must be set:
Without N8N_TRUST_PROXY=true in the Docker environment, Caddy's X-Forwarded-For header causes n8n's rate limiter to abort requests before parsing the body. Set in /opt/n8n/compose.yml.