Netdata SELinux AVC Denial Monitoring¶
A custom charts.d plugin that tracks SELinux AVC denials over time via Netdata. Deployed on all Fedora boxes in the fleet where SELinux is Enforcing.
What It Does¶
The plugin runs ausearch -m avc every 60 seconds and reports the count of AVC denial events from the last 10 minutes. This gives a real-time chart in Netdata Cloud showing SELinux denial spikes — useful for catching misconfigurations after service changes or package updates.
Where It's Deployed¶
Step 1 below has the WRONG path — corrected 2026-07-26
The script must go in /usr/libexec/netdata/charts.d/, NOT /etc/netdata/charts.d/.
This article originally said to create the file under /etc/netdata/charts.d/. That
directory is only used by charts.d.plugin for .conf override files (e.g.
libreswan.conf) — confirmed via strace -f -e trace=openat,stat on netdata 2.10.4.
The actual .chart.sh scripts are read from /usr/libexec/netdata/charts.d/ (resolved
at runtime as plugins.d/../charts.d), the same directory the stock modules
(example.chart.sh, libreswan.chart.sh, opensips.chart.sh) ship in.
A script placed at the (wrong) /etc/netdata/charts.d/ path is silently ignored —
no error, no log line, charts.d.plugin just runs the stock modules, finds nothing else,
prints main: No charts to collect data from. to the plugin's own namespaced journal
(journalctl --namespace=netdata, NOT journalctl -u netdata — netdata's unit uses
LogNamespace=netdata), and the daemon permanently disables the plugin for the rest of
that process's life (PLUGINSD: ... does not generate useful output ... Will not start it
again - it is now disabled.). Only a full systemctl restart netdata re-attempts it —
this is why a live ps aux | grep charts.d.plugin right after a restart can look like the
plugin isn't spawning at all, when it actually ran, found nothing, and exited within
seconds. This is exactly why the 2026-07-20 fix below only looked successful (file existed
at the documented path, sudoers validated, systemctl restart netdata ran clean) but
produced zero actual chart data on majormail and majordiscord.
Root-caused 2026-07-26 by installing strace temporarily and tracing a manual
sudo -u netdata /usr/libexec/netdata/plugins.d/charts.d.plugin 1 run, comparing
openat/newfstatat calls against a host where the chart worked (majorlab — which,
it turns out, has the script correctly placed in /usr/libexec/netdata/charts.d/ in
addition to the /etc/netdata/charts.d/ copy this article told people to make; the
/etc/netdata/charts.d/ copy on majorlab/majorhome was always inert dead weight).
| Host | OS | SELinux | Chart Installed |
|---|---|---|---|
| majorhome | Fedora 44 | Enforcing | Yes (verified 2026-07-20) |
| majorlab | Fedora 44 | Enforcing | Yes (verified 2026-07-20) |
| majormail | Fedora 44 | Enforcing | Yes (fixed 2026-07-26 — correct path) |
| majordiscord | Fedora 44 | Enforcing | Yes (fixed 2026-07-26 — correct path) |
Ubuntu hosts (dca, teelia, tttpod, majortoot, majorlinux) do not run SELinux and do not have this chart.
Installation¶
1. Create the Chart Plugin¶
Create /usr/libexec/netdata/charts.d/selinux.chart.sh (not /etc/netdata/charts.d/ —
see the warning above):
mkdir -p /usr/libexec/netdata/charts.d
cat > /usr/libexec/netdata/charts.d/selinux.chart.sh << 'EOF'
# SELinux AVC denial counter for Netdata charts.d
selinux_update_every=60
selinux_priority=90000
selinux_check() {
which ausearch >/dev/null 2>&1 || return 1
return 0
}
selinux_create() {
cat <<CHART
CHART selinux.avc_denials '' 'SELinux AVC Denials (last 10 min)' 'denials' selinux '' line 90000 $selinux_update_every ''
DIMENSION denials '' absolute 1 1
CHART
return 0
}
selinux_update() {
local count
count=$(sudo /usr/bin/ausearch -m avc -if /var/log/audit/audit.log -ts recent 2>/dev/null | grep -c "type=AVC")
echo "BEGIN selinux.avc_denials $1"
echo "SET denials = ${count}"
echo "END"
return 0
}
EOF
chown root:netdata /usr/libexec/netdata/charts.d/selinux.chart.sh
chmod 750 /usr/libexec/netdata/charts.d/selinux.chart.sh
2. Grant Netdata Sudo Access to ausearch¶
ausearch requires root to read the audit log. Add a sudoers entry for the netdata user:
# Note: the deployed drop-in on majorhome/majorlab is named netdata-ausearch —
# match that name so you don't create a second, conflicting file.
echo 'netdata ALL=(root) NOPASSWD: /usr/bin/ausearch -m avc -if /var/log/audit/audit.log -ts recent' > /etc/sudoers.d/netdata-ausearch
chmod 440 /etc/sudoers.d/netdata-ausearch
visudo -c
The visudo -c validates syntax. If it reports errors, fix the file before proceeding — a broken sudoers file can lock out sudo entirely.
3. Restart Netdata¶
4. Verify¶
Check that the chart is collecting data — wait at least 60–120 seconds after the restart
(the collector's own update_every is 60s, and first_entry/last_entry only advances once a
real cycle has run; checking too soon looks identical to a broken deploy):
curl -s 'http://localhost:19999/api/v1/chart?chart=selinux.avc_denials' | python3 -c "
import sys, json
d = json.load(sys.stdin)
print(f'Chart: {d[\"id\"]}')
print(f'Update every: {d[\"update_every\"]}s')
print(f'Type: {d[\"chart_type\"]}')
print(f'first_entry: {d[\"first_entry\"]}, last_entry: {d[\"last_entry\"]}')
"
first_entry == last_entry after 2+ minutes means it collected exactly once then stalled —
investigate further rather than assuming it's fine. To confirm real ongoing collection, check
again a minute later and confirm last_entry advanced by ~60.
If the chart doesn't appear at all (Chart is not found), the two things to check, in order:
1. Script location — must be /usr/libexec/netdata/charts.d/<name>.chart.sh, not
/etc/netdata/charts.d/. A script in the wrong place is silently ignored (see warning above).
2. The real (namespaced) plugin log, not the systemd unit's own journal:
main: No charts to collect data from. (nothing was discovered) or a check()
function reports failure line naming your module.
If you need to trace exactly which paths the plugin is checking (e.g. a future similar mismatch),
install strace temporarily and run:
sudo -u netdata strace -f -e trace=openat,newfstatat -o /tmp/chartsd_strace.log \
/usr/libexec/netdata/plugins.d/charts.d.plugin 1
grep -i '<your-module-name>\|charts.d/' /tmp/chartsd_strace.log
strace again afterward (dnf remove strace / apt remove strace) — it's not a
standing dependency.
Known Side Effect: pam_systemd Log Noise¶
Because the netdata user calls sudo ausearch every 60 seconds, pam_systemd logs a warning each time:
This is cosmetic. The sudo command succeeds — pam_systemd just can't find a D-Bus user session for the netdata service account, which is expected. The message volume scales with the collection interval (1,440/day at 60-second intervals).
To suppress it, the system-auth PAM config on Fedora already marks pam_systemd.so as -session optional (the - prefix means "don't fail if the module errors"). The messages are informational log noise, not actual failures. No PAM changes are needed.
If the log volume is a concern for log analysis or monitoring, filter it at the journald level:
# /etc/rsyslog.d/suppress-pam-systemd.conf
:msg, contains, "pam_systemd(sudo:session): Failed to check" stop
Or in Netdata's log alert config, exclude the pattern from any log-based alerts.
Fleet Audit¶
To verify the chart is deployed and functioning on all Fedora hosts (checks both existence
and that last_entry is recent, not just a stale single collection):
for host in majorhome majorlab majormail majordiscord; do
echo -n "=== $host: "
ssh root@$host "curl -s 'http://localhost:19999/api/v1/chart?chart=selinux.avc_denials' 2>/dev/null | python3 -c 'import sys,json,time; d=json.load(sys.stdin); age=time.time()-d[\"last_entry\"]; print(d[\"id\"], \"last_entry age:\", int(age), \"s\")' 2>/dev/null || echo 'NOT FOUND'"
done
Confirmed working on all four hosts as of 2026-07-26 (majormail/majordiscord fixed after the
path correction above; script also verified present at /usr/libexec/netdata/charts.d/ on
majorlab/majorhome, with the old inert /etc/netdata/charts.d/ copies removed).