Skip to content

Fail2Ban Digest Mode — Fleet-Wide Quiet Alerts

Now actually deployed fleet-wide — fixed 2026-07-26

The 2026-07-20 audit below was right: the digest script existed on only 3 of 9 hosts (teelia, majorlab, majorhome), and it was never actually Ansible-managed despite its own file header claiming "Deployed by Ansible" — the fail2ban role (a parity-capture design, see ansible-flat-playbooks-to-roles) had no task or captured file for it at all. It had been hand-installed on those 3 hosts outside any playbook.

Fixed 2026-07-26. Added two real tasks to roles/fail2ban/tasks/main.yml (appended after the existing parity-capture config deploy): a ansible.builtin.template task deploying templates/fail2ban-digest.sh.j2 to /usr/local/bin/fail2ban-digest.sh (mode 0755), and an ansible.builtin.cron task scheduling it at 08:00 local daily. There is no separate configure_fail2ban_digest.yml playbook, and no Python jail.local editor — both described lower in this article never existed for the real deployment; the [recidive] action_mwl / silent-default jail configuration is handled entirely by the existing parity-capture config sync, not by digest-specific tooling. See the corrected "Ansible Deployment" section below.

Verified before touching anything: the new template produces byte-identical output to the 3 already-installed scripts (ansible-playbook fail2ban.yml --limit majorlab,majorhome,teelia --check --diff → changed=0). Deployed host-by-host with --check --diff before each apply: dcaprod, majorlinux, tttpod, majortoot, majormail, majordiscord — all 9 hosts now covered, each verified with a real manual run (/usr/local/bin/fail2ban-digest.sh, exit 0, sender confirmed).

Also confirmed during this pass: the [recidive] action_mwl gap this article describes below (majorlinux/teelia/majorlab silent) was already fixed by the time of this rollout — all 9 hosts have action = %(action_mwl)s on recidive. Not clear when that got fixed, but it's no longer a live gap.

Two smaller corrections still apply from the 2026-07-20 pass: fail2ban-client get DEFAULT action / get sshd action fail on the current fail2ban version ("jail 'DEFAULT' does not exist" / "list index out of range") — use fail2ban-client status <jail> and inspect /etc/fail2ban/jail.d/ directly instead.

The Problem

Fail2Ban's default action_mwl sends an email for every single ban — IP address, whois lookup, and relevant log lines. On a fleet of 9 servers running 50+ jails, this floods the inbox with hundreds of emails daily. The vast majority are routine scanner bans (probing .env, wp-login.php, random PHP paths) that require no human attention.

The signal-to-noise ratio is terrible. Genuinely important events — SSH brute-force attempts, recidive escalations — get buried in the noise.

The Solution — Tiered Alert Model

Three tiers replace the firehose:

Tier Jails Action Why
Immediate email recidive action_mwl Repeat offenders only — someone has been banned multiple times across jails
Silent ban Everything else action_ (default) Ban happens, firewall rule applied, no email sent
Daily digest All jails Cron script at 08:00 UTC One summary email per host with ban counts across all jails

This reduces email volume from hundreds per day to ~10 (one digest per host + occasional recidive alerts).

jail.local Configuration

Set silent ban as the default

In [DEFAULT]:

[DEFAULT]
action = %(action_)s

This overrides the stock action_mwl for all jails. Bans still happen — the firewall rule is applied — but no email is sent.

Keep immediate alerts for recidive only

[sshd]
enabled = true
action = %(action_)s

[recidive]
enabled = true
action = %(action_mwl)s

Updated 2026-05-02: sshd was moved to silent (action_). Only recidive (repeat offenders) now triggers immediate email. sshd bans are captured in the daily digest.

Clean up email subjects with fq-hostname

By default, fail2ban uses the system FQDN in email subjects. On Tailscale hosts, this produces ugly subjects like [Fail2Ban] sshd: banned 1.2.3.4 on MajorToot.tail7f2d9.ts.net. Override it in [DEFAULT]:

[DEFAULT]
fq-hostname = majortoot

This sets the <fq-hostname> tag used in action templates, producing cleaner subjects: [Fail2Ban] sshd: banned 1.2.3.4 on majortoot.

Daily Digest Script

A shell script at /usr/local/bin/fail2ban-digest.sh runs via cron at 08:00 local (each host's own system timezone — US hosts America/New_York, UK hosts Europe/London, per vps-migration-baseline-checklist — not literally UTC). It does the following:

  1. Queries every active jail via fail2ban-client status <jail>
  2. Collects four metrics per jail: currently banned, total banned, currently failed, total failed
  3. Builds a plain-text email body with one line per jail
  4. Separates "active" jails (with bans) from "quiet" jails (zero bans) for quick scanning
  5. Sets the subject line to the currently-banned count (real format: Subject: [<hostname -f>] Fail2Ban Digest — <N> banned)
  6. Sends via sendmail -f <FROM> <DEST>

The script is templated (roles/fail2ban/templates/fail2ban-digest.sh.j2) and deployed by Ansible to /usr/local/bin/fail2ban-digest.sh on every host in the fail2ban inventory group (all 9 fleet hosts).

Cron entry

0 8 * * * /usr/local/bin/fail2ban-digest.sh

Managed by ansible.builtin.cron — no manual crontab editing needed.

Ansible Deployment (as actually built, 2026-07-26)

The fail2ban role (roles/fail2ban/, run via fail2ban.yml or site.yml) uses a parity-capture design: jail configs (jail.local, jail.d/*, custom filter.d/*) are deployed verbatim from a per-host snapshot captured from the live fleet (roles/fail2ban/files/hosts/<inventory_hostname>/), not templated or edited in place. See ansible-flat-playbooks-to-roles for why — an earlier flag/template implementation had drifted from the live fleet and would have dropped real jails on rollout. This means the [DEFAULT]/[recidive] action tiering described above is whatever's captured in each host's snapshot, not something the digest tasks touch.

Two tasks, appended to the end of roles/fail2ban/tasks/main.yml, handle the digest email specifically:

- name: Deploy daily fail2ban digest email script (2026-07-26)
  ansible.builtin.template:
    src: fail2ban-digest.sh.j2
    dest: /usr/local/bin/fail2ban-digest.sh
    owner: root
    group: root
    mode: '0755'

- name: Schedule the fail2ban digest via cron (08:00 local, matches captured reference hosts)
  ansible.builtin.cron:
    name: fail2ban-digest
    hour: "8"
    minute: "0"
    job: /usr/local/bin/fail2ban-digest.sh
    user: root

Host-specific sender override

One variable, fail2ban_sender, set in host_vars/<host>/vars.yml where the host has its own public domain identity or a mail identity distinct from the simple hostname pattern:

# host_vars/dcaprod/vars.yml
fail2ban_sender: "fail2ban@dcanalysts.net"

# host_vars/majorlinux/vars.yml
fail2ban_sender: "fail2ban@majorlinux.com"

# host_vars/tttpod/vars.yml
fail2ban_sender: "fail2ban@techtalkthursdays.net"

# host_vars/majortoot/vars.yml -- internal default pattern, not the Mastodon domain
fail2ban_sender: "fail2ban@majortoot.majorshouse.com"

# host_vars/majormail/vars.yml -- matches its real postfix identity (SPF/DKIM alignment)
fail2ban_sender: "fail2ban@mail.majorshouse.com"

The template falls back to fail2ban@{{ inventory_hostname }}.majorshouse.com when fail2ban_sender is unset (majorlab, majorhome, majordiscord — no public domain of their own). Each address was picked deliberately, not auto-generated — see the per-host reasoning in Fleet-Security-Hardening.

Why the jail.local editing gotchas below no longer apply

This article originally described a Python jail.local section-editor as the deployment mechanism, to avoid lineinfile clobbering stock action_mwl macro definitions when trying to change the action = assignment. That editor never actually existed in the real deployment — the parity-capture design sidesteps the whole class of problem by deploying a known-good captured file verbatim instead of line-editing a live one. The gotchas below are kept for historical/reference value (they're real Fail2Ban INI-parsing traps if you ever do need to line-edit jail.local some other way), but they don't describe what this fleet actually runs.

Gotchas (reference — not how the fleet actually deploys this, see above)

lineinfile matches stock action definitions

Using lineinfile with a regex like regexp: '^action\s*=\s*%(action_mwl)s' is dangerous. In a full jail.local that includes stock action definitions (copied from jail.conf), the regex matches lines like:

action_mwl = %(action_mwl)s

This is the stock definition of the action_mwl macro itself — not the action = assignment in [DEFAULT]. lineinfile replaces the wrong line, corrupting the config. Fail2ban then refuses to start because the action_mwl macro is undefined.

Duplicate action lines crash fail2ban

If a section ends up with two action = lines (e.g., from a botched lineinfile run), fail2ban refuses to load:

option 'action' in section 'DEFAULT' already exists

defaults-debian.conf overrides jail.local

On Debian/Ubuntu, /etc/fail2ban/jail.d/defaults-debian.conf is loaded after jail.local. If it contains action = %(action_mwl)s, it silently overrides your silent default — every jail sends email on every ban. If you see per-ban emails after deploying digest mode, check this file first:

grep action /etc/fail2ban/jail.d/defaults-debian.conf

fq-hostname scope

Setting fq-hostname in [DEFAULT] affects all action templates that use the <fq-hostname> tag — including both immediate emails and the digest subject. Be aware that it overrides the system hostname globally within fail2ban.

Verifying the Setup

fail2ban-client get DEFAULT action / get sshd action fail on the current fail2ban version (jail 'DEFAULT' does not exist / list index out of range) — use status and inspect the jail files directly instead:

# Check recidive's action (should be action_mwl -- the one tier that emails)
fail2ban-client status recidive
grep action /etc/fail2ban/jail.d/recidive.conf

# Trigger a test digest -- exit 0 and a real email is the only real proof it works
/usr/local/bin/fail2ban-digest.sh; echo "exit: $?"

# Check cron is installed
crontab -l | grep -A1 fail2ban-digest

# Confirm the sender address matches what's expected for this host
grep '^FROM=' /usr/local/bin/fail2ban-digest.sh

See Also