Apache Outage: Fail2ban Self-Ban + Missing iptables Rules

🛑 Problem

A web server running Apache2 becomes completely unreachable (ERR_CONNECTION_TIMED_OUT) despite Apache running normally. SSH access via Tailscale is unaffected.

---

🔍 Diagnosis

Step 1 — Confirm Apache is running

sudo systemctl status apache2

If Apache is active (running), the problem is at the firewall layer, not the application.

---

Step 2 — Test the public IP directly

curl -I --max-time 5 http://<PUBLIC_IP>

A timeout means traffic is being dropped by the firewall. A connection refused means Apache is down.

---

Step 3 — Check the iptables INPUT chain

sudo iptables -L INPUT -n -v

Look for ACCEPT rules on ports 80 and 443. If they're missing and the chain policy is DROP, HTTP/HTTPS traffic is being silently dropped.

Example of broken state:

Chain INPUT (policy DROP)
  ACCEPT  tcp  --  lo       *    ...   # loopback only
  ACCEPT  tcp  --  tailscale0 *  ...   tcp dpt:22
  # no rules for port 80 or 443

---

Step 4 — Check the nftables ruleset for Fail2ban

sudo nft list tables

Look for table inet f2b-table — this is Fail2ban's nftables table. It operates at priority filter - 1, meaning it is evaluated before the main iptables INPUT chain.

sudo nft list ruleset | grep -A 10 'f2b-table'

Fail2ban rejects banned IPs with rules like:

tcp dport { 80, 443 } ip saddr @addr-set-wordpress-hard reject with icmp port-unreachable

A banned admin IP will be rejected here regardless of any ACCEPT rules downstream.

---

Step 5 — Check if your IP is banned

for jail in $(sudo fail2ban-client status | grep "Jail list" | sed 's/.*://;s/,/ /g'); do
  echo "=== $jail ==="; sudo fail2ban-client get $jail banip | tr ',' '\n' | grep <YOUR_IP>
done

---

✅ Solution

Fix 1 — Add missing iptables ACCEPT rules for HTTP/HTTPS

If ports 80/443 are absent from the INPUT chain:

sudo iptables -I INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
sudo iptables -I INPUT -i eth0 -p tcp --dport 443 -j ACCEPT

Persist the rules:

sudo netfilter-persistent save

If netfilter-persistent is not installed:

sudo apt install -y iptables-persistent
sudo netfilter-persistent save

---

Fix 2 — Unban your IP from all Fail2ban jails

for jail in $(sudo fail2ban-client status | grep "Jail list" | sed 's/.*://;s/,/ /g'); do
  sudo fail2ban-client set $jail unbanip <YOUR_IP> 2>/dev/null && echo "Unbanned from $jail"
done

---

Fix 3 — Add your IP to Fail2ban's ignore list

Edit /etc/fail2ban/jail.local:

sudo nano /etc/fail2ban/jail.local

Add or update the [DEFAULT] section:

[DEFAULT]
ignoreip = 127.0.0.1/8 ::1 <YOUR_IP>

Restart Fail2ban:

sudo systemctl restart fail2ban

---

🔁 Why This Happens

Issue	Root Cause
Missing port 80/443 rules	iptables INPUT chain left incomplete after a manual firewall rework (e.g., SSH lockdown)
Still blocked after adding iptables rules	Fail2ban uses a separate nftables table at higher priority — iptables ACCEPT rules are never reached for banned IPs
Admin IP gets banned	Automated WordPress/Apache probes trigger Fail2ban jails against the admin's own IP

---

⚠️ Key Architecture Note

On servers running both iptables and Fail2ban, the evaluation order is:

1. inet f2b-table (nftables, priority filter - 1) — Fail2ban ban sets; evaluated first
2. ip filter INPUT chain (iptables/nftables, policy DROP) — explicit ACCEPT rules
3. UFW chains — IP-specific rules; evaluated last

A banned IP is stopped at step 1 and never reaches the ACCEPT rules in step 2. Always check Fail2ban after confirming iptables looks correct.

---

🔎 Quick Diagnostic Commands

# Check Apache
sudo systemctl status apache2

# Test public connectivity
curl -I --max-time 5 http://<PUBLIC_IP>

# Check iptables INPUT chain
sudo iptables -L INPUT -n -v

# List nftables tables (look for inet f2b-table)
sudo nft list tables

# Check Fail2ban jail status
sudo fail2ban-client status

# Check a specific jail's banned IPs
sudo fail2ban-client status wordpress-hard

# Unban an IP from all jails
for jail in $(sudo fail2ban-client status | grep "Jail list" | sed 's/.*://;s/,/ /g'); do
  sudo fail2ban-client set $jail unbanip <YOUR_IP> 2>/dev/null && echo "Unbanned from $jail"
done