Diagnosing Intermittent Connection Failures to a Home-Hosted Service

The Problem

A public-facing service (a webhook, an API endpoint) hosted at home behind consumer ISP/router gear intermittently fails to respond — not with an HTTP error, but with a bare connection failure (curl's 000 / exit code 28, "operation timed out"). The failure rate is roughly consistent (not obviously tied to load), and every individual layer you check in isolation looks healthy.

This is a genuinely hard class of problem because the loss is usually happening in a place that produces no log entry anywhere — not in the app, not in the reverse proxy, not in the router's own event log — because it's happening during TCP handshake setup, before any of those things see the connection.

The wrong instinct: assume it's the app or a burst/concurrency issue

Two tempting-but-often-wrong first theories: 1. "It's the app being slow under load." Check by testing paced, fully sequential requests (1–2 seconds apart, zero concurrency). If the failure rate barely changes, it's not a concurrency/burst problem — ruling this out early saves a lot of wasted effort chasing app-side threading/backlog theories. Caveat: a paced test rules out the app, but it does not rule out a shared rate limiter in front of the app (see AsusWRT DoS protection below). A limiter on the router's WAN side counts every new inbound connection from anywhere — internet scanner noise included — against one budget, so your politely paced requests still fail whenever someone else has already spent it. Don't conclude "not a rate/burst problem" from the paced test alone. 2. "A plain ICMP mtr shows heavy loss at hop N, so that's the culprit." This is very often a false signal. Routers frequently deprioritize generating ICMP TTL-exceeded replies (what traceroute-style tools depend on) while forwarding real traffic at full rate. The tell: if hop N shows high loss but hop N+1 (immediately downstream) shows 0% loss, that's impossible if hop N were really dropping that fraction of transiting traffic — it only makes sense if hop N is just bad at replying to probes. Don't treat plain ICMP mtr loss as proven real without this cross-check.

Isolate the layer: does the app even see the failed requests?

Compare the failing client's timeline against the server's own access log (same time window, clocks normalized — see gotcha below). If the server's log shows every request that arrived was answered fast and clean, with zero entries at all for the ones the client reports failed, then those requests never reached the app or reverse proxy — the failure is happening at or before the TCP handshake, not in application logic. This one comparison is the single most useful diagnostic step, and it's often skipped in favor of digging into app config first.

Gotcha: if the client and server are different machines, do not eyeball-correlate timestamps across them without first checking each machine's own clock/timezone (date on both). A 5-hour cross-host timezone mismatch will look exactly like "none of these requests reached the server at all" and can send a whole investigation down the wrong path.

Isolate the phase: time the connection, not just the request

curl -w can break a single request into its component phases:

curl -s -o /dev/null -w \
  "dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total} code=%{http_code} exit=%{exitcode}\n" \
  --max-time 5 https://your-service/endpoint

Run this repeatedly (paced). If connect is fast and consistent (~30ms) on successes but dns/tls/ttfb add almost nothing beyond it, and outright failures show connect=0.000000 with the whole request eaten by the timeout — the delay/failure is entirely in TCP handshake setup, not DNS, not TLS negotiation, not the application. This is a much stronger, more specific finding than "sometimes it's slow."

Note: %{errno} is not supported on all curl versions despite being documented in some references — use %{exitcode} instead (28 = timed out, 7 = connection refused, etc.) if errno produces curl: unknown --write-out variable.

Isolate the mechanism: kernel counters distinguish "queue full" from "packets lost"

On the receiving host, netstat -s (or the raw TcpExt: line in /proc/net/netstat) exposes counters worth checking together, not individually:

• ListenOverflows / ListenDrops — the server's own accept queue rejecting connections because it's full. Check ss -ltn for the actual backlog on the listening port and /proc/sys/net/core/somaxconn for the ceiling. If these counters are near zero and the backlog isn't close to its ceiling, the server's own queue is not the bottleneck — don't spend time raising somaxconn on a guess.
• TCPSynRetrans — the server had to retransmit a SYN/SYN-ACK because it didn't get acknowledged in time. A non-trivial, accumulating count here is the fingerprint of real packet loss affecting handshake-establishing packets specifically — this is consistent with the connect-phase-only delays found via the curl timing breakdown above.
• EmbryonicRsts — connections reset while still half-open (SYN_RECV). Elevated alongside TCPSynRetrans reinforces the same read.
• SyncookiesFailed — some non-zero baseline is normal on any host getting scanned/probed from the open internet; only worth treating as signal if it's climbing unusually.

The combination of "accept-queue counters near zero" + "non-trivial SynRetrans/EmbryonicRsts" points specifically at packets being lost somewhere upstream of this host, not at anything the host's own software is doing wrong.

When the router's own logs come back clean

Consumer router firmware (e.g. AsusWRT) typically logs events — Wi-Fi associations, DHCP leases, daemon restarts — not forwarding-path packet loss or per-connection state. A clean syslog and unremarkable conntrack usage (cat /proc/sys/net/netfilter/nf_conntrack_count vs nf_conntrack_max) on the router is not proof the router/ISP boundary is innocent — it just means nothing was in a position to log the specific thing you're looking for. Treat this as "not incriminated," not "cleared," and say so explicitly rather than overstating the conclusion.

The place silent drops do show up is the packet/byte counters on the router's firewall rules (iptables -L <chain> -v -n -x). A rule that drops packets increments its counter whether or not anything logs it — so check the counters before concluding the router is out of the picture. The next section is the concrete case.

Check the router's firewall counters, not its log: AsusWRT DoS protection

Stock AsusWRT (confirmed on an RT-AX82U, firmware 3.0.0.4.388_25103) has a setting, Firewall → General → Enable DoS protection (nvram fw_dos_x=1), that silently rate-limits every new inbound connection to the whole network. It is easy to turn on, sounds harmless, and never logs a single drop.

What it does

With fw_dos_x=1 the firmware adds a SECURITY chain, and FORWARD jumps to it for traffic arriving on the WAN interface:

-A SECURITY -p tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 1/sec -j RETURN
-A SECURITY -p tcp --tcp-flags FIN,SYN,RST,ACK SYN -j DROP

…plus the same pattern for TCP RST and ICMP echo-request. -m limit --limit 1/sec uses the default burst of 5, and there is one limit bucket for the rule, not one per source or per destination. So the entire house gets roughly one new inbound connection per second, burst of 5 — and the constant background of internet scanners probing your forwarded ports draws from the same bucket as your real clients. Every SYN over budget is dropped without a trace.

How to check it

From an SSH session on the router:

nvram get fw_dos_x                          # 1 = enabled
iptables -L FORWARD -v -n | grep SECURITY   # is FORWARD jumping to it?
iptables -L SECURITY -v -n -x               # exact packet counters per rule

Compare the SYN RETURN (passed) counter against the SYN DROP counter. In the case this section is drawn from, after 3.4 days of router uptime: 161,825 SYNs passed vs 72,572 dropped — about 31% of all new inbound connections silently discarded. Over the same period the syslog was clean, firewall logging was off (fw_log_x=none, the default), and conntrack sat at 717 of 300,000. Every check described in the previous section came back innocent; only the rule counters told the truth.

The connect-time signature

When a client's SYN is dropped, the client's kernel retransmits it on an exponential backoff (1 s, then doubling). So connections that eventually succeed don't spread evenly — they cluster at the retransmit points. Timing connect from an external host (here a VPS, 40 requests, 8 in parallel, using the curl -w '%{time_connect}' breakdown above):

Outcome	Count	time_connect
Fast	6	~0.03 s
Slow	30	clustered at ~1, 2, 3, 5, 7, 11 s
Timed out	4	20 s (--max-time)

A connect time clustered at the client's SYN retransmit intervals means the first SYN (and possibly several) vanished and a later retry got through. A slow app produces a smooth spread in ttfb, not quantized connect times. Combined with the kernel counters from earlier — TCPSynRetrans on the client side, no ListenOverflows on the server — this signature points squarely at something dropping SYNs in front of the host.

The fix

Turn the setting off — Web UI: Firewall → General → Enable DoS protection: No — or over SSH:

nvram set fw_dos_x=0
nvram commit
service restart_firewall

Verify the chain is no longer referenced (expected output 0):

iptables -L FORWARD -v -n | grep -c SECURITY

Then re-run the same external test. After the change: 40/40 fast, average connect 0.029 s, max 0.047 s, zero timeouts.

Rollback: nvram set fw_dos_x=1; nvram commit; service restart_firewall (or flip the Web UI toggle back).

The trade-off

You lose very little. The limiter is a single crude bucket shared by every inbound connection: under a genuine flood it would drop your legitimate traffic just as readily as the attack's, and a flood large enough to matter saturates the WAN link long before the router's firewall gets a say. Real protection comes from elsewhere and stays in place: only the explicit port forwards are open, AsusWRT's AiProtection remains on, and each exposed host keeps its own defenses (its own firewall, rate limiting at the reverse proxy, fail2ban or similar).

The original incident

The 2026-08-27 investigation that prompted this article ended at the "Where this typically bottoms out" conclusion below. In hindsight it was likely this same limiter — the symptoms match — but that was not verified at the time.

Where this typically bottoms out

Before concluding anything about the WAN or the ISP, rule out the router's own SYN rate limiting — see AsusWRT DoS protection above. Other firmware has equivalents (often labelled "SYN flood protection" or "DoS protection"); check the firewall rule counters, not the log.

Once that is ruled out, and absent access to the ISP's own equipment, the most specific finding achievable from the LAN/host side alone is usually: intermittent SYN/SYN-ACK loss on the WAN path, most likely at or near the ISP/router boundary — supported by the curl phase timing + kernel retransmit counters, and consistent with (though not conclusively proven by) any last-mile-hop anomaly seen in a TCP-based mtr (mtr --tcp -P <port>, which follows the actual protocol treatment given to real traffic, unlike plain ICMP mtr). Going further usually requires either ISP-side escalation or accepting the loss rate and adding retry logic on the calling side.

Related

• Loki/Grafana log correlation across hosts should always start with a date check on each host — see the cross-host clock caution above; this has caused real false leads more than once.