Transient LAN Latency With No Root Cause: An Elimination Method¶
Symptom¶
Everything loading slowly; one server apparently unreachable. Measured while it was happening:
| Path | avg | max | loss |
|---|---|---|---|
| workstation → Pi | 717 ms | 2,707 ms | 35% |
| hostA → mesh AP | 6,007 ms | 8,592 ms | 25% |
| hostB → Pi | 864 ms | 3,544 ms | 0% |
| hostA → hostB | 710 ms | 2,891 ms | 0% |
Simultaneously, other pairs were sub-millisecond — including inside a single parallel batch. By the next check everything was clean.
This article is about what to do with that, and it ends without a root cause. That is the point: the elimination is the deliverable.
Hypothesis 1 — a topology split¶
Latency clustered so neatly into two groups that a segmented topology looked obvious: intra-group sub-millisecond, cross-group catastrophic.
Killed by: the owner stating all four hosts were on the same switch via ethernet.
Lesson: latency clustering is not a topology map. Inferring physical layout from timing and then reasoning from that inference compounds one guess into a conclusion. Confirm the physical layer from someone who can see the cables, or from link state — not from ping.
Hypothesis 2 — conntrack exhaustion¶
A NAT/conntrack table filling up produces LAN-wide latency and loss that resolves itself. It fits the shape perfectly, and the file server had been pulling a large download during the window.
Killed by: the metrics. Peak was 694 connections against a 262,144 ceiling, and insert_failed, drop, and early_drop were all zero. The router's own table was 1,073 / 300,000.
Lesson: "fits the shape" is not evidence. Conntrack exhaustion has a specific, checkable signature — the failure counters. If they're zero, the table didn't overflow, no matter how well the story fits.
Hypothesis 3 — a slow link between host groups¶
If a 100 Mbps port or a degraded cable sat between the two apparent groups, saturating it would explain queuing delay while leaving every host's own NIC clean.
Killed by: a throughput test. Cross-group measured 816 Mbps, intra-group 848 Mbps — within 4%. No constrained link exists.
With iperf3 unavailable, SSH with a fixed cipher is a decent substitute:
Not precise — it's CPU-bound on crypto — but it distinguishes ~94 Mbps from ~940 Mbps instantly, which is the question being asked.
Lesson: host NICs negotiating 1000/full says nothing about intermediate links. Measure end-to-end throughput on the actual path.
Hypothesis 4 — the download saturated something¶
Interface counters showed ~58 Mbps inbound during exactly the incident window, tightly correlated in time. Very compelling.
Killed by: longer-range history. That host routinely peaks at 899–984 Mbps inbound; 984 Mbps was its 30-day maximum. 58 Mbps is roughly 6% of a normal day.
Lesson: correlation inside a narrow window is worthless without a baseline. Always ask "is this value actually unusual for this host?" before treating coincidence as cause. Pull a week, not an hour.
The blind spot: syslog is not telemetry¶
With hosts exonerated, attention moved to the router and mesh AP. Both logs covered the window. Both were essentially empty — 19 routine lines and zero lines respectively.
One entry looked promising: a DPI engine Revert process firing at 23:00:20, right inside the window. Checking its frequency killed it — it appears 28 times in the log, roughly hourly. Scheduled housekeeping.
The important conclusion is what a clean syslog does and does not mean:
AsusWRT syslog records events — associations, DHCP, daemon restarts. It never records CPU load, airtime utilisation, or forwarding latency.
So the router was not exonerated. It was unincriminated on absent evidence, which is a much weaker statement than the hosts earned from actual metric history. Keeping those two categories distinct is the single most useful discipline in this kind of investigation — otherwise "nothing in the logs" silently becomes "the router is fine" and the real cause never gets found.
What a documented negative result buys you¶
No root cause was found. Recorded anyway:
- Exonerated on evidence: cabling and switching capacity, NIC speed/duplex, host network stacks, conntrack, host logs.
- Unincriminated on absent evidence: router and mesh AP — no performance telemetry exists.
- Four specific hypotheses, each with the measurement that killed it.
Next time this happens, nobody re-runs those four investigations. The search starts where this one stopped. A negative result that is written down is a permanent narrowing of the problem space; one that isn't gets rediscovered from scratch.
Turning the blind spot into instrumentation¶
The actionable output of a failed investigation is usually monitoring, not a fix:
- A ping mesh across every host, recording every sample — because the degradation was pair-asymmetric and a single vantage point structurally cannot see that.
- A read-only router sampler for load, conntrack and per-radio airtime (
wl chanim_stats) — because the device had no history at all.
Both are covered in LAN Latency Ping Mesh with ping -D -O and systemd.
Writing down "normal" is worth it on its own. The baseline collected immediately afterwards revealed a mesh AP running at a third of the main router's transmit opportunity with ~10× the bad-PLCP rate — chronic congestion nobody had noticed, found purely by measuring what healthy looks like.
Checklist¶
When a transient network event leaves no trace:
- [ ] Measure from multiple vantage points — asymmetry is the highest-value signal.
- [ ] Distinguish ICMP latency from throughput; they answer different questions and consumer gear deprioritises ICMP.
- [ ] For any resource-exhaustion theory, check the failure counters, not just the gauge.
- [ ] Compare any "unusual" value against a week of that host's own history.
- [ ] Confirm the physical layer from link state or a human, never inferred from timing.
- [ ] Ask whether a device retains performance history at all before treating its clean log as exoneration.
- [ ] Verify your own tooling isn't the source — see Ping Latency Spikes That Are Really Collector Scheduling.
- [ ] Write down the negative result with the evidence that killed each hypothesis.