LAN Latency Ping Mesh with ping -D -O and systemd

The problem this solves

A LAN degrades for ten minutes — pages crawl, a file server feels unreachable — and by the time you look, it's fine. You check the hosts: clean. You check the router's syslog: clean. You have nothing.

That is not bad luck, it's a telemetry gap with three specific parts:

1. Consumer routers and APs keep no performance history. AsusWRT syslog records events — associations, DHCP leases, daemon restarts. It never records CPU, airtime utilisation, or forwarding latency. A clean syslog is absence of evidence, not evidence of absence.
2. Host metrics don't see the path. Netdata on a server shows its NIC counters are clean, which tells you nothing about what happened between two other machines.
3. Degradation is often pair-asymmetric. In the event that prompted this build, host A→C measured 0.2 ms while host B→C measured 864 ms in the same instant. A single vantage point structurally cannot see that, and it was the most diagnostic signal available.

The fix is cheap: record every probe, continuously, from several vantage points.

Why ping -D -O specifically

Two flags do the heavy lifting:

• -D prefixes each line with a [unix.microseconds] timestamp, so samples are correlatable across hosts without trusting log write order.
• -O emits an explicit line for a non-reply (no answer yet for icmp_seq=N). Without it, loss is only visible in the summary at the end of a run — useless for a daemon that never ends. With it, loss is a first-class, timestamped event.

fping is a reasonable alternative, but ping is already on every box.

Parsing: match field content, never position

This is the part that bites people. A reply line looks like:

[1786529079.964454] 64 bytes from 192.168.50.94: icmp_seq=1 ttl=63 time=0.754 ms

An obvious-looking parser grabs $2 for the value — and silently records 64, the packet size, as a 64 ms latency. Every packet then looks like a uniform spike, which is exactly wrong enough to look plausible. (This happened during the investigation that led to this article; it was caught only because a perfectly constant 64 ms across three unrelated hosts is physically implausible.)

Parse by scanning for the field that starts with time=:

# parse.awk — turn `ping -D -O` output into CSV + an anomaly log.
# Required vars: -v name= -v ip= -v data= -v thresh=

function refresh(   cmd, d) {
  cmd = "date +%F"; cmd | getline d; close(cmd)
  if (d != today) {
    today = d
    if (out    != "") close(out)
    if (alerts != "") close(alerts)
    out    = data "/lanwatch-" d ".csv"
    alerts = data "/alerts-"   d ".log"
  }
}

BEGIN { n = 0; today = ""; out = ""; alerts = "" }

{
  if (n % 60 == 0) refresh()
  n++

  ts = $1; gsub(/[\[\]]/, "", ts)
  if (ts !~ /^[0-9]+(\.[0-9]+)?$/) next    # banner / untimestamped noise

  seq = ""; rtt = ""; status = "OTHER"
  for (i = 1; i <= NF; i++) {
    if ($i ~ /^icmp_seq=/) { split($i, a, "="); seq = a[2] }
    if ($i ~ /^time=/)     { split($i, b, "="); rtt = b[2] }
  }
  if      (rtt != "")            status = "OK"
  else if ($0 ~ /no answer yet/) status = "TIMEOUT"
  else if ($0 ~ /Unreachable/)   status = "UNREACHABLE"

  printf "%s,%s,%s,%s,%s\n", ts, name, seq, status, rtt >> out
  fflush(out)

  if (status != "OK" || (rtt + 0) > thresh) {
    printf "%s %s %s seq=%s %s %s\n", ts, name, ip, seq, status, rtt >> alerts
    fflush(alerts)
  }
}

fflush() matters — without it you lose the most interesting samples when the process is killed mid-incident.

Test the parser before you trust it

Monitoring that lies is worse than no monitoring. A tiny fixture catches the whole class of bug:

cat > input.txt <<'EOF'
PING 192.168.50.94 (192.168.50.94) 56(84) bytes of data.
[1786529079.964454] 64 bytes from 192.168.50.94: icmp_seq=1 ttl=63 time=0.754 ms
[1786529080.995849] no answer yet for icmp_seq=2
[1786529081.123456] From 192.168.50.1 icmp_seq=3 Destination Host Unreachable
[1786529082.222222] 64 bytes from 192.168.50.94: icmp_seq=4 ttl=63 time=1234 ms
EOF

awk -v name=test -v ip=192.168.50.94 -v data=. -v thresh=50 -f parse.awk < input.txt

Assert, at minimum: the banner is skipped, TIMEOUT/UNREACHABLE are distinguished, a high RTT keeps its real value, and no row ever has an RTT of exactly 64. That last assertion is the regression guard for the packet-size bug above.

One trap in the test harness itself: grep -c prints 0 and exits 1 on no match, so $(grep -c ... || echo 0) emits two lines and breaks a string comparison. Let the printed 0 stand alone.

The collector

#!/usr/bin/env bash
set -uo pipefail

DIR="${LANWATCH_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)}"
DATA="$DIR/data"
INTERVAL="${LANWATCH_INTERVAL:-1}"
THRESH_MS="${LANWATCH_THRESH_MS:-50}"
RETAIN_DAYS="${LANWATCH_RETAIN_DAYS:-14}"

mkdir -p "$DATA"

TARGETS=(
  "router:192.168.50.1"
  "aimesh:192.168.50.87"
  "hostA:192.168.50.94"
  "hostB:192.168.50.128"
  "hostC:192.168.50.238"
)

find "$DATA" -name 'lanwatch-*.csv' -mtime +"$RETAIN_DAYS" -delete 2>/dev/null
find "$DATA" -name 'alerts-*.log'   -mtime +"$RETAIN_DAYS" -delete 2>/dev/null

pids=()
cleanup() { [ ${#pids[@]} -gt 0 ] && kill "${pids[@]}" 2>/dev/null; exit 0; }
trap cleanup TERM INT

for entry in "${TARGETS[@]}"; do
  name="${entry%%:*}"; ip="${entry##*:}"
  ping -D -O -i "$INTERVAL" "$ip" 2>&1 \
    | awk -v name="$name" -v ip="$ip" -v data="$DATA" -v thresh="$THRESH_MS" \
          -f "$DIR/parse.awk" &
  pids+=("$!")
done

wait

Derive DIR from the script's own location, not $HOME. Install to /opt and run as a system service and a $HOME-based default silently points the report tool at /root/... while the collector writes elsewhere — it looks like "no data" while data is being collected fine.

Cost: ~16 MB/day/host for five targets at 1 pps, roughly 2.5 kbit/s of ICMP. Negligible against any modern link.

systemd unit — and the Nice trap

[Unit]
Description=lanwatch — continuous LAN latency/loss recorder
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Environment=LANWATCH_DIR=/opt/lanwatch
ExecStart=/opt/lanwatch/lanwatch.sh
Restart=always
RestartSec=5
Nice=-5

[Install]
WantedBy=multi-user.target

Do not deprioritise a latency-measurement tool. ping computes RTT in userspace (gettimeofday at send and receive), so if the process is descheduled the delay is charged to the measurement. Setting Nice=10 on two busy hosts produced readings up to 690 ms on a completely idle LAN. Nice=-5 fixed it; max RTT dropped to 2.68 ms. Negative nice needs CAP_SYS_NICE, so a user unit cannot do it — use a system unit, or accept Nice=0.

See Ping Latency Spikes That Are Really Collector Scheduling for how to recognise this in data you have already collected.

System unit vs user unit

	User unit	System unit
Survives reboot	needs loginctl enable-linger	yes
Negative Nice	no	yes
Fits	workstations, your own login	servers

On a headless server reached as root, use a system unit. Root user-units need lingering and buy nothing.

SELinux hosts

Write the unit file in place (cat > /etc/systemd/system/...), never mv it from /tmp — a file created in /tmp carries tmp_t and systemd will be denied reading it. Written in place it inherits systemd_unit_file_t correctly. Verify with ls -Z, and check for denials with a bounded scan — ausearch -ts recent can hang for minutes on a large audit log:

tail -3000 /var/log/audit/audit.log | grep -i denied | grep -ic lanwatch   # expect 0

Windows hosts must be told to answer

A Windows machine on the LAN will show 100% loss with no network fault: Windows Firewall ships with inbound ICMPv4 Echo disabled. Add a scoped rule (elevated):

New-NetFirewallRule -Name lanwatch-icmp4-echo-in `
  -DisplayName 'ICMPv4 Echo Request (LAN monitoring)' `
  -Direction Inbound -Protocol ICMPv4 -IcmpType 8 `
  -Profile Private -RemoteAddress 192.168.50.0/24 -Action Allow

Scope it to the LAN subnet. If the machine runs Tailscale, WireGuard or similar, that adapter is very likely also on the Private profile — an unscoped Private rule exposes ICMP to every VPN peer, not just the LAN.

Until that rule exists, leave the Windows host out of the target list. Otherwise it generates a permanent 100%-loss alert stream, and an always-firing channel is one nobody reads.

Sampling a router that keeps no metrics

The LAN mesh records the symptom. To record the suspected cause on AsusWRT (and Broadcom-based Asus firmware generally), poll read-only over SSH.

The gold is chanim_stats, which exposes real airtime:

wl -i $(nvram get wl0_ifname) chanim_stats

chanspec tx inbss obss nocat nopkt doze txop goodtx badtx glitch badplcp knoise idle timestamp
0x1003   5  1     9    1     4     0    81   1      0     2520   40      -80    85   924969272

Useful columns: txop (percentage of time a transmit opportunity was available), idle, obss (other-BSS interference), glitch/badplcp (interference and collisions), knoise (noise floor). Combine with /proc/loadavg, /proc/stat and /proc/sys/net/netfilter/nf_conntrack_count.

Read only. On AiMesh, acsd2 re-asserts channel/width changes made via nvram/wl within seconds. Query freely; change settings in the web UI.

Use SSH ControlMaster or you will poison the evidence

Polling every 30 s with a fresh SSH login means ~2,880 logins/day, each writing a Password auth succeeded line into the very syslog you need for forensics. Multiplex instead — one authentication, many command channels:

SOCK=/run/user/$UID/rw-router.sock

ensure_master() {
  if ! ssh -S "$SOCK" -O check "$USER@$IP" >/dev/null 2>&1; then
    rm -f "$SOCK"
    sshpass -p "$PW" ssh -M -S "$SOCK" -fN -p 1025 \
      -o StrictHostKeyChecking=accept-new -o ServerAliveInterval=30 \
      -o PubkeyAuthentication=no -o HostKeyAlgorithms=+ssh-rsa "$USER@$IP"
  fi
}

sample() { ssh -S "$SOCK" -p 1025 "$USER@$IP" "$PAYLOAD"; }

Measured result: zero additional logins across 240 s of steady-state polling. Verify this rather than assuming it — count Password auth succeeded in the router's log across a quiet window.

Keep the router password in memory only: decrypt it once at start-up from your secret store and never write it to disk or echo it.

Calibrate thresholds from measured baseline, not intuition

A "sensible" floor of txop < 40 fired on every single sample from one AP whose 2.4 GHz radio normally sits at 25–36%. That produces a permanently-alerting channel — the same failure as an alert rule keyed on a label nobody populates.

Collect a baseline first, then set floors below observed-normal so they flag genuine collapse:

Device	2.4 GHz txop / idle	5 GHz txop / idle	badplcp 2.4
main router	76–85% / 84–90%	86–95% / 90–98%	12–25
mesh node	25–50% / 32–55%	75–83% / 76–85%	148–270

Two things fall out of a table like this. First, the floor belongs near 15%, not 40%. Second, that mesh node is chronically congested on 2.4 GHz — a real finding surfaced purely by writing down what "normal" is.

Re-derive thresholds from percentiles once you have a week of data.

Reading the results

Per-target loss and RTT, plus worst minutes ranked, is enough to spot an event at a glance:

lanwatch-report.sh 2026-08-11 22:40 23:40

TARGET       SAMPLES     LOST   LOSS%    MIN_ms    AVG_ms    MAX_ms    >50ms
aimesh           532        0    0.0%     0.358     0.520     2.960        0
hostB            532        0    0.0%     0.133     0.290     0.925        0
router           532        0    0.0%     0.311     0.578     5.440        0

Run it on every collector and compare. Divergence between vantage points is the signal; agreement is background.

Interpretation rule

• All targets from one host spike to the same value at the same timestamp → collector artifact (scheduling, CPU starvation). Not the network.
• Specific pairs degrade, and vantage points disagree → real network event. This is the shape worth investigating.

Related

• Ping Latency Spikes That Are Really Collector Scheduling
• Transient LAN Latency With No Root Cause: An Elimination Method
• Fleet Log Aggregation — Loki + Grafana + Alloy