Skip to content

Ping Latency Spikes That Are Really Collector Scheduling

Symptom

A freshly deployed 1-second ping monitor starts logging alerts on a LAN that is demonstrably healthy:

hostA  ...262.42  hostB 54.2ms   + apNode  54.6ms
hostB  ...263.97  hostC 80.6ms   + hostA   80.8ms
hostB  ...273.23  hostC 127ms    + hostA   127ms

Spikes ranged to 690 ms on a wired gigabit LAN that measured 0.3–0.9 ms at rest and had just passed an 820 Mbps throughput test.

The tell

Look at what spikes together.

Every target probed from a given host jumped to the same value at the same microsecond. hostC and hostA are different machines on different ports; there is no shared component that could delay both by exactly 127 ms while a third host measuring the same two targets saw nothing.

That pattern only has one explanation: the delay was added after the packets came back, inside the measuring host.

Why it happens

ping computes round-trip time in userspace. It records a timestamp when it sends, and another when recvmsg() returns. Both happen in the ping process.

If that process is not scheduled promptly — CPU contention, a busy host, an unfavourable nice value — replies sit in the socket buffer while the clock runs. When ping finally runs, it timestamps everything late and charges the delay to every in-flight probe equally.

The network was fine. The stopwatch was asleep.

Root cause in this case

The systemd unit deploying the collector carried:

Nice=10

That was copied from a "be a good citizen, this is only monitoring" instinct — and it is precisely backwards. It deprioritised a latency-measurement tool on two genuinely busy hosts (a media server running transcodes and downloads, and a Docker host running a few dozen containers). Under load the collector lost the CPU race and manufactured latency that did not exist.

Fix

Give the collector normal or elevated scheduling priority:

[Service]
Nice=-5

Result on the same hosts, same LAN, same 90-second window:

Before (Nice=10) After (Nice=-5)
Max RTT 690 ms 2.68 ms
Threshold alerts 27 0
Samples — 390/host

Notes:

  • Negative nice requires CAP_SYS_NICE. A systemd user unit cannot set it; the unit silently fails to start or the setting is ignored depending on version. Use a system unit, or settle for Nice=0.
  • Nice=0 (the default) is usually sufficient. The active mistake is going positive.
  • For very tight measurement, consider CPUSchedulingPolicy=fifo with a low CPUSchedulingPriority — but on a homelab LAN, Nice=-5 is plenty and far less risky.

Distinguishing artifact from real event

Keep this rule where you read the data:

Pattern Meaning
All targets from one host, identical value, identical timestamp Collector artifact — scheduling, CPU starvation, VM steal time
Specific pairs degrade; vantage points disagree Real network event

The second shape is what a genuine problem looks like. In the incident that prompted building this monitoring, hostA→hostC measured 0.2 ms while hostB→hostC measured 864 ms in the same instant. No scheduling artifact can produce disagreement between observers about the same target — that requires the path to actually differ.

This is also why a mesh beats a single collector. With one vantage point, an artifact and an outage look identical.

  • VMs and steal time. A collector on a contended hypervisor produces the same signature. Check %st in top, or /proc/stat's steal column.
  • Containers with CPU limits. A CPU-throttled container running ping will report the throttling as network latency.
  • ping on a laptop that sleeps. Suspend/resume shows as one enormous spike across all targets — same fingerprint, different cause.
  • The reverse error: treating a real pair-specific event as an artifact because "monitoring is probably lying again." Once you have fixed nice, trust asymmetry.