Skip to content

Tailscale Boot Race Conditions (SSH Unreachable After Reboot)

Two related race conditions can make a host unreachable via Tailscale after reboot. Both stem from systemd services starting before Tailscale or the network is ready.

Same race, Docker flavor: a container publishing a port on a Tailscale IP hits the identical bind race at boot — but there the right fix is the ip_nonlocal_bind sysctl, not an interface-ready gate (gating all of docker.service on Tailscale takes every container hostage). See Docker Port Bind to a Tailscale IP Fails at Boot (majorlab/Loki, 2026-07-09).

Race 1: ssh.socket Binds Before Tailscale Is Up (Ubuntu)

Symptom

SSH to a host via Tailscale IP times out. tailscale ping works, tailscale status shows active; direct, but SSH on port 22 refuses connections. No access via Hetzner console if root password is unset.

Cause

Ubuntu 24.04 uses systemd socket activation for SSH (ssh.socket instead of persistent ssh.service). When the socket override binds to a Tailscale IP, it can start before tailscaled.service is ready. The bind may succeed initially (Tailscale state file caches the IP), but a later Tailscale reconnect or interface reset invalidates the bound address silently — SSH dies with no recovery path.

Diagnosis

# From another host:
tailscale ping <IP>          # succeeds — host is up
ssh root@<IP>                # times out — sshd not listening

# After gaining console access or reboot:
systemctl status ssh.socket  # check Listen: address
journalctl -b -1 -u ssh     # likely empty — sshd never spawned
journalctl -b -1 -u ssh.socket  # socket started before tailscaled

Fix (current — 2026-07-20)

After=tailscaled.service orders against the service becoming active — not against the tailscale0 interface actually having an IPv4 address. tailscaled flips to active within a second of starting, but the kernel doesn't have the address bound to the interface until DERP relays connect and the control plane confirms the node. ssh.socket attempting ListenStream=<TS IP>:22 in that window fails with Cannot assign requested address, the socket goes into a failed state, and there is no automatic retry.

The proper gate is a dedicated readiness service that waits for the tailscale0 IPv4 address to exist. Note where it attaches: the readiness service itself is unremarkable — the critical detail is that the dependency on it goes on ssh.service, never on ssh.socket (see the danger callout below):

# /etc/systemd/system/tailscale-wait-ready.service
[Unit]
Description=Wait until tailscale0 has an IPv4 address
After=tailscaled.service
Requires=tailscaled.service
ConditionPathExists=/usr/sbin/ip

[Service]
Type=oneshot
RemainAfterExit=yes
TimeoutStartSec=120
ExecStart=/usr/bin/bash -c 'for i in $(seq 1 120); do ip -4 -o addr show tailscale0 2>/dev/null | grep -q "inet " && exit 0; sleep 1; done; exit 1'

[Install]
WantedBy=multi-user.target

The readiness gate goes on ssh.service — NEVER on ssh.socket

ssh.socket is pulled in by sockets.target early in boot. Any [Unit] dependency on it creates an ordering cycle, which systemd resolves by deleting the ssh.socket start job — killing SSH on boot with no retry and no console-free recovery.

This has happened three times on this fleet: dcaprod 2026-05-23, majorlinux 2026-06-04 (3rd occurrence), and majortoot was found latently armed 2026-06-07 (would have lost SSH on its next reboot).

An earlier revision of this article put [Unit] After=/Requires=tailscale-wait-ready.service in ssh.socket.d/override.conf. That was the cycle-causer. Corrected 2026-07-20 — the socket override must contain only [Socket], and Requires= must never be used for this gate (use Wants=, so a failed gate degrades instead of blocking).

The socket override carries the bind address and nothing else — no [Unit] section at all:

# /etc/systemd/system/ssh.socket.d/override.conf
# The Tailscale-readiness gate lives on ssh.service, NOT here.
[Socket]
ListenStream=
ListenStream=<TAILSCALE_IP>:22

The gate goes on ssh.service, using Wants= rather than Requires=:

# /etc/systemd/system/ssh.service.d/tailscale-wait.conf
# Uses Wants (not Requires) to avoid ordering cycles.
[Unit]
After=tailscale-wait-ready.service
Wants=tailscale-wait-ready.service

Reload + restart:

systemctl daemon-reload
systemctl enable tailscale-wait-ready.service
systemctl restart ssh.socket
ss -tlnp | grep :22   # verify bound to Tailscale IP

# Prove no ordering cycle was introduced — MUST be empty:
journalctl -b | grep -i "ordering cycle\|Job ssh.socket/start deleted"

Verify before you reboot

A cycle is silent until the next boot. After any change here, confirm the socket override has no [Unit] section and the cycle grep above is empty — before rebooting a host whose only access path is SSH.

cat /etc/systemd/system/ssh.socket.d/override.conf   # expect [Socket] only

Evolution of this fix

  • 2026-05-19 v1 — After=tailscaled.service + BindsTo=tailscaled.service. Worked initially but caused a shutdown-time ordering cycle.
  • 2026-05-23 v2 — BindsTo swapped for Requires to break the cycle. Fixed the cycle but did not wait for tailscale0 to actually have an IP — just for tailscaled to be active. Hosts continued losing SSH after some reboots (intermittent, depending on whether the race won).
  • 2026-05-31 v3 — Added tailscale-wait-ready.service to gate on the interface having an address, but attached the gate to ssh.socket. Correct readiness signal, wrong attachment point — this is the revision that re-armed the ordering cycle.
  • 2026-07-20 v4 (current) — Same readiness service, gate moved to ssh.service with Wants=; ssh.socket.d/override.conf reduced to [Socket]-only. This matches what is actually deployed fleet-wide and is the canonical fix.

Do NOT use BindsTo — and do not swap it for Requires on the socket

BindsTo=tailscaled.service creates a systemd ordering cycle during shutdown: basic.target → sockets.target → ssh.socket → tailscaled.service → basic.target. Systemd breaks the cycle by deleting jobs unpredictably, which can prevent ssh.socket from starting on the next boot.

Superseded 2026-07-20: this callout previously advised "use Requires= for startup ordering." That is the v2 fix, and it is still wrong — Requires= on ssh.socket carries an implicit ordering relationship that re-creates the cycle. The dependency does not belong on the socket at any strength. Put the gate on ssh.service with Wants=, and leave ssh.socket.d/override.conf as [Socket]-only.

Affected Hosts

Ubuntu hosts locked via the tailscale role (ssh_only_ubuntu task, formerly configure_tailscale_ssh_only.yml): majorlinux, dcaprod-hetzner, tttpod-hetzner, majortoot-hetzner.

The Ubuntu playbook shipped the cycle pattern until 2026-06-07

Despite the 2026-06-04 resolution above, configure_tailscale_ssh_only.yml in the repo kept deploying the [Unit] Requires=tailscale-wait-ready.service gate on ssh.socket (the cycle-causer) and never added the ssh.service gate — so re-running it re-armed the ordering cycle. Caught 2026-06-07: it clobbered majorlinux's hand-fix, and majortoot-hetzner was found already armed with the latent cycle (would have lost SSH on its next reboot). Both restored/defused; playbook corrected in MajorAnsible e0d35aa (gate on ssh.service, dependency-free socket).

Fleet audited & reconciled 2026-06-07: dcaprod-hetzner + tttpod-hetzner had the dependency-free socket already but were missing tailscale-wait-ready.service (their ssh.service gate referenced a non-existent unit → inert → latent bind race, not a cycle); the corrected playbook was applied to both, deploying the service and activating the gate. teelia uses Tailscale SSH (no sshd, ssh.socket/ssh.service disabled) — immune to Race 1. Correction 2026-07-20: it is NOT immune to Race 2 — Tailscale SSH is served by tailscaled, so a Race 2 failure locks you out entirely with no sshd fallback. teelia and tttpod were both found missing the Race 2 drop-in. All Ubuntu hosts now run the same pattern: dependency-free ssh.socket bind + ssh.service readiness gate + tailscale-wait-ready.service.

Fedora hosts are NOT automatically immune (corrected 2026-06-07)

The firewalld method (configure_tailscale_ssh_only_fedora.yml) binds sshd on 0.0.0.0:22 and enforces Tailscale-only via the firewall, so it has no dependency on the Tailscale address — unless a host also carries a leftover manual ListenAddress <tailscale-ip> drop-in (/etc/ssh/sshd_config.d/tailscale-only.conf) from the pre-firewall lockdown. Then sshd.service hits the same boot bind-race (Bind to port 22 on <ts-ip> failed: Cannot assign requested address) and flaps every reboot. Hit on majordiscord 2026-06-07; fixed by removing the redundant drop-in (firewall stays the enforcing layer). The Fedora playbook now removes it automatically (MajorAnsible b4a9090).

Race 2: tailscaled Starts Before Network Is Online (All Hosts)

Symptom

Host reboots but never appears on Tailscale. tailscale ping times out entirely. SSH is dead because Tailscale never connects. The host is up (accessible via provider console) but isolated from the Tailscale network.

Cause

tailscaled.service ships with After=network-pre.target, which fires before the network interface has an IP. On VPS hosts (especially Hetzner), the interface can take several seconds to come online. Tailscale starts, sees no network (SetNetworkUp(false), link state: defaultRoute= ifs={} v4=false v6=false), fails DNS bootstrap and DERP relay connections, and gets stuck — never retrying.

Diagnosis

# From Hetzner console or another access method:
journalctl -b -u tailscaled | grep -E "SetNetworkUp|link state|error|DERP"
# Look for:
#   magicsock: SetNetworkUp(false)
#   link state: interfaces.State{defaultRoute= ifs={} v4=false v6=false}
#   health: Tailscale could not connect to any relay server

Fix

Deploy a systemd drop-in to wait for full network connectivity:

# /etc/systemd/system/tailscaled.service.d/override.conf
[Unit]
After=network-online.target
Wants=network-online.target

Then reload and restart:

systemctl daemon-reload
systemctl restart tailscaled

Affected Hosts

All hosts where Tailscale is the primary access path. Particularly impactful on VPS hosts with slow interface bringup. Both Fedora and Ubuntu hosts are affected.

Race 2 hits Tailscale-SSH hosts hardest — they have no sshd fallback

Race 1 is an sshd concern, so a host running Tailscale SSH (no sshd) is immune to it. Race 2 is the opposite: it is worse there. Tailscale SSH is served by tailscaled itself, so if tailscaled fails to come up, there is no second door — recovery requires the provider's web console.

An earlier revision of this article said teelia was "immune to both races." That was wrong for Race 2, and is likely why the drop-in was never deployed there. Audited 2026-07-20: /etc/systemd/system/tailscaled.service.d/ is absent on teelia and tttpod.

teelia is the sharpest case on the fleet — ssh unit disabled, sshd unit not-found, all access via Tailscale SSH, out-of-band recovery only via the DigitalOcean droplet console. Verify every Tailscale-SSH host has the Race 2 drop-in:

ls /etc/systemd/system/tailscaled.service.d/override.conf

Prevention

  • Set root passwords on all VPS hosts for emergency console access
  • The tailscale role deploys all fixes automatically (run via tailscale.yml / site.yml):
  • network_wait task — tailscaled network-online dependency (all hosts)
  • ssh_only_ubuntu task — dependency-free ssh.socket bind + ssh.service readiness gate + tailscale-wait-ready.service (Ubuntu group)
  • ssh_only_fedora task — firewalld Tailscale-only lockdown; removes any leftover ListenAddress drop-in (Fedora group)

References