Skip to content

tailscale cert Can Silently Return the Stale Cached Cert Instead of Renewing

🛑 Problem

A cron job runs tailscale cert <domain> monthly to renew a MagicDNS TLS certificate. It exits 0 and prints Wrote public cert to ... every time. Everything looks healthy — until the certificate actually expires and every HTTPS-dependent service in front of it (Caddy, nginx, whatever's terminating TLS) starts failing hard, with logs showing an expired certificate error, not a renewal failure. There was never a warning. The cron job "succeeded" the entire time.

🔁 Why This Happens

tailscale cert is supposed to fetch a fresh Let's Encrypt certificate through Tailscale's HTTPS certificate feature and write it out. In practice (tracked upstream as tailscale/tailscale#8725 and #16788), the command can hit its local certificate cache under /var/lib/tailscale/certs/ and simply return the cached cert it already has — including a cert that's stale or nearly expired — instead of forcing a genuine reissuance. The exit code is still 0. The Wrote public cert to <file> message is still printed, because a write did happen — it just wrote the same old bytes it already had.

Nothing about the command's own output distinguishes "renewed" from "handed back what was cached." A script that trusts the exit code, or greps stdout for "Wrote public cert," will report success on every single run, genuine renewal or not.

Important

Exit code 0 and reassuring stdout are not evidence of renewal for this command. The only reliable signal is whether the certificate's own notBefore timestamp actually moved.

🔍 Diagnosis

Compare the certificate's notBefore date before and after a renewal attempt:

openssl x509 -in /path/to/cert.crt -noout -startdate
# notBefore=Aug  1 00:00:00 2026 GMT

Run tailscale cert <domain> again, then check notBefore a second time. If it's unchanged, the command returned the cached cert — no real renewal occurred, regardless of exit code or printed message. This is the only trustworthy check; do not infer success from notAfter alone (a cert extended in the cache but never actually reissued can still show a plausible expiry that simply hasn't moved either).

On this fleet the bug surfaced concretely on majorlab: a monthly cron line (tailscale cert majorlab.tail7f2d9.ts.net && cp ...) had been silently returning the same cached cert for multiple scheduled runs — including a run within days of the actual expiry window — with zero indication anything was wrong until the expiry itself became imminent.

✅ Fix

Force genuine reissuance by deleting the cached cert and key before every renewal attempt, so there is nothing for the command to hand back except a freshly issued cert — and verify success by comparing notBefore, not by trusting the exit code.

#!/usr/bin/env bash
set -euo pipefail

DOMAIN="majorlab.tail7f2d9.ts.net"
CERT_FILE="/etc/caddy/majorlab.crt"
KEY_FILE="/etc/caddy/majorlab.key"
LOG="/var/log/tailscale-cert-renew.log"

OLD_NOTBEFORE=$(openssl x509 -in "$CERT_FILE" -noout -startdate 2>/dev/null | cut -d= -f2 || echo "none")

# Delete the cached cert/key AND any prior output copy — both can be sources
# of the stale return.
rm -f /var/lib/tailscale/certs/"$DOMAIN".* "$CERT_FILE" "$KEY_FILE"

tailscale cert --cert-file="$CERT_FILE" --key-file="$KEY_FILE" "$DOMAIN"

NEW_NOTBEFORE=$(openssl x509 -in "$CERT_FILE" -noout -startdate | cut -d= -f2)

if [ "$OLD_NOTBEFORE" = "$NEW_NOTBEFORE" ]; then
    echo "$(date): FAILED — notBefore unchanged, tailscale cert returned stale cache" | tee -a "$LOG"
    mail -s "[majorlab] Tailscale cert renewal FAILED" marcus@majorshouse.com < "$LOG" || true
    exit 1
fi

echo "$(date): renewed OK ($OLD_NOTBEFORE -> $NEW_NOTBEFORE)" >> "$LOG"
systemctl reload caddy

Two things matter beyond the delete-then-renew pattern itself:

  • Verify, don't trust. The notBefore comparison is the actual check; the script fails loudly (and emails) if it doesn't move, rather than silently repeating the same no-op every month.
  • Delete every copy, not just the Tailscale-internal cache — an output copy sitting in /etc/caddy/ (or wherever the consuming service reads from) can independently mask the problem even if the internal cache behaves.

Deployed on majorlab as /usr/local/sbin/tailscale-cert-renew.sh, installed root:root 750, and run weekly by a systemd timer (tailscale-cert-renew.timer) rather than cron.

Updated 2026-10-04. This originally ran from a monthly root cron as /root/scripts/tailscale-cert-renew.sh. It was merged with a second, weekly job that had been added alongside it, and the cron line was removed. The deployed version keeps the technique above and adds three things: it only renews when the certificate has fewer than 30 days left (so a weekly schedule doesn't reissue every week), it checks the web server's config is valid before reloading, and it leaves the live certificate untouched until the new one has passed every check.