Skip to content

Rotating a Shared Secret: Prove Every Consumer, Not Just the Push

Rotating a password, API key or webhook secret feels done the moment the new value is in the vault and the playbook ran green. It isn't. In my fleet, eight separate rotations "succeeded" and still broke something — and in almost every case the rotation's own verification said everything was fine. The longest one silently dropped every security alert for three days.

The Short Answer

A rotation is finished when every consumer has been proven working with the new value, through its real path, and the old value has been proven dead. Pushing the new value is one step of eight. The two failures that keep recurring:

  1. You don't know every place the secret is used. Searching your config repo tells you where a secret is managed, not where it is used.
  2. A running process doesn't re-read its config just because the file changed. The new value sits on disk while the process keeps sending the old one.

0. Before: Map Every Consumer

Write down every place the secret is used before you touch it. Your repo search is the start, not the answer. Also check:

  • Hand-written host config — .env files, docker run -e, compose files that live outside your automation, ~/.aws/credentials on every workstation.
  • Credentials saved through a web UI — these live in the app's own database: n8n credentials, notification settings in container managers, Nextcloud's Mail app, uptime monitors.
  • Workstation tooling — environment files sourced by your shell, desktop app configs, every mail client on every device.
  • Third-party cloud copies — some mail apps keep your mailbox password on their own servers to do push.

Two rules while you search:

  • Search by fingerprint, never by value. Hash the old value and compare it against candidate files, printing only "match / no match". Grepping for the secret itself puts it in your terminal history — and in any AI session transcript.
  • Check precedence. If a tool reads several sources (environment variable, then config file, then vault), a stale higher-priority copy silently wins. I once had a correct key in the vault being ignored on every call because an old environment variable outranked it — and the error looked exactly like a revoked key.

If the map took real work to build, save it as a per-secret list. My mailbox password now has one, built after a rotation missed two clients.

1. Create the New Value — Keep the Old One Alive If You Can

Most cloud providers (AWS, Backblaze B2, DigitalOcean, Hetzner) let a user hold two keys at once. Create the new key alongside the old one, switch consumers over one at a time, and you get no outage window at all.

Where that isn't possible — a single shared webhook header, a mailbox password — plan the order so the receiver and the senders change as close together as possible, and expect authentication failures in between.

2. Deploy It Everywhere on the Map

Run the playbooks for managed consumers; update hand-written and UI-stored ones by hand. A changed=0 on a host you expected to change is an alarm, not a pass.

3. Make Every Running Process Load It

This is the step that cost me three days of alerts. A file on disk is not what the running process uses. For each consumer, confirm the process that sends the secret started after the change:

# systemd service
systemctl show -p ActiveEnterTimestamp <service>

# container
docker inspect -f '{{.State.StartedAt}}' <container>

Traps I've hit:

  • docker compose up -d is a no-op when the compose file didn't change. My Wazuh manager got its new webhook secret in its config file, Ansible's handler ran docker compose up -d, nothing restarted, and the alert-forwarding process kept sending the old secret. Every alert was rejected for three days until an unrelated update restarted the container.
  • Apps with separate worker processes — Mastodon's web and Sidekiq processes both need a restart to read .env.production.
  • Apps that save their settings from memory — Obsidian's Local REST API plugin overwrites its data.json on its next save. Rotate the key while Obsidian is running and the plugin quietly writes the old key back. Quit the app first.
  • Desktop apps — a full quit and relaunch, on every machine that holds the value.
  • Mail clients keep retrying the old password, and enough failures will get your own IP banned by fail2ban.

4. Prove Each Consumer, End to End

One real request per sender, through the real path, confirmed at the receiving end: the alert actually arrives in your inbox, the upload actually lands in the bucket, the receiver's access log shows a success for that sender.

  • Beware endpoints that report success without checking auth. Obsidian's REST API returns HTTP 200 on / whether or not your key is valid — test a protected path.
  • Tick each consumer individually. "Most of them work" is how one silent failure hides behind several working ones.

5. Prove the Old Value Is Dead

Send one request with the old value and expect a rejection (401, InvalidAccessKeyId, an auth failure). Do it in a way that never prints the value.

6. Watch One Full Monitoring Cycle

Some consumers only run nightly or weekly — backups, prune jobs, daily checks. After one full cycle, look at the receivers for authentication failures from any source: reverse-proxy 401s, mail-server auth failures, fail2ban bans of your own addresses, systemd units in a failed state. An authentication failure nobody can explain is a consumer missing from your map.

7. Retire the Old Value Carefully

Deactivate, wait one monitoring cycle, then delete. And note any leftover copies of the old value — backups, .bak files, shell history, session databases — as dead, so a later audit doesn't mistake them for live credentials.

8. Write It Down

Log the rotation with the date, which consumers were proven, and anything you found that wasn't on the map. Update the per-secret list so the next rotation starts from a better map.

What This Checklist Would Have Caught

Every row is a real rotation in my fleet that broke something its own verification missed:

Secret What was missed Found by Step that catches it
Mail server password An automation tool's saved SMTP credential kept the old password; an alert workflow failed silently for about three weeks Later investigation 0, 4
Obsidian REST API key The running plugin would have written the old key back; the health endpoint returns 200 without auth Caught during the rotation 3, 4
Mailbox password A web mail app and a desktop mail client kept the old password — an hourly auth-failure loop for three days Mail-server log investigation 0, 6
Registrar API key An old environment variable outranked the vault "API key invalid" errors 0
AWS access key (Mastodon) Mastodon's .env.production wasn't in the repo, so the repo search said "no consumers" — every media upload failed Mastodon's own media health check, not the rotation 0, 4
AWS access key (workstation) One workstation's ~/.aws/credentials was never updated Security assessment three weeks later 0
Backblaze B2 key Rotated in the vault, never redeployed — a daily check job failed for almost eight hours Monitoring alert on the failed unit 2, 6
Alert webhook secret Wazuh's config was updated but the process was never restarted — every security alert dropped for three days Noticed by hand; fixed by an unrelated restart 3, 4

The pattern is the same every time: the verification checked the push, not the use.