Skip to content

Changing ansible_port Breaks Every Control Node's known_hosts (the ssh-askpass Red Herring)

The Problem

You change a host's ansible_port (or ansible_host) in host_vars, verify it works, merge it — and later that host is UNREACHABLE in every playbook, with an error that points at something entirely unrelated:

TASK [Gathering Facts] *********************************************************
[ERROR]: Task failed: Data could not be sent to remote host "10.0.0.42". Make sure
this host can be reached over ssh: ssh_askpass: exec(/usr/libexec/openssh/ssh-askpass):
No such file or directory
Host key verification failed.

fatal: [myhost]: UNREACHABLE! => {"changed": false, "unreachable": true}

Meanwhile plain ssh myhost from the same shell works fine.

ssh-askpass is not the problem, and installing it will not fix this. That line is two steps downstream of the real cause.

Why It Happens

SSH keys known_hosts entries by host and port. These are two distinct identities:

10.0.0.42                 <- the entry you already have (port 22, implicit)
[10.0.0.42]:2222          <- the entry the new ansible_port needs

The moment ansible_port changes, your existing entry becomes valid but useless. SSH now considers the target an unknown host, and unknown hosts trigger an interactive trust prompt.

Ansible runs without a TTY, so the prompt can't be shown. SSH falls back to $SSH_ASKPASS — a GUI helper that is not installed on a headless box — and that failure is what surfaces. The final Host key verification failed is the honest error; the askpass line is noise generated on the way there.

So the message you get names a missing binary, when what actually happened is "I don't recognise this host and couldn't ask you about it."

The Second Fault It Hides

Fix the known_hosts entry and you may immediately hit a different error:

Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

This is common in one specific case: the managed host is also a control node. A machine that runs playbooks against other hosts never needs to SSH to itself — so its own key is frequently absent from its own ~/.ssh/authorized_keys, while every peer's key is present. Changing the connection path can be the first thing that ever exercises the loopback.

Check the machine's own key against its own authorized_keys before assuming the key deployment is broken fleet-wide.

The Fix

1. Add the correct host:port entry — after verifying the key, not before.

If you have shell access on the target, compare against its own host key file. This is the strongest verification available; it makes an interposed host impossible rather than merely unlikely:

# on the target
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

# on the control node
ssh-keyscan -p 2222 -t ed25519 10.0.0.42 2>/dev/null | ssh-keygen -lf -

Only if the fingerprints match, append it:

cp ~/.ssh/known_hosts ~/.ssh/known_hosts.bak-$(date +%Y%m%d-%H%M%S)
ssh-keyscan -p 2222 -t ed25519 10.0.0.42 >> ~/.ssh/known_hosts
ssh-keygen -F '[10.0.0.42]:2222'     # confirm it resolves

Never pipe ssh-keyscan straight into known_hosts without that comparison — it trusts whatever answers.

2. If the host is its own control node, authorize its own key.

Codify it rather than hand-editing, so a rebuild doesn't silently undo it:

# configure_ansible_user.yml
ansible_authorized_pubkeys:
  - "ssh-ed25519 AAAA... ansible@example"
  - "ssh-ed25519 AAAA... admin@controlnode-a"
  - "ssh-ed25519 AAAA... admin@controlnode-b"   # each control node authorizes its own key

Confirm the authorized_key task does not set exclusive: true before adding to a shared list — with exclusive, a key missing from the list gets stripped fleet-wide on the next run, which can lock a control node out of every host at once.

How to Diagnose This

Get the exact command Ansible ran — the port is right there in it:

ansible-playbook site.yml --limit myhost -vvvv 2>&1 | grep 'SSH: EXEC'
# ... -o Port=2222 ...   <- the port your known_hosts entry probably doesn't cover

Then check the three things in order:

grep -c '2222' ~/.ssh/known_hosts               # 0 = fault 1 confirmed
ssh-keygen -F '[10.0.0.42]:2222'                # empty = not trusted
stat -c '%y' ~/.ssh/known_hosts                 # older than the port change = never added

A known_hosts mtime that predates the host_vars commit is conclusive: the entry was never written on this machine.

Reproduce it without Ansible, which strips away all the module noise:

ssh -p 2222 -o BatchMode=yes -v user@10.0.0.42 true

Why This Gotcha Is Invisible

Both faults live in per-control-node state — ~/.ssh/known_hosts and ~/.ssh/authorized_keys on one specific machine. Neither is in the repo, neither syncs, and neither is visible to any other host.

That means a change can be genuinely "verified end-to-end" and still be broken:

  • Control node B accepts the new host:port key when it first connects, and passes.
  • Control node A, which never ran the verification, keeps failing forever.
  • If the target is also a control node, a run from B succeeds because B's key is in the target's authorized_keys — while the target still can't reach itself.

A host_vars change that alters the connection path must be verified from every control node, not just the one that made the change. Host-key trust and self-authorization are per-machine facts; the commit that triggers the breakage carries neither of them.