Skip to content

Ansible: Ubuntu Reboot Detection Misses Kernel Upgrades

Problem

update.yml runs across the Ubuntu fleet, a kernel package is upgraded, but the executive summary reports No reboot needed — even though a reboot is genuinely required. Running uname -r on the host confirms it's still on the old kernel.

Example: majortoot had linux-image-6.8.0-117-generic installed on May 16 after a Tailscale update triggered needrestart, but the playbook kept reporting clean.

Root Cause

The standard check for Ubuntu reboot state is:

- name: Check if a reboot is required for Ubuntu servers
  ansible.builtin.stat:
    path: /var/run/reboot-required
  register: ubuntu_reboot_flag

/var/run/reboot-required is written by update-notifier-common's notify-reboot-required script, called by /etc/kernel/postinst.d/update-notifier when a kernel package is installed via apt.

The problem is needrestart. It runs after every apt invocation via a DPkg::Post-Invoke hook (apt-pinvoke -m u). In unattended mode (-m u), needrestart detects the pending kernel upgrade and calls announce_ver() in NeedRestart::UI::Ubuntu — but that function only prints to stdout. It does not call _write_reboot_file(). Only announce_ucode() (microcode upgrades) calls _write_reboot_file().

So the sequence is:

  1. apt installs kernel → notify-reboot-required creates /run/reboot-required ✅
  2. Some later apt run (e.g. Ansible installs Tailscale) → needrestart -m u runs → detects kernel mismatch → calls announce_ver() → prints to stdout (suppressed in Ansible) → does not recreate the sentinel file
  3. Next Ansible run: stat check finds no file → reports No reboot needed ❌

The /run filesystem is tmpfs and clears on reboot, but the sentinel file can disappear between reboots any time needrestart runs without recreating it.

Fix — Dual Check in update.yml

Add a parallel kernel comparison task after the existing stat check:

- name: Check running kernel vs installed kernel (Ubuntu)
  ansible.builtin.shell: |
    RUNNING=$(uname -r)
    INSTALLED=$(dpkg -l 'linux-image-[0-9]*-generic' 2>/dev/null \
      | awk '/^ii/{print $2}' \
      | sed 's/linux-image-//' \
      | sort -V | tail -1)
    if [ -n "$INSTALLED" ] && [ "$RUNNING" != "$INSTALLED" ]; then
      echo "KERNEL_MISMATCH"
    fi
  register: kernel_mismatch_check
  changed_when: false
  when: ansible_facts['os_family'] == "Debian"

Then update the host_summary Jinja2 template to OR both conditions:

{%- if ansible_facts['os_family'] == 'Debian' and (
    (ubuntu_reboot_flag is defined and ubuntu_reboot_flag.stat is defined and ubuntu_reboot_flag.stat.exists)
    or
    (kernel_mismatch_check is defined and 'KERNEL_MISMATCH' in (kernel_mismatch_check.stdout | default('')))
  ) -%}
  {%- set _ = parts.append('REBOOT REQUIRED') -%}

Caveat — Exclude WSL2 Guests

The kernel comparison above is structurally unsatisfiable on WSL2 and must be guarded, or every Debian-family WSL host reports KERNEL_MISMATCH forever.

A WSL2 guest does not boot a distro-packaged kernel — it runs Microsoft's:

$ uname -r
6.18.33.2-microsoft-standard-WSL2

Meanwhile dpkg -l 'linux-image-[0-9]*-generic' still reports whatever distro kernel the package manager installed. The two values can never converge, so RUNNING != INSTALLED is permanently true, and the summary prints REBOOT REQUIRED on every single run regardless of actual state.

Detect WSL from the kernel string and skip the comparison:

- name: Detect WSL guests (Microsoft kernel, no in-guest reboot path)
  ansible.builtin.set_fact:
    is_wsl: "{{ ansible_facts['kernel'] is search('(?i)microsoft') }}"

- name: Check running kernel vs installed kernel (Debian-family, non-WSL)
  ansible.builtin.shell: |
    ...
  when:
    - ansible_facts['os_family'] == "Debian"
    - not is_wsl

The RedHat path is different — the flag there is real

On a RedHat-family WSL guest the equivalent check is needs-restarting -r, and it is not a false positive. It reports on userspace restarts, so a systemd upgrade legitimately sets rc=1:

Core libraries or services have been updated since boot-up:
  * systemd
Reboot is required to fully utilize these updates.

Do not suppress that one. But note it cannot be cleared from inside the guest — reboot in a WSL2 shell does not restart the instance. The fix is Windows-side:

wsl --shutdown
wsl -d <distro>     # or simply open a new WSL shell

Because of that, the summary string should name the Windows-side action rather than just saying REBOOT REQUIRED, or the flag reads as stuck:

{%- if is_wsl | default(false) -%}
  {%- set _ = parts.append('REBOOT REQUIRED (WSL: wsl --shutdown on Windows host)') -%}
{%- else -%}
  {%- set _ = parts.append('REBOOT REQUIRED') -%}
{%- endif -%}

Restarting a WSL2 guest remotely can strand it. If the instance only starts when a user logs into the Windows host, a remote wsl --shutdown may leave it down — and unreachable to Ansible — until someone logs in physically. Confirm the host's startup behaviour before doing this over SSH.

Common Mistake — Comparing the Wrong dpkg Field

An initial version of this fix used $3 (the package version) and cut:

# WRONG — version field never matches uname -r
INSTALLED=$(dpkg -l 'linux-image-*-generic' | awk '/^ii/{print $3}' | sort -V | tail -1 | cut -d- -f1-4)
Field Example value
dpkg $3 (version) after cut 6.8.0-57.59
uname -r 6.8.0-57-generic

These formats never match. Every Ubuntu host permanently reports KERNEL_MISMATCH. Always use the name column ($2), strip the linux-image- prefix, and compare directly to uname -r.

Also use linux-image-[0-9]*-generic (not *-generic) to exclude the linux-image-generic meta-package from the sort.

Verification

Run against a known-pending host before and after reboot:

ansible-playbook update.yml --limit majortoot

Before reboot: majortoot: 0 pkg(s) upgraded | REBOOT REQUIRED
After reboot: majortoot: 0 pkg(s) upgraded | No reboot needed