Skip to content

Postfix Starts Before Its Milters at Boot, Failing the First Inbound Message

TL;DR — The stock postfix.service unit only orders itself after one milter (if any), not all of them, and milters that run as Type=simple are counted "started" by systemd the instant they're exec'd — not when their Unix socket actually exists. On a cold boot, Postfix can come up and accept a connection before OpenDKIM's or OpenDMARC's socket is ready, deferring that one message with 4.7.1 Service unavailable. Nothing is lost if milter_default_action = tempfail is set, but it pages every reboot. Fix: a systemd drop-in that adds After= for every milter and polls for their sockets before Postfix starts, fail-open so a milter that never comes up can't hang the mail server.

Symptom

  • A single alert fires shortly after a reboot: a milter-failure monitor (Grafana/Loki pattern match, a Netdata custom check, or just grepping mail logs) catches warning: connect to Milter service local:/run/opendkim/opendkim.sock: No such file or directory followed by milter-reject: CONNECT ... 4.7.1 Service unavailable - try again later.
  • It's always right at boot, always only the first inbound connection or two, and it self-resolves within a minute or two — by the time anyone looks, mail is flowing normally.
  • journalctl -b -u postfix and the milter units (opendkim.service, opendmarc.service, spamass-milter.service) all show "active" within the same second or two of each other.
  • If smtpd_milters has milter_default_action = tempfail (the recommended setting), nothing is actually lost — the sender's MTA retries and the message gets through a few minutes later. Without that setting, check carefully: a looser default action could mean the message was silently accepted unscanned, or bounced.

Why it happens

Postfix's own vendor unit only declares After= for whichever milter the package maintainer anticipated — often just one, or none at all. If you run multiple milters (DKIM signing, DMARC evaluation, spam filtering), systemd has no reason to start them before Postfix, and on a cold boot — when everything is launching in parallel and I/O is still catching up — Postfix frequently wins the race.

Even adding a correct After= line for every milter isn't enough by itself. Milters commonly run as Type=simple, which tells systemd the unit counts as "started" the moment the process is exec'd — not when it's actually ready to accept connections. A milter that forks, loads its config, and opens a Unix socket takes a little real time to do that. systemd doesn't know to wait for the socket; it just sees the process running and immediately starts the next unit in the After= chain. So Postfix can be told to wait for OpenDKIM "the unit," start right after it, and still beat OpenDKIM to having a working socket.

Diagnose (read-only)

# When did each service actually become active?
systemctl show postfix opendkim opendmarc spamass-milter -p ActiveEnterTimestamp

# What does the postfix vendor unit currently order itself after?
systemctl cat postfix | grep -A2 '\[Unit\]'

# Any milter connection failures logged this boot?
journalctl -b -u postfix | grep -i milter

If the ActiveEnterTimestamps for Postfix and a milter are within the same second, and the vendor unit's After= line is missing one or more of the milters you actually run, you've found it. A clean boot shows no connect to Milter service … No such file or directory line at all in that grep.

Fix

Add a systemd drop-in rather than editing the vendor unit directly, so package updates don't clobber it:

# /etc/systemd/system/postfix.service.d/milter-order.conf
[Unit]
After=opendkim.service opendmarc.service spamass-milter.service

[Service]
ExecStartPre=-/bin/sh -c 'for n in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do test -S /run/opendkim/opendkim.sock && test -S /run/opendmarc/opendmarc.sock && test -S /run/spamass-milter/spamass-milter.sock && exit 0; sleep 2; done; echo "milter sockets not ready after 30s, starting postfix anyway" >&2; exit 0'

Two parts, and both matter:

  1. After= for every milter you run, not just one. This fixes unit ordering but not readiness — see below.
  2. ExecStartPre polling for each milter's actual socket file, up to 30 seconds (15 × 2s), before Postfix is allowed to start. This is what closes the Type=simple gap: it waits for the thing that actually matters (a listening socket), not just "the process exists."

Deliberately fail-open. The - prefix on ExecStartPre and the trailing exit 0 mean this check can never itself keep Postfix down — if a milter never comes up, Postfix starts anyway after the 30s cap, exactly as it would have before this fix. milter_default_action = tempfail is what makes that safe: a still-missing milter just means messages get deferred, not dropped or passed through unscanned. Don't add this socket-wait pattern without also setting milter_default_action to something fail-safe — otherwise a slow milter can turn into either a stuck mail queue or an unscanned-mail leak, depending which way your default action leans.

Reload and apply:

systemctl daemon-reload
systemctl restart postfix

The fix only takes effect the next time Postfix actually starts — a daemon-reload alone doesn't retroactively fix an already-running instance's ordering. If you deploy this without restarting Postfix immediately, confirm the ordering at the next reboot rather than assuming it's live: check ActiveEnterTimestamp is newer than your deploy time, and watch journalctl -b -u postfix for the absence of the connection-refused warning.

Gotchas found while fixing it

  • No $ in the ExecStartPre command. systemd expands $ in unit files before handing the line to the shell, which can mangle a loop variable or break the one-liner in ways that are confusing to debug from the resulting (wrong) command line in the journal. Keep the inline shell script free of $-prefixed variables, or escape them doubled ($$).
  • daemon-reload is not enough to prove the fix landed. NeedDaemonReload=no and a clean systemd-analyze verify only confirm the unit file parses and is loaded — they say nothing about whether the running Postfix process picked up the new ordering, since it was already started under the old rules. Check ActiveEnterTimestamp against your deploy time to know which case you're in.
  • A 30-second cap is a judgment call, not a universal constant. It's long enough to cover a milter's normal startup time plus some boot-time I/O contention slack, short enough that a genuinely dead milter doesn't stall the mail server noticeably. Tune it to how slow your specific milters are to initialize, not the number used here verbatim.

Prevent a repeat

  • Verify across a real reboot, not just a service restart. systemctl restart postfix alone won't reproduce the parallel-startup race condition a cold boot creates — the milters are usually already up and settled by the time you manually restart just Postfix. The only real test is the next actual reboot.
  • If you don't already have milter_default_action = tempfail, set it before relying on this fix. It's the safety net that makes a slow or dead milter a deferred-mail problem instead of a lost-mail or unscanned-mail problem.
  • Add (or confirm you have) a log-pattern alert on the milter connection-refused signature, scoped so a single self-resolving boot-time blip doesn't page unnecessarily, but a milter that's actually stuck down does.