Skip to content

Fleet Log Aggregation — Loki + Grafana + Alloy Setup and Runbook

Centralized, searchable logs for the fleet — additive to the per-host Logwatch digest emails (which stay in place), not a replacement for them. Project plan and phasing live in Fleet-Log-Aggregation; this article is the reusable how-it-works and how-to-fix-it reference.

Architecture at a glance

Each fleet host runs Grafana Alloy as a native systemd service, tailing journald and pushing to Loki (Docker, majorlab) over Tailscale. Grafana (Docker, majorlab) queries Loki and is served publicly at logs.majorshouse.com by majorlab's host-level Caddy. Nothing in the log path is publicly exposed except Grafana's login page.

journald ──> alloy (systemd, per host)
                │  push http://<loki-host>:3100/loki/api/v1/push  (Tailscale-only)
                ▼
             loki (docker, majorlab, :3100) ── 30d retention via compactor
                ▲
             grafana (docker, majorlab, 127.0.0.1:3005)
                ▲
             host Caddy ── https://logs.majorshouse.com

Agent choice note: the original plan said Promtail. Promtail is deprecated upstream (release line frozen at 3.6.x; successor is Alloy). We deployed Promtail to the canary for a few hours on 2026-07-03, then switched to Alloy the same day while the blast radius was one host. If you find references to a promtail role, they're historical — the alloy role removes promtail artifacts from any host it touches.

Server side: loki-stack on majorlab

Compose stack at /root/loki-stack/ (canonical copies in the Claude project folder MajorInfrastructure/loki-stack/):

  • grafana/loki:3.7 — single-binary mode, filesystem storage, TSDB v13 schema. Retention is compactor-based: retention_enabled: true + delete_request_store: filesystem in compactor, retention_period: 720h in limits_config (the retention period lives in limits_config in Loki 3.x, not under compactor).
  • grafana/grafana:13.0 — bound to 127.0.0.1:3005 (3000 was taken by umami; 3001 kuma, 3003 open-webui, 3004 forgejo). Admin-only auth, sign-up disabled, password in Vaultwarden. Loki datasource provisioned from grafana-provisioning/datasources/loki.yml (http://loki:3100, container DNS on the compose network).
  • Loki's push port is published on two host IPs: the Tailscale IP (for agents) and loopback (for local checks and Kuma). Never 0.0.0.0.
  • Both images pinned to minor tags: Watchtower picks up patch releases, major bumps stay a deliberate act.
  • Host Caddy entry: logs.majorshouse.com { reverse_proxy :3005 }. DNS was already covered by the wildcard record.

Agent side: the alloy Ansible role

~/MajorAnsible: roles/alloy + logging.yml. Originally targeted just the backups group (the fleet's restic-backup scope, 8 hosts) on the reasoning that it was already the fleet's "real servers" list. Corrected 2026-07-26: teelia was missing Alloy and it turned out to be a genuine oversight, not a deliberate exclusion like majorpi (Raspbian/LAN-only) or majorrig-wsl (workstation) — nothing in this project or Fleet-Backups ever reasoned about excluding teelia specifically, it just was never in backups (teelia has no restic/B2 vars configured, unrelated to logging). Adding teelia to backups directly would have also pulled it into backups.yml's restic role with no credentials, so a separate inventory group was added instead:

# inventory/inventory.yml
logging_only:
  # Hosts that should ship logs but aren't in `backups` (no restic vars, would
  # break backups.yml if added there). teelia: genuine gap found 2026-07-26.
  hosts:
    teelia:

logging.yml's hosts: line is now backups:logging_only (union pattern) — 9 hosts total. Confirmed via ansible-playbook logging.yml --limit teelia and Loki's host label values query, which now lists all 9: dcaprod, majordiscord, majorhome, majorlab, majorlinux, majormail, majortoot, teelia, tttpod.

Follows the restic_backup conventions: pinned binary from GitHub releases with a creates: guard, templated config + systemd unit, per-host overrides in host_vars.

  • Config is journald-only: loki.source.journal (max_age 12h) → loki.relabel (adds unit from __journal__systemd_unit, level from __journal_priority_keyword) → loki.write. Static labels: job=systemd-journal, host=<inventory_hostname>.
  • Per-host app-log pipelines later: append raw Alloy blocks via alloy_extra_config in host_vars/<host>/vars.yml.
  • Unit has MemoryHigh=250M / MemoryMax=350M — sized after measuring ~70MB RSS on majorlinux, the tightest-RAM host (1.9GB).
  • Alloy's own HTTP server binds loopback only (127.0.0.1:9080); readiness at /-/ready (note the dash — Promtail used /ready).

Deploy: ansible-playbook logging.yml --limit <host> (canary) or bare (fleet).

Dashboards (Phase 3, provisioned as code)

Three starter Grafana dashboards, provisioned via grafana-provisioning/dashboards/ (not click-ops) so they survive a Grafana restart cleanly:

  • Fleet Overview — log volume by host and error-level logs by host, 24h window.
  • Per-Host Logs — $host template variable (label_values(host)), error/warning trend plus a live logs panel for whichever host is selected.
  • SSH & Auth Failures — failed SSH logins by host (pattern match on sshd.service/ssh.service), fail2ban bans by host, raw fleet-wide logs.

Provider config points at a json/ subfolder of the existing provisioning mount, so no new Docker volume was needed — just files dropped alongside the existing datasources/loki.yml. Verify a dashboard loaded with docker logs grafana | grep 'finished to provision dashboards' (no accompanying error line) or curl -u admin:<pw> 'http://127.0.0.1:3005/api/search?query='.

Verification checklist (per host)

systemctl is-active alloy → active; curl 127.0.0.1:9080/-/ready → "Alloy is ready."; journalctl -u alloy -p err → empty. Then on majorlab: curl -G 'http://127.0.0.1:3100/loki/api/v1/label/host/values' should list the host, and a query_range on {host="<host>"} should return entries seconds old.

Gotchas (all hit on first deployment, 2026-07-03)

Loki image is distroless — Docker healthchecks are impossible. No /bin/sh, no wget/curl in grafana/loki:3.x. A compose healthcheck: fails with exec: "/bin/sh": no such file. Don't fight it: leave the service without a healthcheck and monitor externally (Uptime Kuma on /ready). Grafana's image is fine (has a shell).

/ready returns 503 for ~15s after Loki starts. "Ingester not ready: waiting for 15s after being ready" is the warmup window, by design. If a Kuma monitor is created (or fires) during a restart it will briefly show 503 — it self-heals on the next check. Not a fault; don't tune thresholds around it.

SELinux (Fedora hosts): label bind mounts before first compose up. Standard majorlab failure mode. All loki-stack bind mounts use :Z, and data dirs were additionally pre-labeled (chcon -Rt container_file_t) before first start. Verify with ls -Zd (want container_file_t) and confirm zero denials with ausearch -m avc -ts recent. Also pre-chown: loki runs as uid 10001, grafana as 472.

Grafana publishes no major-only image tags. grafana/grafana:12 → manifest unknown. Use minor tags (13.0); Loki does publish both (3, 3.7).

Uptime Kuma monitors must be added by hand. Kuma 2.x API keys are /metrics-only; monitor CRUD needs admin login via socket.io, and those creds are deliberately not in the Ansible vault. Monitors: https://logs.majorshouse.com/api/health and http://<loki-host>:3100/ready, 60s / 2 retries.

Check-mode artifact in the role: handlers that restart the service fail under --check because the unit was never really written. The restart handler carries when: not ansible_check_mode — keep that pattern in new roles.

Loki loses its port binding after a host reboot (Docker/Tailscale race) — hit 2026-07-07, recurred 2026-07-09, now fixed for real. A routine Fedora kernel-update reboot restarted every container fine except loki: it came back Up, but docker inspect loki --format '{{json .NetworkSettings.Ports}}' showed {} — no ports at all, despite docker-compose.yml explicitly mapping the Tailscale IP and loopback. Cause: Docker tries the Tailscale-IP bind before tailscaled has assigned the address to tailscale0. The failure mode varies: sometimes the container comes up port-less (07-07), sometimes it dies outright with Exited (128) / cannot assign requested address (07-09) — and in the latter case restart: unless-stopped never retries, because a network-setup failure at start isn't a container exit.

  • Immediate recovery: docker compose up -d --force-recreate loki — a plain restart or even up -d isn't enough; ports are only (re)applied at container creation, and a failed start can leave a half-wired endpoint (running but Ports: {}, no container IP) that only a recreate clears.
  • ~~Root-cause fix (2026-07-07)~~ — insufficient, kept as belt-and-suspenders: systemd drop-in wait-for-tailscale.conf with After=tailscaled.service / Wants=tailscaled.service. The 2026-07-09 reboot proved the trap: After= orders against tailscaled going active, not against tailscale0 actually having its IP — the same v2 mistake documented in Tailscale Boot Race Conditions. tailscaled went active at 09:05:12, Docker started at 09:05:16, and the bind still failed at 09:05:15 (container start) because the IP arrived later.
  • Root-cause fix (2026-07-09, race-free): net.ipv4.ip_nonlocal_bind = 1 via /etc/sysctl.d/99-tailscale-nonlocal-bind.conf — the kernel allows binding an address that isn't (yet) assigned, so Docker's bind succeeds no matter when Tailscale finishes. sysctl.d applies before any service starts. Full write-up: Docker Port Bind to a Tailscale IP Fails at Boot.

Uptime Kuma did correctly flag the outage (ECONNREFUSED on the Loki monitor, logged every minute from right after the reboot) — but it sat unnoticed for a while, so double-check notification routing on any monitor watching a Tailscale-IP-bound service, not just monitor existence. Full incident write-ups: majorlab Incident Log, 2026-07-07 and 2026-07-09.

Operations

  • Restart stack: cd /root/loki-stack && docker compose restart (expect the 15s /ready warmup).
  • Logs of the log system: docker logs loki, docker logs grafana, journalctl -u alloy on agents.
  • Disk: Loki data at /root/loki-stack/loki-data/ — 30d TTL, estimated 5–20GB at full fleet. Excluded from backups by design (regenerable, TTL'd — matches the caches-excluded precedent in Fleet-Backups).
  • Upgrade Alloy: bump alloy_version in roles/alloy/defaults/main.yml, delete /usr/local/bin/alloy on targets (the creates: guard skips existing binaries), rerun logging.yml.