deploy

Imported from DanielH2018/server (.claude/skills/deploy/SKILL.md). Install upstream with npx skills add DanielH2018/server --skill deploy. Copyright stays with the author.

Deploy a service using Ansible.

If the user provided a service name as an argument, use it directly. Otherwise ask which service to deploy.

In the post-merge path, do not ask anything. When this deploy is the follow-through on a PR that just merged (CLAUDE.md → After a PR Merges — Pull, Deploy, Verify), the service is already determined by the merged diff and the user has already asked. Skip step 2's dry-run question and go straight to the deploy, from /home/ubuntu/server on master rather than from a worktree. Everything else below — the platform split, the lock, the verification gate — is unchanged.

First, determine the platform — the verification step differs and the Docker one is dead on the cluster nodes:

• Role under ansible/roles/k8s/<service>/, entry has platform: k8s in host_vars/daniel-box.yml → k3s workload (this is nearly everything).
• Role under ansible/roles/containers/<service>/, entry in host_vars/daniel-pi.yml → Docker on the Pi (docker-proxy, wg-easy, glances, autoheal).

daniel-server and daniel-box have had no Docker since 2026-08-14 — never verify a deploy there with a docker command; it doesn't exist on those hosts.

Checking a k8s change without deploying it

Three modes, and they check genuinely different things — reaching for the wrong one is how a manifest bug reaches production.

--dry-run renders to a temp dir, applies with --dry-run=server, and discards the temp dir. Nothing is staged, applied, patched or rolled. It runs unlocked, because it mutates nothing. It does not catch scheduling, PVC binding, probe or rollout behaviour — those need a real deploy.

Two limits worth knowing before you trust a green dry run:

• It refuses the roles named in k8s_dry_run_unsupported — count them with grep -A20 "^k8s_dry_run_unsupported:" ansible/inventory/group_vars/all.yml; don't hand-maintain the number, it read "~17" against a real 15 for two commits. Roles that mutate outside roles/k8s/manifests (sidecar ConfigMaps built with kubectl create, netpol-probe Jobs, exec -i into a live pod) would half-apply, so deploy.yml fails fast and names them. ansible/tests/deploy/test_k8s_dry_run.py re-derives the list from the role sources so it cannot drift.
• A brand-new service is only half-checked. volume-claim is skipped (it is a dependency of 25 roles and mutates), and nothing at admission verifies that a referenced PVC exists — so the Deployment validates while the volume is never proven provisionable.

Steps:

1. Confirm the service name matches a role, and note which of the two trees it's in.

2. Ask if they want a dry run first. For a k8s workload that means --dry-run, which is the only mode that shows the manifests to an API server; --check answers a different question and is the right choice only when the doubt is about task wiring.

3. If dry run: ./scripts/deploy.sh --tags "<service>" --dry-run (or --check, per step 2)

4. If dry run passes or they skip it: ./scripts/deploy.sh --tags "<service>" (add -e target=daniel-pi for a Pi service)

   Deploy through scripts/deploy.sh, not ansible-playbook directly — it takes /var/lock/server-git-tree.lock, the same lock gitops-deploy.service and the secret-rotate cron use, long enough to copy HEAD into a snapshot worktree, then runs the playbook from that snapshot under /var/lock/server-deploy-<tag>.lock, one per service. So deploys of the same service serialize and deploys of different services do not. A Pi deploy takes both too, even though the writes land on the Pi. --check runs unlocked, from the working tree. The snapshot is of HEAD: an uncommitted edit is not deployed. Exit 75 means a lock stayed busy and nothing was deployed — that is not a playbook failure.

5. Verify it actually came up healthy — Ansible reporting ok/changed only means the playbook ran, not that the workload is up (it can apply cleanly then crash-loop or fail its probes).

   • k3s: uv run python scripts/diagnostics/probe.py health <service> is the primary check — allow-listed, k8s-native. Exit 0 only when the rollout is fully complete (observed generation caught up, every replica updated/ready/available) and no container restarted in the last 180s; an unreadable restart timestamp counts as recent (fails closed). That restart window is exactly what kubectl rollout status can't see — readiness flips a Deployment Available before a bad liveness probe starts killing it, so a rollout-status check alone can report green on a crashlooping pod.
     • On failure, drill down: kubectl -n <namespace> rollout status deployment/<service> --timeout=120s, kubectl -n <namespace> get pods -l app=<service>, kubectl -n <namespace> describe pod <pod>, and kubectl -n <namespace> logs <pod> --tail=50. These verbs are allow-listed and read-only, so they run without a prompt.
     • A pod that stays Running but never becomes ready is a probe failure, not a deploy failure — read describe's Events.
     • If a ConfigMap/Secret change appears not to have taken effect, check whether the Deployment carries a checksum/config pod annotation; without it the pod isn't rolled. Also note kubectl apply leaves stale Secret keys behind — a key removed from the manifest persists live until patched out.
   • Docker (Pi only): uv run python scripts/diagnostics/probe.py health <service> --docker — exit 0 = running + healthy; allow-listed. --docker inspects the local Docker daemon, and the Pi's is remote, so run it over ssh (ssh daniel-pi ...) or verify via the Pi's Uptime Kuma monitor instead.
   • For a config-only run (--skip-tags deploy), the workload isn't recreated, so this is just a liveness check, not a deploy verification.

6. Report the result, including the verification line. If the gate fails, surface the failing probe/event and pull recent logs (kubectl logs, or uv run python scripts/diagnostics/probe.py loki-query '{container="<service>"}') before declaring success.

Run all commands from /home/ubuntu/server. Always go through uv run — bare ansible-playbook (the uv-tool shim) lacks the module deps and fails. For a service on the Pi, add -e target=daniel-pi (deploy.yml defaults hosts: to the local hostname — --limit alone matches nothing).

The command reference

The bare ansible-playbook forms are what the wrapper runs. They work, but they have none of the locks, the snapshot, the tag check or the staleness check — use one only when you deliberately want that.

# Deploy a specific service
./scripts/deploy.sh --tags "<service-name>"

# Target the Pi. NB `-e target=`, NOT `--limit` — the play's hosts: defaults to the local
# hostname, so --limit daniel-pi matches zero hosts. The Pi is ansible_connection=ssh, so
# this reaches it from either node. `-e target=` a LOCAL-connection host (either cluster
# node) and the tasks run on the machine you typed it on — see ansible/inventory/hosts.ini.
uv run ansible-playbook ansible/deploy.yml --tags "<service-name>" -e target=daniel-pi

# Deploy everything
uv run ansible-playbook ansible/deploy.yml

# Check mode (task wiring only — the apply is skipped, no API server is involved)
uv run ansible-playbook ansible/deploy.yml --tags "<service-name>" --check

# Validate the k8s manifests against the live API server without applying them
./scripts/deploy.sh --tags "<service-name>" --dry-run

# Config-only: render dirs/templates/host config WITHOUT touching the container.
# Every container-role task is block-tagged config/deploy/cron, and tags UNION in Ansible,
# so scope with --skip-tags. `--skip-tags config` is NOT supported — the registered
# config-change facts feed docker_deploy's recreate decision.
uv run ansible-playbook ansible/deploy.yml --tags "<service-name>" --skip-tags deploy

# Edit encrypted secrets
sops ansible/vars/secrets.yml

# List the services --dry-run refuses to cover
grep -A20 "^k8s_dry_run_unsupported:" ansible/inventory/group_vars/all.yml

# Trigger a GitOps tick now instead of waiting for the 10-min timer (daniel-box only).
# Runs the identical code path the timer runs — there is no dry-run mode.
./scripts/deploy_tools/gitops_tick.sh

# Initial server setup. The first-host bring-up ORDER (uv -> SOPS onboarding -> this) is in
# ansible/README.md
uv run ansible-playbook ansible/initial_setup.yml

Why deploy.sh rather than the playbook

It takes two kinds of lock (ADR-0017). /var/lock/server-git-tree.lock — the same lock gitops-deploy.service (10-min timer) and the weekly secret-rotate cron hold — guards the local git tree every deploy renders from, which gitops-deploy rewrites with a git pull mid-run; deploy.sh holds it only to copy HEAD into a detached worktree under /tmp/homelab-deploy-snapshots/. It then releases that and holds one /var/lock/server-deploy-<tag>.lock per service across the playbook, which is what stops two deploys of the same service racing. A -e target=daniel-pi deploy takes both.

Because the snapshot is of HEAD, an uncommitted edit is not deployed — commit first.

Six of its non-zero exits mean nothing was deployed, and each is a resume point rather than a failure. The seventh, 20, is the inverse: the playbook ran and changes are live.

Exit 3 is --changed's answer, and it is the operator path: --changed refuses a broad change rather than guessing. The TICK asks a second question first -- deploy_tags.py narrow <old> <new>, which maps a deploy-plane range to the services it reaches and exits 3 only when it cannot. Running narrow by hand is read-only and prints its derivation; it does not change what --changed or deploy.sh do.

Exit 4 exists because a stale tree renders stale templates and reverts live config while every repo-side check still reads green (scripts/deploy_tools/deploy_staleness.py). It runs ahead of --check and --dry-run too, since a green dry run against a stale tree is itself the misleading signal. Being ahead of master is normal branch work and is never refused.

With --tags, only a commit reaching those tags refuses. deploy.sh hands the tag list to the gate, which classifies every path in HEAD..origin/master with the deployer's own mapper: a path reaching one of the tags, or any broad path (shared templates, ansible/inventory/, the setup plane), refuses and the message names the commits and paths responsible. A tail that touches only other roles prints one line saying how many commits behind the tree is and that none of them reach the tags, and the deploy proceeds. Without --tags the deploy is unscoped, so any commit behind refuses — the rule this guard has always had. The narrowing matters because the GitOps deployer fast-forwards to the newest GREEN commit in its range rather than to the tip, so the primary checkout is legitimately behind a pending tip while every landing deploys from it.

Exit 4 is decided before exit 2. deploy.sh asks whether the tree is stale before it validates --tags, so a stale tree carrying a tag it does not recognise reports 4, not 2 (issue #1566). A tag check against a stale tree answers about the wrong tree: the first landing of a new role reads as a tag miss until the tick fast-forwards the merge commit, and land.sh retries a stale tree while it reports a tag miss as a failed deploy.

Read the second paragraph of an exit 4 before you rebase. When the deployer has not fast-forwarded for longer than four ticks — its own behind_since marker, whose stamp any tick that moved the tree renews — the refusal appends a line saying so: the tree is behind because the PRIMARY checkout is parked, not because this worktree is stale, and rebasing here deploys nothing. The repair is the deployer's — journalctl -t gitops-deploy names the skip reason. The SessionStart banner carries the same decision (scripts/lib/deployer_park.py is the one copy of it), but only reaches a session as it opens; this reaches one that has been running for an hour (issue #1429).

Exit 2 exists because Ansible itself exits 0 on an unmatched tag, so the wrapper checks tags against containers_list first (scripts/deploy_tools/deploy_tags.py). --skip-tag-check bypasses it.

Exit 20 exists because ansible-playbook's own codes collide with the four above: it returns 2 on a failed host, 3 on an unreachable one and 4 on a parse error. deploy.sh returned that status verbatim until 2026-09-02, so a play that applied its manifests and then failed on a post-apply assert exited 2 and read as the tag miss (issue #840). Every non-zero playbook status is now collapsed onto 20; ansible's own number is printed on stderr rather than returned.