Imported from ZaganJade/Agents-Reviewer-Skills (hermes-skills/devops/zero-downtime-server-change-guard/SKILL.md). Install upstream with npx skills add ZaganJade/Agents-Reviewer-Skills --skill zero-downtime-server-change-guard. Copyright stays with the author (MIT).

Zero-Downtime Server Change Guard

Priority

This is a highest-priority guard skill. Treat it as priority -1: load it before any narrower deployment, Docker, nginx, database, swap, install, cleanup, or service-specific skill when the user asks to deploy, install, restart, reconfigure, optimize, clean, or otherwise change the server.

The purpose is to keep the VPS reachable and keep production services alive while making changes.

Overview

Use this skill for any server change that could affect uptime, resource usage, networking, SSH access, Hermes access, Docker, nginx, databases, swap, disks, firewall, or running services.

Do not jump directly into the requested command if the command can create load, restart services, consume disk, break port 22, break nginx, or disrupt Hermes. First run a quick guard preflight, then execute work in a safe order, then verify access and service health.

This skill is intentionally broad. It should be loaded even when another skill is also relevant, for example Docker deploys, Jadenode-style builds, Laravel deployments, nginx/TLS changes, database operations, swap changes, Tailscale changes, package installs, and cleanup.

When to Use

Load this skill whenever the user asks to:

• Deploy, build, rebuild, or init a project.
• Install, remove, upgrade, or configure a system package or service.
• Change Docker Compose stacks, containers, images, volumes, networks, or restart policies.
• Change nginx, Certbot, firewall, Tailscale, DNS, ports, reverse proxies, or TLS.
• Change swap, disk mounts, fstab, partitions, cleanup, journals, Docker cache, or storage layout.
• Change databases, queues, workers, cron jobs, systemd units, or monitoring.
• Run heavy jobs such as pnpm/npm builds, Composer installs, Docker builds, migrations, model downloads, or large file operations.

Do not skip this because the requested action looks simple. If it can make the server heavy or unreachable, this guard applies.

Non-Negotiable Safety Goals

1. SSH port 22 must remain reachable.
2. Hermes/agent access must remain available unless the user explicitly asks to restart or modify Hermes.
3. Public services should not be restarted all at once unless unavoidable.
4. Existing healthy containers should not be disturbed when unrelated.
5. Root disk must not be filled.
6. Swap and RAM must have enough headroom before heavy work.
7. nginx config must be tested before reload.
8. fstab/systemd/network/firewall changes must be validated before applying permanently.
9. Database/state volumes must not be deleted or pruned without explicit user approval.
10. If risk increases mid-task, pause heavy work and stabilize before continuing.

Guard Preflight

Before any risky change, capture current state:

date
uptime
free -h
swapon --show
df -hT / /mnt/disk2 2>/dev/null || df -hT /
ss -ltnp | grep -E ':(22|80|443)\b' || true
docker ps --format 'table {{.Names}}\t{{.Status}}'
systemctl --failed --no-pager || true

For heavy Docker/deploy work also run:

docker system df
docker stats --no-stream --format 'table {{.Name}}\t{{.MemUsage}}\t{{.CPUPerc}}'

For networking/nginx/firewall work also run:

sudo nginx -t 2>/dev/null || true
sudo ufw status verbose 2>/dev/null || true
sudo iptables -S 2>/dev/null | head -80 || true

Do not print secrets from .env, Docker env, credentials files, or API keys.

Priority Order for Changes

Execute changes in this order when possible:

1. Preserve access: confirm SSH port 22, current shell, Hermes session, and sudo access.
2. Check resource headroom: RAM, swap, root disk, load, Docker disk usage.
3. Backup or snapshot config files before editing: nginx sites, fstab, compose files, systemd units, env templates.
4. Validate generated config before applying.
5. Apply the smallest isolated change first.
6. Restart/recreate only the affected service, not the whole stack.
7. Verify the affected service.
8. Verify unrelated critical services are still healthy.
9. Continue to the next change only after the previous step is stable.
10. Final health check and concise report.

Resource Gates

Use these gates on Ikhsan's small VPS unless the live machine has clearly changed:

• RAM is small, so treat high load as a risk even if swap exists.
• Swap should be active and checked before build/deploy.
• Root disk should ideally have at least 4-6 GB free before Docker builds. Treat <2 GB free as a hard blocker for Docker/BuildKit/containerd deploys because layers may be unpacked under /var/lib/containerd/... on / even when the app/project directory is on disk2.
• If swap free is below roughly 1.5 GB before a heavy build, pause and inspect.
• If load is already high, wait or identify the process before starting new heavy work.
• Build frontend/backend services one at a time.

If a gate fails, do not improvise destructive cleanup. Prefer safe cleanup first:

sudo apt-get clean
sudo journalctl --vacuum-size=200M
docker builder prune -af

Never run docker volume prune without explicit approval.

Access Preservation

Before firewall/network/nginx/Tailscale/systemd changes, verify port 22 and active access:

ss -ltnp | grep ':22 '
systemctl is-active ssh || systemctl is-active sshd || true

Rules:

• Do not close or remap SSH port 22 unless the user explicitly asks and an alternate confirmed access path exists.
• Do not reload firewall rules that omit SSH.
• Do not make Tailscale/Funnel/Serve changes that replace the only management path without checking SSH.
• Do not restart Hermes or its panel unless the task is about Hermes and the user asked for it.

Docker / Compose Guard

Before changing a compose stack, prove which checkout actually owns the running container. Project names and duplicate directory names are not sufficient evidence:

docker inspect <container> --format '{{json .Config.Labels}}'
# Check com.docker.compose.project.config_files and project.working_dir.
readlink -f <candidate-repo>
git -C <active-working-dir> status --short
git -C <active-working-dir> log -1 --oneline

Treat the running container's Compose labels as authoritative. If multiple checkouts exist, do not edit or build from a candidate path until its realpath, commit, compose file, and .env ownership match those labels. Building an identically named production tag from the wrong checkout can silently replace the tag even while the old container remains healthy.

Before creating a deployment/CI fix branch from production main, prove local and remote state instead of jumping to git reset --hard:

git status --short --branch
git diff --stat
git diff --cached --stat
git fetch origin main
git rev-parse HEAD
git rev-parse origin/main
git rev-list --left-right --count HEAD...origin/main
git diff --name-status HEAD..origin/main

If the tracked worktree is clean and divergence is 0 0, create the branch directly. If local main is merely behind, use git switch main && git merge --ff-only origin/main; this stops on divergence without discarding files. Use git reset --hard only after proving what it would remove and explicitly preserving any tracked production diff. Untracked operational backups need separate handling because both git diff and git reset --hard ignore them.

Before changing the stack itself:

docker compose config --services
docker compose ps

Rules:

• For verification builds from a secondary checkout, use an isolated temporary tag (service-verify:temp) rather than the production :latest tag.
• Record the running container image ID and the newly built image ID before recreate; confirm they differ for the intended reason.
• Prefer docker compose up -d --no-deps <service> for isolated changes.
• When .env / env_file values change, Docker containers do not reload them automatically; recreate only the affected services with docker compose up -d --no-deps --force-recreate <service...> and verify the new env from inside the container without printing secret values.
• Before recreating a container that accepts runtime uploads (payment proofs, QRIS, avatars, documents), prove whether upload directories are bind-mounted/volumed. If files live only inside the container, copy them to a persistent host path first, add explicit mounts in compose, then recreate. Verify public URLs still return the original file bytes after recreate.
• Build one service at a time on low-memory hosts.
• Avoid docker compose down unless necessary; it may stop healthy dependencies.
• Avoid --remove-orphans until you understand which containers will be removed.
• Never delete volumes as a fix without explicit approval.
• Check logs after recreate.

nginx / TLS Guard

Before reload:

sudo nginx -t

After reload:

sudo systemctl reload nginx
curl -I http://127.0.0.1:<port>/ 2>/dev/null || true
curl -I https://domain.example/ 2>/dev/null || true

Rules:

• Do not reload nginx on a failed config test.
• Preserve existing working sites unless the requested change targets them.
• For gateway/API routes, do not add Basic Auth to API paths unless explicitly requested.

fstab / Disk / Swap Guard

Before fstab or mount changes:

sudo cp /etc/fstab /etc/fstab.bak.$(date +%Y%m%d-%H%M%S)
findmnt --verify --tab-file /etc/fstab || true

Rules:

• Activate new swap before disabling old swap.
• Never format or repartition disk2/root disk unless explicitly requested.
• Do not assume a file named swap is active; confirm with swapon --show.
• For fstab, prefer adding valid absolute mount targets and verify before reboot.
• On hosts where /mnt/disk2 is a bind alias, do not trust directory existence alone. Compare lsblk and findmnt -T /mnt/disk2 and prove the mount source is the actual secondary disk; an empty root-disk bind can make projects, Redis state, uploads, and swap appear deleted.
• Make bind aliases explicitly depend on the real disk mount (for example bind,x-systemd.requires-mounts-for=/data/disk) so systemd cannot mount the alias first during boot.
• If Docker reports a file bind source is “not a directory” after boot, inspect the source with stat before deleting anything. Docker may have auto-created a directory at an absent file path because the real disk was not mounted.
• After correcting a mount, activate swap and restore stateful dependencies before starting app services. Recreate only containers whose existing bind mounts still reference the pre-correction filesystem.
• When explaining an outage, distinguish proof from inference: simultaneous healthcheck timeouts show starvation, but claim OOM only with kernel OOM evidence; Power key pressed proves a virtual ACPI request but not who initiated it; PostgreSQL recovery after an unclean stop is usually a consequence, not the cause.

Database / State Guard

Rules:

• Do not delete DB containers, volumes, or data dirs to fix startup loops.
• Read logs first.
• For migrations, verify database connectivity and backup/rollback context.
• Run migrations after app container is stable, not during unrelated service builds.

Heavy Install / Build Guard

For package installs and builds:

1. Check resources first.
2. Install/build one major component at a time.
3. Prefer foreground commands with a generous timeout for bounded tasks.
4. For long bounded tasks, use background with notify_on_complete=true.
5. Verify the process did not OOM: check exit code, logs, free -h, and dmesg -T | tail if killed.

Common heavy tasks:

• docker compose build
• pnpm install / pnpm build
• npm install / npm run build
• composer install
• database migrations/seeds
• model downloads
• system package upgrades

Final Verification

After the change, run enough checks to prove the server is still safe:

uptime
free -h
swapon --show
df -hT / /mnt/disk2 2>/dev/null || df -hT /
docker ps --format 'table {{.Names}}\t{{.Status}}'
ss -ltnp | grep -E ':(22|80|443)\b' || true
systemctl --failed --no-pager || true

Also verify affected public/local endpoints with curl or service-specific checks.

Stop Conditions

Stop and stabilize before continuing if:

• SSH port 22 is not listening.
• Root disk is near full.
• Swap is almost exhausted.
• Load is climbing and a build/install is not making progress.
• nginx config test fails.
• A database/stateful container is unhealthy after an unrelated change.
• A command proposes deleting volumes, data directories, system partitions, or broad config trees.

When explaining an outage, distinguish proof from inference: simultaneous healthcheck timeouts show starvation, but claim OOM only with kernel OOM evidence; Power key pressed proves a virtual ACPI request but not who initiated it; PostgreSQL recovery after an unclean stop is usually a consequence, not the cause.

Debug-only in-place patching of a compiled container (no image rebuild)

When you need to add a single defensive log line, a new error branch, or a one-off redaction to a service running in a pnpm deploy --prod (or similar flattened) production image, do not rebuild the image on a low-memory VPS. Instead:

1. Confirm the image is flattened: entrypoint is node dist/index.js; no node_modules/<package> outside the deploy root, no source src/.
2. Confirm the import chain is dist/index.js → dist/<module>.js so a single dist/*.js swap is enough. If the chain reaches into source (tsx watch), this technique does not apply.
3. Build the change on the host: tsc -p <package>/tsconfig.json produces updated dist/*.js. Verify with grep -c <new_symbol> dist/<module>.js.
4. Backup inside the container first: docker exec <c> sh -c 'cp /app/dist/<module>.js /app/dist/<module>.js.bak.$(date +%Y%m%d-%H%M%S)'. Repeat for any cross-package compiled files (e.g. shared types.js).
5. docker cp <host>/<module>.js <c>:/app/dist/<module>.js.
6. Restart the container (docker restart <c>) — not recreate, because recreate will re-pull the original pinned-digest image and undo the cp. Use recreate only after a tagged image bakes the change.
7. Verify with a probe that exercises the new code path; the new log line should appear in docker logs <c> --since 1m.

Constraints:

• This is debug-only. Document the divergence (<module>.js.bak.<ts> left in place) and track the follow-up to bake the change into a new image tag.
• If the change touches many files or you need to verify other adapters, prefer a real rebuild + new image tag (service-verify:temp) instead.
• If a CDN/edge layer caches responses (Cloudflare, nginx with proxy_cache), restart alone may not be enough — flush or wait for the cache TTL.

Common Pitfalls

1. Loading only a narrow deploy skill and skipping access/resource checks.
2. Restarting all Docker services for a one-service change.
3. Trusting swap size alone. Swap helps survival but does not eliminate CPU, disk, or OOM risk.
4. Filling root disk with Docker layers while disk2 is empty.
5. Reloading nginx before nginx -t.
6. Editing fstab without backup and verification.
7. Running docker volume prune as routine cleanup.
8. Breaking SSH/firewall rules while connected remotely.
9. Reporting success before checking port 22, Hermes-adjacent access, and affected endpoints.
10. Using docker compose up -d --force-recreate to deploy a debug-only logging change — it pulls the original image and undoes the in-place docker cp. Use docker restart after docker cp until the change is baked into a new tag.

Final Report Style

For Ikhsan, keep the final answer short and evidence-based:

Udah Bree, deploy/restart sudah staged. SSH 22 masih listen, swap 4G aktif sisa 2.8G, root disk sisa 6.4G, container target healthy, endpoint 200. Gak ada service lain yang ikut ke-restart.

If something is blocked:

Ketahan Bree: nginx -t gagal di baris X, jadi aku belum reload nginx. SSH 22 masih aman, service lama tetap jalan. Perlu benerin config ini dulu: ...

Verification Checklist

• This skill was loaded before narrower server/deploy skills.
• SSH port 22 checked before risky network/firewall/system change.
• Hermes/agent access was not intentionally disrupted.
• RAM, swap, load, root disk checked before heavy work.
• Config backups created before editing persistent system files.
• Config validation run before reload/restart.
• Changes applied in smallest safe unit.
• Heavy builds/installs run sequentially.
• Affected service verified with logs and endpoint/health check.
• Critical unrelated services still checked after change.
• No state/data volumes deleted without explicit approval.