Imported from SSujitX/docklift (.agent/skills/system_administration/SKILL.md). Install upstream with npx skills add SSujitX/docklift --skill system_administration. Copyright stays with the author.

System Administration Guide

Docklift includes built-in system management features accessible through the UI and API.

System Dashboard (/system)

The system page shows real-time server health metrics:

• CPU: Usage percentage, model, core count, temperature
• Memory: Used/total/percentage (reads host /proc/meminfo for accuracy)
• GPU: Model, VRAM, temperature, utilization (if available)
• Disk: Mount points, used/total/percentage
• Network: Bytes sent/received, speeds
• Processes: Top 10 by CPU (uses nsenter to read host processes)
• Server Info: Hostname, distro, kernel, uptime, public IP, location

API Endpoints

Maintenance Operations

Purge Resources

API: POST /api/system/purge

DockLift-scoped only. Requires password step-up (requireStepUpPassword / body password).

What it does (step-up required):

1. 409 if any deployment is in_progress before mutations (and again before image cleanup if a deploy raced in)
2. Remove unused docklift-* app tags outside each project’s keep-2 set (last 2 success image_tags + in-use container refs)
3. docker builder prune -af unless a deploy is in progress after image cleanup (skip -af rather than wipe live build cache)
4. Return real counts in details[]

What it must never do from the panel:

• docker system prune / delete non-docklift-* images
• Restart foreign containers
• Swap cycling, drop_caches, journal vacuum, apt clean, /tmp wipe

Related automatic hygiene: after successful deploy only — keep-2 project images + builder prune -f (unused). Failed deploy does not strip the last good release. Project → Deployments → Restore (step-up) pairs with keep-2.

UI copy: “unused Docklift images + clear BuildKit cache” — not “dangling only” / no-op.

Load averages on /api/system/stats come from real os.loadavg() — never fabricated load5/load15.

Server Control

Interactive Web Terminal

Route: /terminal WebSocket: ws://host:8000/ws/terminal (proxied via Nginx /ws/)

A full-featured xterm.js-based interactive terminal providing direct root access to the host.

Architecture

• Frontend: xterm.js + WebSocket
• Backend: ws server + child_process.spawn('script', ...)
• PTY: Uses Linux script command for TTY emulation (zero native dependencies)
• Container: Runs inside docklift-backend (Alpine) but has host access via Docker privileged mode & PID host.

Features

• Real-time PTY: Supports tab completion, history, colors, ncurses (htop/nano).
• Root Access: Session starts in /root with full host privileges.
• Host strip UI (TerminalView): quiet text actions above the shell — update packages, upgrade Docklift, purge unused Docklift images + BuildKit, reset stack, reboot. No rainbow action cards; shell chrome follows light/dark theme.
• Upgrade / package dialogs: Confirm first (panel goes offline; Docklift upgrade shows Now → Installing version + password on the same dialog; wrong password re-asks inline). After start, a wait dialog (Esc/overlay blocked; Dismiss only after countdown or in simulated dev) tells the operator to refresh in ~1–2 minutes. Sidebar “Upgrade now” routes to /terminal?confirm=upgrade (does not POST from the rail) and defers shell WS auth so operators are not prompted twice. Legacy ?action=upgrade opens the same confirm, not a fake in-progress state.
• Resizing: Bi-directional resize sync between frontend/backend. Resize inputs are validated (cols: 1–500, rows: 1–200) to prevent injection.
• Persistence: Auto-reconnect on network drops.
• Security:
  • Double Authentication: JWT (connect) + Password (interactive).
  • Rate Limiting: Max 5 logins/minute.
  • Session Limits: Max 3 concurrent connections per user.
  • Idle Timeout: Auto-disconnect after 30 minutes of inactivity.

Graceful Shutdown

The backend handles SIGTERM/SIGINT signals for clean exit:

• Stops accepting new HTTP connections.
• Cleans up all active terminal PTY sessions via cleanupAllSessions().
• Disconnects Prisma database client.
• Applied in: index.ts.

System Logs

API: GET /api/system/logs/:service (SSE stream)

Mapping lives in LOG_SERVICE_CONTAINERS (backend/src/routes/system.ts):

Version Check

API: GET /api/system/version

• Compares local package.json version against GitHub releases/latest
• Returns { current, latest, updateAvailable, githubOk, checkedAt }
• TTL: ~15m when update available, ~2m when current, ~30s when GitHub fails — never cache a failed probe as “no update” for an hour (that hid new releases)
• Concurrent callers share one GitHub fetch (single-flight). ?refresh=1 revalidates only if the cache is older than 30s (not an unconditional bypass).
• UI: sidebar footer only; one shared poller (desktop+mobile rails); no routine refresh=1

Backup & Restore System

All backup/restore routes are in backend/src/routes/backup.ts, mounted at /api/backup.

Paths

• BACKUP_PATH defaults to <DATA_PATH>/backups (not a hardcoded /data/backups).
• Production/dev Compose set BACKUP_PATH=/data/backups with a ./backups mount.
• Env names are DEPLOYMENTS_PATH, NGINX_CONF_PATH, DOCKER_NETWORK, DATA_PATH (not DOCKLIFT_* for those).

Backup

SQLite integrity: create uses VACUUM INTO a temp snapshot, then archives that file — never a live raw copy of docklift.db. Missing DB aborts with [ERROR] (UI must not toast success).

Frontend streams use frontend/src/lib/streamProgress.ts: require res.ok and fail on [ERROR] lines.

Restore

Safety:

• tryAcquireRestoreLock() before Multer (lib/restoreLock.ts) — concurrent restore → 409. Upload filenames are server-chosen (restore-<ts>-<random>.zip), never trusted client names.

• Password step-up for authenticated restores; setup-token path for fresh install (no users yet) via setupTokenAuth.

• All three restore routes track dbReplaced and call rollbackDatabaseFromPreRestore on later failure.

• Commit gate (finishRestoreCommit / decideRestoreCommit): consume setup secrets only when reconcile succeeds and restored DB has role: 'admin'. Setup-token + incomplete reconcile → abort + roll DB back. If rollback itself fails → critical seal:

  • Persist data/.restore-critical, hold restore lock, keep maintenance
  • Reload seal on backend boot (loadRestoreCriticalOnBoot)
  • Block all /api/backup/restore* until POST /api/backup/clear-critical-restore (password step-up)
  • Clear must verify the marker is gone before exiting maintenance — deletion failure stays sealed

• enterMaintenance() blocks all API traffic except /api/health and the active restore stream (not just mutating methods).

• Files/certs/proxy directory swaps are still not fully transactional with the DB.

• Directory restores use lib/fsCopy.ts (fs.promises.cp + content-level replace inside the target) — never shell cp, and never rename bind-mount roots (/deployments, /nginx-conf, /etc/letsencrypt). Staging/prev live under the mount as .docklift-restore-staging-* / .docklift-restore-prev-*. Failed promote or partial move-aside restores from prev; orphaned prev dirs are folded in on the next successful replace.

• Restore does not wipe other backup ZIPs.

Auto-Restore (reconcileSystem)

After restoring files, the system automatically:

1. Reads restored database — reconnectPrisma() then read projects
2. Recreates persistent volumes — Each PersistentVolume row is re-created as an external labelled Docker volume before the project starts, so mounts resolve
3. Brings every project back up — docker compose -f <runtime-compose> -p <composeProject> up -d, where the compose file is deployments/.docklift/<projectId>/compose.yml and the project name comes from composeProjectName(). Older backups without generated runtime state fall back to a docker-compose.yml at the source root; if neither exists the project is skipped and must be deployed manually
4. Reloads Nginx proxy — docker exec docklift-nginx-proxy nginx -s reload
5. Self-restarts backend — process.exit(0) triggers Docker's restart: unless-stopped policy

Any per-project or nginx failure emits [ERROR]; the stream ends with RESTORE INCOMPLETE so the UI does not toast success. Backend still restarts so the restored DB is loaded.

The -p name must match deploy-time naming, or restore creates a second set of containers and images alongside the originals.

What's Backed Up

Named volume contents are not archived — the volumes are re-created empty if missing. Snapshot application data separately if it matters.

Install / Upgrade / Uninstall Scripts

uninstall.sh targets DockLift-owned resources only (core names, dl_* / dl-*, dl-net-*, com.docklift.* labels). Never host-wide docker system prune / builder prune, and never remove Docker Engine/git.

Server Access Requirements

The backend container needs these host-level permissions (defined in docker-compose.yml):

• privileged: true — required for nsenter into host PID 1 (system update / reboot)
• pid: host — For host process visibility (reboot, system info)
• Docker socket mount: /var/run/docker.sock
• Host file mounts: /etc/hostname, /etc/os-release, /proc (read-only)
• ./nginx-proxy/certbot/conf:/etc/letsencrypt — read-write, so restores can put certificates back (the proxy mounts the same path read-only)

Panel purge must not use privileged host maintenance (drop_caches, journal, apt, swap, /tmp).