server-setup

Imported from timzarhansen/pi-config (skills/server-setup/SKILL.md). Install upstream with npx skills add timzarhansen/pi-config --skill server-setup. Copyright stays with the author.

Server Setup PI

Project Overview

A home lab server management system for deploying and managing self-hosted services on a remote VPS (161.13.12.139, user zarbokk). Uses Ansible as the automation engine, built on top of MASH-playbook.

Key Principles:

• Each service = separate Git repo + separate Ansible role
• Secrets never committed to Git
• Version pinning for stability
• Single root secret, all others derived
• MASH-playbook remains independent/unchanged (git submodule)

Project Location

PROJECT_ROOT: /workspaces/opencodeTestProject/projects_ws/server_setup_home/

All paths in this skill are relative to PROJECT_ROOT. Use this absolute path directly for all file operations. Do NOT search for the project — use this path.

Before You Start

1. Verify the project exists — use Read on /workspaces/opencodeTestProject/projects_ws/server_setup_home/ to confirm it's accessible.
2. If the directory does NOT exist at the absolute path above, use glob with pattern **/server_setup_home/deploy.sh as a fallback. Do NOT use glob with **/server_setup_home alone — Glob matches files, not directories.
3. If still not found → tell the user and ask for the correct path.

Do NOT try to read files from unverified paths. Do NOT use relative paths like projects_ws/server_setup_home — always use the absolute PROJECT_ROOT.

Before Configuring a MASH Galaxy Role Service

When configuring any MASH-managed service (a galaxy role), you MUST read its service documentation first.

Step 0: Read the service doc

Open mash-playbook/docs/services/<service-name>.md using the absolute path:

/workspaces/opencodeTestProject/projects_ws/server_setup_home/mash-playbook/docs/services/<service-name>.md

This doc contains critical information that the SKILL.md does not:

• Variable activation prefix (e.g. wg_easy_ for wg-easy)
• Dependencies on other MASH services (traefik, postgres, etc.)
• Critical warnings (e.g. "only takes effect on initial setup")
• Network/IPv6 requirements
• Required vs optional variables

Step 1: Read the role defaults

Open mash-playbook/roles/galaxy/<service-name>/defaults/main.yml for the full list of available variables with defaults.

Step 2: Set variables in vars.yml

Use the activation prefix from the service doc (or defaults/main.yml) when setting variables in inventory/host_vars/zarbokk.me/vars.yml.

Why this matters: Galaxy roles define their own variable prefixes, defaults, and MASH-specific behaviors. Skipping the service doc will cause misconfiguration, missing dependencies, and silent failures.

Architecture

server_setup_home/                          # Standalone git repo
├── deploy.sh              ← Master deploy script (single entry point)
├── matrix_deploy.sh       ← Matrix-specific deploy
├── ansible.cfg            ← Ansible config (role paths)
├── stable_versions.json   ← Version quarantine config (stable + pending)
├── version_history.json   ← Audit trail of promotions/rollbacks
├── roles/                 ← Custom Ansible roles (7 services)
│   ├── weather_mcp_server/  # Built from source
│   ├── note_mark/           # Built from source
│   ├── openwebui/           # Pre-built Docker image
│   ├── homepage/            # Pre-built Docker image
│   ├── seafile/             # Pre-built Docker image
│   ├── searxng/             # Pre-built Docker image
│   └── authentik_outpost/   # Pre-built Docker image
├── inventory/
│   ├── hosts              ← Server definitions (public)
│   │   ├── [mash_servers]           ← Main server: zarbokk.me
│   │   ├── [mash_servers_immich_deps] ← Dedicated deps for Immich
│   │   └── [matrix_servers]         ← Matrix servers (zarbokk.me-matrix)
│   └── host_vars/
│       ├── zarbokk.me/
│       │   └── vars.yml   ← SECRET (gitignored, MASH + custom services)
│       ├── zarbokk.me-immich-deps/
│       │   └── vars.yml   ← SECRET (gitignored, Immich's dedicated DB/Redis)
│       └── zarbokk.me-matrix/
│           └── vars.yml   ← SECRET (gitignored, Matrix services)
├── deploy_repos/          ← Local repo cache (gitignored, for source-code services)
├── mash-playbook/         ← MASH-playbook (git submodule — NEVER MODIFY)
│   ├── templates/setup.yml   ← Source template (never run directly)
│   ├── templates/requirements.yml ← Full galaxy role list (version-pinned at runtime)
│   ├── setup.yml             ← Generated at runtime (gitignored)
│   └── roles/                ← MASH roles (galaxy + mash/, galaxy installed here)
└── matrix_playbook/       ← Matrix playbook (git submodule)

Key Principle: MASH is Untouched

• mash-playbook/ is a git submodule — never modify files inside it
• Custom roles live in roles/ at the root level
• deploy.sh generates mash-playbook/setup.yml at runtime by copying the template and injecting custom role entries via sed
• All roles (MASH + custom) run in one single Ansible play — this is critical for variable scope sharing

Host Groups

• mash_servers — Main server (zarbokk.me, IP 161.13.12.139, user zarbokk). All MASH services + custom roles run here.
• mash_servers_immich_deps — Dedicated host for Immich's Valkey + Postgres (Immich requires separate database/redis instances, deployed on this host group)
• matrix_servers — Matrix server hosts (zarbokk.me-matrix). Pointed to zarbokk.me-matrix host for complete variable isolation from MASH. Has its own vars.yml with root secret, domain, and federation config (port 443). Matrix-side backup_borg and exim_relay are disabled (handled by MASH).

Server Path Conventions (CRITICAL — Do Not Mix These)

Files on the Pi follow two completely different path schemes depending on whether the service is MASH-managed or custom. Never apply the custom path convention to MASH services.

Common mistakes:

• cat /opt/traefik/config/provider.yml — WRONG, traefik is MASH-managed, it's at /mash/traefik/config/provider.yml
• cat /opt/authentik/.env — WRONG, authentik is MASH-managed, it's at /mash/authentik/.env
• cat /opt/weather-mcp-server/source/main.py — CORRECT, weather-mcp-server is a custom service

Rule of thumb: If the service is a MASH galaxy role (traefik, authentik, headscale, postgres, memos, etc.), its data lives under /mash/. If it's a service you created in roles/, it lives under /opt/.

When in doubt, check the role's defaults/main.yml on the Pi:

cat /mash/traefik/defaults/main.yml   # MASH role
cat /opt/weather-mcp-server/defaults/main.yml  # Custom role

Where to Run Commands

deploy.sh runs on your local machine (MacBook), NOT on the server. It uses Ansible over SSH to configure the remote Pi. The sed -i '' syntax in the script is macOS-only — if it fails, you're running it on the wrong machine.

# CORRECT — run locally on MacBook:
cd /workspaces/opencodeTestProject/projects_ws/server_setup_home/
./deploy.sh install-all

# WRONG — never SSH into the server to run deploy.sh:
ssh zarbokk@161.13.12.139 "cd ~/server_setup_home && ./deploy.sh install-all"

Commands that check logs or inspect running containers (journalctl, docker ps, systemctl status) should be run on the server via SSH.

Deploy Workflow (deploy.sh install-all)

The Flow

./deploy.sh install-all
  → sync_repos()        # Clone/fetch repos, create tarballs
  → prepare_playbook()  # Copy templates, inject custom roles
  → run_ansible()       # Run single-play Ansible

Note: install-all deploys files but does NOT start services. Run start-all after to start them.

sync_repos() — Clone + Tarball

Only services that need source code sync are handled here. Currently: weather-mcp and note-mark (openwebui now uses a pre-built Docker image, so it is NOT synced here).

For each synced service, this function:

1. Clones or fetches the repo into deploy_repos/SERVICE_NAME/
2. Checks out the desired branch/tag from stable_versions.json
3. Creates a tarball: deploy_repos/SERVICE_NAME.tar.gz (.git excluded)

Why tarballs? ansible.builtin.synchronize (rsync) is unreliable for this use case — it may report "no changes" while not actually transferring files. Tarballs are deterministic and work every time.

Services that do NOT need sync: openwebui, searxng, homepage, seafile, authentik_outpost — these use pre-built Docker images and have no repo sync block in deploy.sh.

prepare_playbook() — Inject Custom Roles

1. Copies mash-playbook/templates/setup.yml → mash-playbook/setup.yml
2. Copies mash-playbook/templates/group_vars_mash_servers → inventory/group_vars/mash_servers
3. Comments out galaxy roles we don't want: cinny, etherpad, prunemate, tsdproxy (both in setup.yml and group_vars/mash_servers)
4. Calls inject_custom_roles() which uses sed to insert role entries before the # role-specific:auxiliary marker in setup.yml

Why injection? MASH's setup.yml is a single play with a roles: block. By injecting our roles into that block, all roles share the same variable scope. This is what makes mash_playbook_service_identifier_prefix, traefik_identifier, and other MASH variables available to custom roles.

Commented-out galaxy roles: cinny, etherpad, prunemate, tsdproxy — these are skipped because they're not needed (cinny/etherpad are Matrix clients, running via matrix_playbook; prunemate and tsdproxy are commented out due to missing variables).

run_ansible() — Run the Playbook

Runs ansible-playbook mash-playbook/setup.yml --tags=... with extra vars (-e) that pass absolute paths to the local repo directories.

Why extra vars? playbook_dir resolves to mash-playbook/ (where setup.yml lives). Custom role defaults can't reliably compute the path to deploy_repos/ at the root. Extra vars override defaults with absolute paths.

Current extra vars passed:

• weather_mcp_server_local_repo_path (for weather-mcp)
• note_mark_local_repo_path (for note-mark)

Add more with -e "<service>_local_repo_path=$SCRIPT_DIR/deploy_repos/<service>" when adding new services that need repo sync.

deploy.sh Functions (what to modify when adding a service)

Version Quarantine System

stable_versions.json tracks approved versions for all components (service repos, MASH, Galaxy roles). New versions sit in pending for 14 days before deployment.

./deploy.sh seed                 # Auto-populate stable_versions.json (non-destructive merge)
./deploy.sh seed --used          # Seed only roles referenced in vars.yml (trims unused)
./deploy.sh seed --force         # Overwrite existing stable versions with detected values
./deploy.sh list-stable          # Show all current stable versions
./deploy.sh list-pending         # Show pending versions with countdown
./deploy.sh promote <comp> <ver> # Promote pending → stable (2-week gate)
./deploy.sh promote --force      # Force promote (skip 2-week gate, emergencies)
./deploy.sh rollback <comp> <ver> # Revert to a previous version
./deploy.sh history [comp]       # Show promotion/rollback log

Seed behavior (non-destructive):

• Existing stable values are preserved (never auto-overwritten)
• Existing pending values are always preserved
• Use --force to overwrite stable values with detected versions
• Use --used to detect only roles referenced in vars.yml (infra roles like traefik, docker, systemd_docker_base are kept via prefix presence)

De-bloat: rm stable_versions.json && ./deploy.sh seed --used trims the file from ~260 roles to only ~30 active ones.

Workflow: seed → new versions appear in pending → after 14 days promote → install-all to deploy → rollback if something breaks.

Role Pattern (Reference: weather_mcp_server)

File Structure

roles/SERVICE_NAME/
├── defaults/main.yml           # All variables (inline Jinja2, NOT multiline YAML blocks)
├── handlers/main.yml           # restart handler (DEPRECATED — start task in main.yml consumes restart_needed fact directly)
├── meta/main.yml               # Metadata (copy from weather_mcp_server, update name)
├── tasks/
│   ├── main.yml                # validate → install → start → uninstall → stop blocks
│   ├── install.yml             # The critical file: dirs → tarball → build → deploy
│   ├── uninstall.yml           # Stop → remove service → network → image → path
│   └── validate_config.yml     # Assert required vars, path_prefix, Traefik hostname
└── templates/
    ├── env.j2                  # KEY=value env file for --env-file
    ├── labels.j2               # Traefik labels for --label-file
    └── systemd/
        └── service-name.service.j2  # systemd unit (docker create + start --attach)

defaults/main.yml — Critical Variables

SERVICE_NAME_identifier: "{{ mash_playbook_service_identifier_prefix }}service-name"
SERVICE_NAME_enabled: false
SERVICE_NAME_local_repo_path: "{{ playbook_dir | dirname }}/deploy_repos/service-name"  # overridden by -e
SERVICE_NAME_hostname: ''
SERVICE_NAME_path_prefix: /service-name
SERVICE_NAME_container_port: 8000
SERVICE_NAME_base_path: "/opt/service-name"
SERVICE_NAME_source_path: "{{ SERVICE_NAME_base_path }}/source"
SERVICE_NAME_docker_image: "{{ SERVICE_NAME_identifier }}"
SERVICE_NAME_docker_tag: "latest"
SERVICE_NAME_container_network: "{{ SERVICE_NAME_identifier }}"
SERVICE_NAME_container_network_deletion_enabled: true

# ═══ Systemd Dependencies ═══════════════════════════════════════════
# Hard dependencies (Requires): if these stop, this service stops too.
SERVICE_NAME_systemd_required_services_list: "{{ SERVICE_NAME_systemd_required_services_list_auto + SERVICE_NAME_systemd_required_services_list_custom }}"
SERVICE_NAME_systemd_required_services_list_auto: []
SERVICE_NAME_systemd_required_services_list_custom: []

# Soft dependencies (Wants): startup ordering only, no cascade stop.
SERVICE_NAME_systemd_wanted_services_list: "{{ SERVICE_NAME_systemd_wanted_services_list_auto + SERVICE_NAME_systemd_wanted_services_list_custom }}"
SERVICE_NAME_systemd_wanted_services_list_auto: "{{ ([container_socket_proxy_identifier ~ '.service'] if container_socket_proxy_enabled | default(false) else []) + ([traefik_identifier ~ '.service'] if traefik_enabled | default(false) and SERVICE_NAME_hostname else []) }}"
SERVICE_NAME_systemd_wanted_services_list_custom: []
# ... (full template in llm_instructions/README.md section 7)

tasks/install.yml — Order Matters

1. Create directories (base_path, source_path)
2. Copy tarball (ansible.builtin.copy from local_repo_path.tar.gz to /tmp/)
3. Extract tarball (ansible.builtin.unarchive with remote_src: yes)
4. Remove temp archive
5. Verify Dockerfile exists
6. Check if Docker image exists (ansible.builtin.command: docker image inspect)
7. Build Docker image (ansible.builtin.command: docker build) — only if image missing or files changed
8. Create container network (ansible.builtin.command: docker network create)
9. Deploy env file, labels file, systemd service (via ansible.builtin.template)
10. Reload systemd, determine restart needed (sets *_restart_needed fact)

Note: install.yml does NOT start the service. The start block in main.yml handles that (tagged with start / start-all).

templates/labels.j2 — Middleware Chain

Custom middleware is chained via loop (added in hardening commit):

{% for custom_mw in SERVICE_NAME_container_labels_traefik_middleware_custom_names %}
{% set _ = middlewares.append(custom_mw) %}
{% endfor %}

Set SERVICE_NAME_container_labels_traefik_middleware_custom_names: ["global-ratelimit@file", "global-security-headers@file"] in vars.yml.

templates/systemd/*.service.j2 — Docker Pattern

Uses docker create + docker start --attach pattern (NOT docker run):

• ExecStartPre: docker rm -f old container
• ExecStartPre: docker create --rm --name=... --env-file=... --label-file=... --network=...
• ExecStart: docker start --attach
• ExecStartPost: docker network connect for additional networks
• ExecStop: docker stop then docker rm

Systemd dependencies use both Requires and Wants:

• Requires block — hard deps (if these stop, this service stops too). Currently empty for all custom roles.
• Wants block — soft deps (startup ordering only, no cascade stop). Contains container_socket_proxy and traefik so restarting them doesn't cascade-stop custom services.

{% for service in SERVICE_NAME_systemd_required_services_list %}
Requires={{ service }}
After={{ service }}
{% endfor %}
{% for service in SERVICE_NAME_systemd_wanted_services_list %}
Wants={{ service }}
After={{ service }}
{% endfor %}

Adding a New Service — Step by Step

Step 1: Create Role Directory

# Run from PROJECT_ROOT: /workspaces/opencodeTestProject/projects_ws/server_setup_home/
mkdir -p roles/my_service/{defaults,handlers,meta,tasks,templates/systemd}

Step 2: Create Files

Create these 8+ files following the weather_mcp_server pattern. Full templates are in llm_instructions/README.md section 7. Replace SERVICE_NAME (snake_case) and service-name (kebab-case) throughout.

Step 3: Modify deploy.sh (4 places)

A. sync_repos() — add clone + tarball (source-code services only):

# my-service
local repo_path="$SCRIPT_DIR/deploy_repos/my-service"
local my_service_stable
my_service_stable=$(get_stable_version "my-service")
if [ ! -d "$repo_path/.git" ]; then
    log_info "Cloning my-service..."
    git clone "git@github.com:user/my-service.git" "$repo_path"
else
    log_info "Fetching my-service updates..."
    git -C "$repo_path" fetch --all --tags
fi
log_info "my-service stable version: $my_service_stable"
git -C "$repo_path" checkout "$my_service_stable"
git -C "$repo_path" reset --hard "origin/$my_service_stable" 2>/dev/null || git -C "$repo_path" reset --hard "$my_service_stable"
tar -czf "$repo_path.tar.gz" -C "$repo_path" --exclude='.git' .

B. inject_custom_roles() — add sed injection block:

    # role-specific:my_service\
    - role: roles/my_service\
      tags:\
        - my-service\
        - install-all\
        - setup-all\
    # /role-specific:my_service\

Current services injected (in order): openwebui, authentik_outpost, weather_mcp_server, homepage, seafile, searxng, note_mark. Insert before # role-specific:auxiliary.

After injecting custom roles, inject_custom_roles() also comments out unused galaxy roles (cinny, etherpad, prunemate, tsdproxy) so they don't run.

C. run_ansible() — add extra var (source-code services only):

# Pre-built image services (homepage, openwebui, authentik-outpost, seafile, searxng) need no extra vars
-e "weather_mcp_server_local_repo_path=$SCRIPT_DIR/deploy_repos/weather-mcp" \

D. case statement — add to nested case blocks:

The CUSTOM_SERVICES and NEEDS_SYNC variables at the top of deploy.sh control which services use ansible vs systemctl for start/stop, and which need repo sync:

CUSTOM_SERVICES="weather-mcp-server note-mark homepage openwebui authentik-outpost seafile searxng"
NEEDS_SYNC="weather-mcp-server note-mark"

install)
    ...
    case "$SERVICE" in
        weather-mcp-server|note-mark|my-service)  # source-code: needs sync
            sync_repos
            prepare_playbook
            run_ansible "install-all,$SERVICE" "$@"
            ;;
        homepage|openwebui|authentik-outpost|seafile|searxng)  # pre-built: no sync
            prepare_playbook
            run_ansible "install-all,$SERVICE" "$@"
            ;;
        *)
            prepare_playbook
            run_ansible "install-all,$SERVICE" "$@"
            ;;
    esac
    ;;
start)
    ...
    case "$SERVICE" in
        weather-mcp-server|note-mark|homepage|openwebui|authentik-outpost|seafile|searxng|my-service)
            prepare_playbook
            run_ansible "start,$SERVICE" "$@"
            ;;
        *)
            # MASH role fallback — systemctl over SSH
            UNIT="$(get_mash_service_unit "$SERVICE")"
            ansible -i "$INVENTORY" mash_servers -a "systemctl start ${UNIT}.service" -b "$@"
            ;;
    esac
    ;;
stop)
    ...
    case "$SERVICE" in
        weather-mcp-server|note-mark|homepage|openwebui|authentik-outpost|seafile|searxng|my-service)
            prepare_playbook
            run_ansible "stop,$SERVICE" "$@"
            ;;
        *)
            # MASH role fallback — systemctl over SSH
            UNIT="$(get_mash_service_unit "$SERVICE")"
            ansible -i "$INVENTORY" mash_servers -a "systemctl stop ${UNIT}.service" -b "$@"
            ;;
    esac
    ;;
restart)
    ...
    case "$SERVICE" in
        weather-mcp-server|note-mark|homepage|openwebui|authentik-outpost|seafile|searxng|my-service)
            prepare_playbook
            log_info "Stopping $SERVICE..."
            run_ansible "stop,$SERVICE" "$@"
            log_info "Starting $SERVICE..."
            run_ansible "start,$SERVICE" "$@"
            ;;
        *)
            UNIT="$(get_mash_service_unit "$SERVICE")"
            ansible -i "$INVENTORY" mash_servers -a "systemctl restart ${UNIT}.service" -b "$@"
            ;;
    esac
    ;;

Step 4: Configure in vars.yml

Add to inventory/host_vars/zarbokk.me/vars.yml:

my_service_enabled: true
my_service_hostname: "myservice.zarbokk.me"
# ... other variables

Step 5: Deploy

# Run from PROJECT_ROOT: /workspaces/opencodeTestProject/projects_ws/server_setup_home/
./deploy.sh install my-service    # Test just the new service
./deploy.sh start my-service      # Start it (custom roles via ansible, MASH via systemctl)
./deploy.sh install-all           # Full install
./deploy.sh start-all             # Start all services

Critical Gotchas (Pitfalls — read before making changes)

G1: when: Conditions — Use | length > 0 for Strings

# WRONG — Ansible 2.20+ rejects string-as-boolean:
when: some_string_var | bool

# CORRECT:
when: some_string_var | length > 0

G2: Docker Commands — Use ansible.builtin.command, NOT community.docker.*

# WRONG — runs on control machine (MacBook), not remote host:
community.docker.docker_network:
  name: "my-network"
  state: present

# CORRECT — runs on the Pi:
ansible.builtin.command:
  cmd: "docker network create my-network"

G3: Never Use synchronize for Repo Transfer

Use tarball + copy + unarchive. synchronize (rsync) reports "no changes" without transferring.

G4: Role Defaults — Inline Jinja2, NOT Multiline YAML Blocks

# WRONG — produces a YAML string:
my_var: |
  {{ ([something] if condition else []) }}

# CORRECT — produces native Python list:
my_var: "{{ [something] if condition else [] }}"

G5: playbook_dir Resolves to mash-playbook/

# WRONG — resolves to mash-playbook/deploy_repos/ (doesn't exist):
my_service_local_repo_path: "{{ playbook_dir }}/deploy_repos/my-service"

# CORRECT — override via -e extra var in deploy.sh:
# -e "my_service_local_repo_path=$SCRIPT_DIR/deploy_repos/my-service"

G6: Docker DNS on the Pi

If docker build fails with Temporary failure in name resolution, fix DNS:

echo '{"dns": ["8.8.8.8", "8.8.4.4"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker

G7: git clone Has No mode Parameter

# WRONG:
ansible.builtin.git:
  repo: "..."
  mode: "0755"

# CORRECT — set permissions separately:
ansible.builtin.git:
  repo: "..."
  dest: "..."

G8: Stop Task Must Have when: Guard

The stop task in tasks/main.yml must have when: SERVICE_NAME_enabled | bool. Without it, Ansible tries to stop a non-existent service when the service is disabled, causing a fatal error:

- name: Stop service-name service
  ansible.builtin.systemd:
    name: "{{ SERVICE_NAME_identifier }}"
    state: stopped
  when: SERVICE_NAME_enabled | bool
  ignore_errors: true
  tags:
    - service-name
    - stop-service
    - stop
    - stop-all

All other tasks (validate, install, start) have when: guards. The stop task is the only one that commonly forgets this.

Tags: Every stop task must include stop, stop-all, and stop-service tags so ./deploy.sh stop-all picks it up. The start task must include start and start-all tags.

G9: sed Injection — 4-Space Indentation, No when: Guard

The sed in inject_custom_roles() must use 4-space indentation to match MASH setup.yml. On Linux use sed -i, on macOS sed -i ''.

Critical: Do NOT add a when: condition to the role injection block. Enable/disable is handled inside each task file (tasks/main.yml) via when: SERVICE_NAME_enabled | bool. Adding when: to the role entry itself would prevent the uninstall task from running when the service is disabled — the role must always execute so its uninstall block can clean up.

G10: MASH Galaxy Roles Use /mash/ — NOT /opt/

MASH-managed services (galaxy roles like traefik, authentik, postgres, memos, headscale, etc.) store their data under /mash/<service-name>/ on the Pi, not /opt/<service-name>/. The /opt/ convention only applies to custom services you created in roles/.

# WRONG — LLMs will default to /opt/ for everything:
cat /opt/traefik/config/provider.yml

# CORRECT — MASH services live under /mash/:
cat /mash/traefik/config/provider.yml

# CORRECT — custom services under /opt/:
cat /opt/weather-mcp-server/source/main.py

How to tell which is which:

• If the service is a MASH galaxy role (in mash-playbook/roles/galaxy/), it uses /mash/
• If the service has a role in your roles/ directory, it uses /opt/
• When configuring a MASH service, always check mash-playbook/roles/galaxy/<service>/defaults/main.yml for the actual base_path value — it may not even be /mash/<service>/ (some roles override it)

G11: Ghost Router Trap — Middleware Labels Target Wrong Router Names

MASH galaxy roles generate Traefik routers using {{ SERVICE_NAME_identifier }} (e.g., mash-headplane), but some roles define default additional_labels that reference a different name (e.g., headplane, wg_easy, adguard_home). If you add middleware via *_container_labels_additional_labels without checking the actual router name, it creates an unused "ghost router" instead of wiring to the real one.

How it happens: A label like traefik.http.routers.headplane.middlewares=... creates a new router named headplane that never receives traffic. Real traffic goes to mash-headplane, which has no middleware.

Fix: Always target the actual router name (mash-<kebab-case>) in vars.yml. Check docker inspect mash-<service> --format '{{json .Config.Labels}}' to find the real router name.

# WRONG — creates ghost router "headplane" that never handles traffic:
headplane_container_labels_additional_labels: |
  traefik.http.routers.headplane.middlewares=authentik-forward-auth@file

# CORRECT — targets the real router that MASH actually creates:
headplane_container_labels_additional_labels: |
  traefik.http.routers.mash-headplane.middlewares=authentik-forward-auth@file

Common router name mismatches (MASH galaxy roles):

G12: Authentik Outpost Returns HTTP 500 Without Forwarded Headers

Authentik's Traefik outpost (/auth/traefik endpoint) needs X-Forwarded-Host, X-Forwarded-Proto, X-Forwarded-For, and X-Real-IP headers to construct redirect URLs. Traefik doesn't forward these by default to forwardAuth targets. If the outpost returns 500 with "failed to detect a forward URL from Traefik", add a separate headers middleware and chain it before the forwardAuth middleware.

Symptoms: Outpost logs show "failed to detect a forward URL from Traefik", "status": 500. Direct test wget http://<outpost>:9000/outpost.goauthentik.io/auth/traefik returns 500.

Fix — two middlewares chained together:

traefik_configuration_extension_yaml: |
  http:
    middlewares:
      authentik-forward-headers:
        headers:
          requestHeaders:
            - X-Forwarded-Host
            - X-Forwarded-Proto
            - X-Forwarded-For
            - X-Real-IP
      authentik-forward-auth:
        forwardAuth:
          address: "http://mash-authentik-outpost:9000/outpost.goauthentik.io/auth/traefik"
          trustForwardHeader: true
          authResponseHeaders:
            - X-authentik-username
            - X-authentik-groups
            - X-authentik-email
            - X-authentik-name
            - X-authentik-uid
            - X-authentik-jwt
            - X-authentik-meta-jwks
            - X-authentik-meta-outpost
            - X-authentik-meta-provider
            - X-authentik-meta-app
            - X-authentik-meta-version

Then chain them on each service (headers first, then auth):

headplane_container_labels_additional_labels: |
  traefik.http.routers.mash-headplane.middlewares=authentik-forward-headers,authentik-forward-auth@file

Why two separate middlewares? The headers middleware is reusable across all protected services. The forwardAuth middleware is the Authentik-specific part. Keeping them separate makes it easy to add/remove auth from individual services without touching the global config.

G13: Authentik Outpost Image Must Match Server Version

Use the exact version tag matching your Authentik server version, e.g., ghcr.io/goauthentik/proxy:2026.2.3. Do NOT use ghcr.io/goauthentik/outpost:latest or untagged latest. Mismatched versions cause silent failures and incompatibility errors.

Correct deployment:

docker run -d \
  --name mash-authentik-outpost \
  --network traefik \
  -e AUTHENTIK_HOST="https://authentik.zarbokk.me" \
  -e AUTHENTIK_INSECURE="true" \
  -e AUTHENTIK_TOKEN="<your-token>" \
  -e AUTHENTIK_OUTPOSTS__ENABLED=true \
  ghcr.io/goauthentik/proxy:2026.2.3

Wrong images: ghcr.io/goauthentik/outpost:latest, goauthentik.io/authentik/outpost-proxy:latest, ghcr.io/goauthentik/proxy:latest

G14: Systemd Dependencies — Use Wants, NOT Requires

Never put container_socket_proxy or traefik in systemd_required_services_list. Use systemd_wanted_services_list instead.

Why: Requires creates a hard dependency — if container-socket-proxy or traefik restart during a playbook run, systemd cascade-stops the dependent service. Wants is a soft dependency: it provides startup ordering only, with no cascade stop.

# WRONG — cascade stop when traefik restarts:
SERVICE_NAME_systemd_required_services_list_auto: >-
  {{ ([container_socket_proxy_identifier ~ '.service'] ...) + ([traefik_identifier ~ '.service'] ...) }}

# CORRECT — startup ordering only, no cascade stop:
SERVICE_NAME_systemd_required_services_list_auto: []
SERVICE_NAME_systemd_wanted_services_list_auto: >-
  {{ ([container_socket_proxy_identifier ~ '.service'] if container_socket_proxy_enabled | default(false) else []) + ([traefik_identifier ~ '.service'] if traefik_enabled | default(false) and SERVICE_NAME_hostname else []) }}

Every custom role must follow this pattern. The systemd template must include both a Requires loop and a Wants loop:

{% for service in SERVICE_NAME_systemd_required_services_list %}
Requires={{ service }}
After={{ service }}
{% endfor %}
{% for service in SERVICE_NAME_systemd_wanted_services_list %}
Wants={{ service }}
After={{ service }}
{% endfor %}

This was fixed in commit 86e2429 across all 7 custom roles. If you add a new role, follow this pattern from the start.

Modifying an Existing Service

Change Traefik Labels / Middleware

1. Edit roles/SERVICE_NAME/templates/labels.j2
2. If adding new middleware, ensure SERVICE_NAME_container_labels_traefik_middleware_custom_names is defined in defaults/main.yml
3. Set the middleware names in inventory/host_vars/zarbokk.me/vars.yml
4. Deploy: ./deploy.sh install <service-name> then ./deploy.sh start <service-name>

Change Environment Variables

1. Edit roles/SERVICE_NAME/templates/env.j2
2. If new variable needs to come from vars.yml, add corresponding default in defaults/main.yml
3. Set value in inventory/host_vars/zarbokk.me/vars.yml
4. Deploy: ./deploy.sh install <service-name> then ./deploy.sh start <service-name>

Change Systemd Service

1. Edit roles/SERVICE_NAME/templates/systemd/service-name.service.j2
2. Deploy: ./deploy.sh install <service-name> then ./deploy.sh start <service-name>

Add Global Traefik Middleware (e.g., rate limiting, security headers)

Define in inventory/host_vars/zarbokk.me/vars.yml:

traefik_configuration_extension_yaml: |
  http:
    middlewares:
      global-ratelimit:
        rateLimit:
          average: 100
          burst: 50
      global-security-headers:
        headers:
          # ... security headers config

Then wire to service:

weather_mcp_server_container_labels_traefik_middleware_custom_names:
  - "global-ratelimit@file"
  - "global-security-headers@file"

Common Commands

All commands run from PROJECT_ROOT (/workspaces/opencodeTestProject/projects_ws/server_setup_home/).

Two-step flow: install-all deploys files but does NOT start services. Run start-all (or restart-all) after to start them.

Matrix Deploy (matrix_deploy.sh)

Matrix uses a separate playbook (matrix_playbook/ submodule) with its own deploy script. Matrix config is isolated in inventory/host_vars/zarbokk.me-matrix/vars.yml to prevent variable conflicts with MASH.

./matrix_deploy.sh install-all   # Full Matrix install (installs galaxy roles + runs install)
./matrix_deploy.sh start-all     # Start all Matrix services
./matrix_deploy.sh stop-all      # Stop all Matrix services
./matrix_deploy.sh setup-all     # ⚠️ ALIASED to install-all — prevents Traefik uninstall via 'other-traefik-container'
./matrix_deploy.sh install-roles # Install/update galaxy roles only
./matrix_deploy.sh update        # Pull latest matrix_playbook upstream, install roles
./matrix_deploy.sh status        # Check Matrix service status (docker ps)

Warning: setup-all is aliased to install-all because running the full setup playbook with other-traefik-container would trigger Traefik uninstall. Only use install-all for Matrix. MASH handles Traefik for the whole server.

Matrix services are NOT backed up by MASH's backup_borg — they manage their own backup scope. The /matrix directory is added to MASH Borg sources for protection.

Extra Args

All commands accept Ansible arguments:

./deploy.sh install-all --limit zarbokk.me
./deploy.sh install-all --tags=weather-mcp-server --diff

Service Status on Server

systemctl status mash-service-name.service
docker ps | grep service-name
journalctl -u mash-service-name.service -f
docker logs mash-service-name

Variable Naming Convention

Galaxy Roles (MASH Services)

Location

All MASH-managed services live in mash-playbook/roles/galaxy/. Each subdirectory is a separate Ansible role from an external Git repo.

Path: $PROJECT_ROOT/mash-playbook/roles/galaxy/

250+ services are available. The complete list is defined in mash-playbook/templates/requirements.yml, which maps:

• Role name → Git source URL → Version → Activation prefix (variable prefix in vars.yml)

Example from requirements.yml:

- src: git+https://github.com/mother-of-all-self-hosting/ansible-role-authentik.git
  version: v2026.2.3-0
  name: authentik
  activation_prefix: authentik_

This means all authentik variables are prefixed with authentik_ in vars.yml.

Role Structure

Each galaxy role follows the standard Ansible role layout:

mash-playbook/roles/galaxy/SERVICE_NAME/
├── defaults/main.yml        # ALL variables with defaults ← READ THIS
├── tasks/
│   └── main.yml             # Install/uninstall/start/stop logic
├── templates/
│   ├── env.j2               # Environment file template
│   ├── labels.j2            # Traefik labels template
│   └── systemd/             # Systemd unit templates (if applicable)
├── meta/main.yml            # Role metadata
└── README.md                # Role-specific docs

Service Documentation

Each service has a documentation file at: Path: $PROJECT_ROOT/mash-playbook/docs/services/SERVICE_NAME.md

Example: mash-playbook/docs/services/authentik.md, mash-playbook/docs/services/headplane.md

These docs contain:

• Required configuration variables
• Optional variables with descriptions
• Example configurations
• Usage instructions

How to Find Variables for Any Service

When you need to configure a service (e.g., grafana), follow this procedure:

1. Check the service docs — read mash-playbook/docs/services/grafana.md for required vars
2. Read the role defaults — open mash-playbook/roles/galaxy/grafana/defaults/main.yml for ALL available variables
3. Set variables in vars.yml — prefix with the activation prefix (e.g., grafana_enabled: true)

Common Variable Patterns (Across ALL Galaxy Roles)

Every galaxy role shares these universal variable patterns:

Enable/Disable

SERVICE_NAME_enabled: false          # Enable or disable the service

Network & Identity

SERVICE_NAME_container_network: "{{ SERVICE_NAME_identifier }}"   # Docker network name
SERVICE_NAME_identifier: service_name                              # Internal identifier

Traefik Labels (most services)

SERVICE_NAME_container_labels_traefik_enabled: true
SERVICE_NAME_container_labels_traefik_docker_network: "{{ SERVICE_NAME_container_network }}"
SERVICE_NAME_container_labels_traefik_hostname: ''
SERVICE_NAME_container_labels_traefik_path_prefix: /
SERVICE_NAME_container_labels_traefik_rule: "Host(`{{ SERVICE_NAME_container_labels_traefik_hostname }}`)"
SERVICE_NAME_container_labels_traefik_entrypoints: web-secure
SERVICE_NAME_container_labels_traefik_tls: true
SERVICE_NAME_container_labels_traefik_tls_certResolver: default

Middleware Support (most services)

# List of middleware names to append to the router
SERVICE_NAME_container_labels_traefik_middleware_custom_names:
  - "global-ratelimit@file"
  - "global-security-headers@file"

Additional Labels (inject raw Traefik labels)

# Most roles support string or list format:
SERVICE_NAME_container_labels_additional_labels: ""          # String (e.g., headplane)
SERVICE_NAME_container_labels_additional_labels_custom: []   # List (e.g., authentik, traefik)

Additional Networks

# Connect container to extra Docker networks
SERVICE_NAME_container_additional_networks_custom: []

Additional Volumes

SERVICE_NAME_container_additional_volumes_custom: []

Additional Environment Variables

SERVICE_NAME_environment_variables_additional_variables: ""

Base Path & Data

SERVICE_NAME_base_path: "/{{ SERVICE_NAME_identifier }}"
SERVICE_NAME_data_path: "{{ SERVICE_NAME_base_path }}/data"

Docker Image

SERVICE_NAME_container_image: "{{ SERVICE_NAME_container_image_registry_prefix }}image:{{ SERVICE_NAME_container_image_tag }}"
SERVICE_NAME_container_image_tag: "version"

Systemd Dependencies

SERVICE_NAME_systemd_required_services_list_custom: []
SERVICE_NAME_systemd_wanted_services_list_custom: []

Network Gotcha — The #1 Problem

When a role's ForwardAuth or any inter-container communication requires another container's network:

• WRONG: Add the target network to the caller's additional_networks — the network may not exist yet (deployment order issue)
• CORRECT: Add the caller's network to the target's additional_networks — the caller's network is always created first

Example (authentik → traefik ForwardAuth):

# In vars.yml, authentik section:
authentik_container_additional_networks_custom:
  - traefik
# Then override the docker network label:
authentik_container_labels_traefik_docker_network: traefik

Authentik Outpost network: The outpost is a standalone Docker container (not a MASH role). It must be manually attached to the traefik network at creation time:

docker run -d --name mash-authentik-outpost --network traefik \
  -e AUTHENTIK_HOST="https://..." -e AUTHENTIK_TOKEN="..." \
  ghcr.io/goauthentik/proxy:2026.2.3

Quick Reference — Currently Used Galaxy Roles

Authentik Outpost (Standalone Docker Container)

Security Model

• Single root secret (mash_playbook_generic_secret_key) — generate with pwgen -s 64 1
• Derived secrets — deterministic pattern:

  {{ 'username:' + (mash_playbook_generic_secret_key + ':service.suffix') | hash('sha512') | password_hash('bcrypt') }}

• HTTPS/TLS — via Traefik reverse proxy
• Basic Auth — via Traefik middleware for sensitive services
• API Key auth — for MCP services (instead of basic auth)
• Secrets never committed — inventory/host_vars/*/vars.yml is gitignored

Service Inventory

Custom Roles (7)

Custom roles live in roles/ at the root level. They are injected into the MASH playbook at runtime by deploy.sh. Each custom role manages a single service.

Note: galaxy is NOT a custom role — it contains Ansible Galaxy roles (~20 services) installed by install_galaxy_roles() in deploy.sh.

MASH Native Services

These are built-in MASH-playbook services (not custom roles). They are configured in vars.yml and deployed via MASH's own roles. They do NOT have entries in roles/.

Currently configured in vars.yml:

Key distinction: Custom roles (in roles/) = your code, injected at runtime. MASH native services = MASH's built-in roles, configured in vars.yml.

Galaxy Roles (~20)

wikimore, woodpecker_ci_server, woodpecker_ci_agent, wordpress, writefreely, yacy, yggstack, yourls, and more.

Key Galaxy Roles (use these before proposing new ones)

Data Paths Reference

All service data lives in two locations depending on service type:

Critical: When planning backups, monitor, or migrations, always check which pattern applies:

• Custom roles → /opt/
• MASH native (configured in vars.yml) → /mash/

Backup

MASH provides built-in galaxy/backup_borg role for BorgBackup + borgmatic. Already in setup.yml (injected by deploy.sh).

Enabling backup_borg

Add to inventory/host_vars/zarbokk.me/vars.yml:

backup_borg_enabled: true
backup_borg_schedule: "*-*-* 03:02:00"
backup_borg_location_source_directories_custom:
  - /mash
  - /opt
  - /etc
backup_borg_location_exclude_patterns_custom:
  - /opt/*/source
  - /proc
  - /sys
  - /dev
  - /tmp
  - /var/lib/docker/overlay2
  - /swap*
backup_borg_location_repositories:
  - /mnt/nas-backup/borg
backup_borg_storage_encryption_passphrase: "<independent passphrase>"
backup_borg_postgresql_enabled: true
backup_borg_container_extra_arguments_custom:
  - "--mount"
  - "type=bind,src=/mnt/nas-backup/borg,dst=/mnt/nas-backup/borg,rw"

Key backup considerations

• Use /mash not /opt — MASH native services store data at /mash/<service>/. /opt/ only covers custom roles.
• Enable Postgres — backup_borg_postgresql_enabled: true auto-dumps all managed databases.
• Separate passphrase — Never derive borg passphrase from mash_playbook_generic_secret_key. Rotate independently.
• NFS mount — Use nofail in fstab so server boots if NAS unavailable.
• Daily schedule — backup_borg_schedule: "*-*-* 03:02:00" matches retention (7 daily + 4 weekly + 12 monthly).
• Exclude build sources — /opt/*/source is rebuildable from git/tarballs.
• Borg Web UI — Optional galaxy/borg_ui role for browser-based restore.
• Verify backups — Periodic restore tests. A backup you can't restore is useless.

Deploy

./deploy.sh install-all --tags=backup-borg
./deploy.sh start-all --tags=backup-borg

Verify

systemctl list-timers mash-backup-borg.timer
journalctl -u mash-backup-borg.service -f

Restore

# Full restore
borgmatic restore /mnt/nas-backup/borg --target /tmp/restore/

# Single path
borgmatic restore /mnt/nas-backup/borg --target /tmp/restore/ --source /mash/authentik

# Single file
borgmatic restore /mnt/nas-backup/borg --target /tmp/ --source /mash/traefik/ssl/acme.json

Git Workflow

• server_setup_home is a standalone git repo — commit changes independently
• Service repos (weather-cli, etc.) are separate git repos — commit independently
• mash-playbook is a git submodule — never commit changes to it
• inventory/host_vars/*/vars.yml is gitignored — never commit secrets
• deploy_repos/ is gitignored — local cache only
• inventory/group_vars/ is gitignored — generated by deploy.sh
• mash-playbook/setup.yml is gitignored — generated by deploy.sh

Gitignore Rules

inventory/host_vars/*/vars.yml          # Secrets
inventory/group_vars/*/vars.yml         # Secrets
mash-playbook/setup.yml                 # Generated
mash-playbook/requirements.yml          # Generated
inventory/group_vars/                   # Generated
roles/galaxy                            # Installed roles
deploy_repos/                           # Local repo cache

Ansible Config

ansible.cfg sets roles_path = mash-playbook/roles:roles:matrix_playbook/roles so MASH, custom, and Matrix roles are all discoverable. Pipelining enabled for performance.

Traefik Debugging Quick Reference

When a service returns 404 or a middleware error:

1. Check container is running: docker ps | grep <service>
2. Check Traefik sees the router (requires API enabled):

   curl -s http://localhost:8080/api/http/routers | jq '.[] | select(.name | contains("<service>"))'
   

3. "middleware X does not exist" → the label references a middleware that isn't defined in any file provider. Check traefik_configuration_extension_yaml in vars.yml and verify the middleware name matches exactly.
4. "404 page not found" → Traefik has no matching router for the hostname. Check that SERVICE_NAME_container_labels_traefik_hostname and SERVICE_NAME_container_labels_traefik_rule produce the expected host match.
5. Verify labels: docker inspect <container> --format '{{json .Config.Labels}}' | jq
6. Check network: Verify container is on the traefik network: docker inspect <container> --format '{{json .NetworkSettings.Networks}}' | jq
7. Check Traefik logs: docker logs mash-traefik 2>&1 | grep <service> (or grep for error, middleware, router)

Troubleshooting

# Check service status
sudo systemctl status mash-<service>.service
docker ps | grep <service>

# View logs
sudo journalctl -u mash-<service>.service -f
docker logs mash-<service>

# Verify port usage
sudo lsof -i :<port>

# Check Traefik routing
docker logs mash-traefik
dig <hostname>

# Rebuild specific service
./deploy.sh install <service>

"I can't find the project" / File not found errors

• Always use the absolute path: /workspaces/opencodeTestProject/projects_ws/server_setup_home/
• Never use relative paths like projects_ws/server_setup_home — the LLM may not know the workspace root
• Verify with Read on the absolute directory path before attempting any file operations
• If the absolute path doesn't exist, fallback to glob with **/server_setup_home/deploy.sh

Ansible errors

• G1-G13 above cover the most common pitfalls — read them before making changes
• If docker build fails with DNS errors, see G6
• If when: conditions fail, see G1 (use | length > 0 for strings)
• If file paths on the Pi are wrong, see G10 — MASH services use /mash/, custom services use /opt/

deploy.sh issues

• deploy.sh is the single entry point — never run ansible-playbook directly
• After adding a new service, all 4 locations in deploy.sh must be updated (sync_repos, inject_custom_roles, run_ansible, case statement)
• mash-playbook/setup.yml is generated by deploy.sh — never edit it manually
• Per-service start/stop commands: custom roles use ansible tags, MASH roles fall back to systemctl over SSH
• patch_borg_service() is automatically called from install-all and setup-all. For standalone borg deployment, run ./deploy.sh patch-borg manually.

File References

All paths below are relative to PROJECT_ROOT (/workspaces/opencodeTestProject/projects_ws/server_setup_home/).

Last updated: June 2026 Reference role: roles/weather_mcp_server/