Imported from FreeshardBase/freeshard (.claude/skills/freeshard-build-and-env/SKILL.md). Install upstream with npx skills add FreeshardBase/freeshard --skill freeshard-build-and-env. Copyright stays with the author.

name: freeshard-build-and-env description: Set up and verify a working shard_core development environment. Use when starting from a fresh clone or bare machine, creating a git worktree for an issue, running uv sync, wondering what a justfile recipe does (run-dev, cleanup, get-types, set-version, run-from-backup), reading the Dockerfile, or hitting env-shaped failures (import errors, "config.toml not found", pytest can't bind port 5433, docker network 'portal' already exists, black/ruff not found). TRIGGER on: "set up the repo", "uv sync", ".worktrees", "justfile", "just cleanup", "Dockerfile", "CI installs differently than local", fresh-environment smoke test. SKIP when: actually running the shard or the compose stack, dev server usage, backup/restore ops (use freeshard-run-and-operate); writing or debugging tests and fixtures (use freeshard-testing-and-qa); type-sync policy and controller drift details (use freeshard-ecosystem-contracts); config keys and env-override semantics (use freeshard-config-and-flags).

Freeshard build & environment

Goal: from a bare Linux box (or a fresh worktree) to a state where pytest collects, ruff passes, and you can trust your results. Repo = shard_core, the FastAPI backend that runs on each Freeshard VM ("shard") and orchestrates user apps via Docker Compose behind Traefik.

All commands assume CWD = repo root (/home/ubuntu/projects/freeshard/freeshard on the main dev VM; adjust for your clone/worktree).

When NOT to use this skill

Prerequisites

uv can provision Python itself (uv python install 3.13) if the system Python is too old. Docker + internet are needed even for "just running the tests": the test suite starts a real postgres:17 container and logs into a real container registry (see traps below).

From bare Linux to productive

git clone https://github.com/FreeshardBase/freeshard.git
cd freeshard
uv sync --extra dev            # creates .venv from uv.lock; --extra dev pulls pytest, ruff, etc.
.venv/bin/pytest tests --collect-only -q   # expect "137 tests collected" (origin/main as of 2026-07-03; 134 on the older e55ce51 checkout — count drifts, cross-check freeshard-testing-and-qa), zero errors
.venv/bin/ruff check .         # expect no findings on a clean main

Notes:

• uv sync installs from uv.lock — the single source of truth for dependency versions. Never pip install . into this venv.
• Dev tools live in the optional extra [project.optional-dependencies].dev (pyproject.toml:47-59). Without --extra dev you get no pytest/ruff.
• black is not listed in the dev extra, but just cleanup calls .venv/bin/black. It works anyway because black is a transitive dependency of datamodel-code-generator (see uv.lock). If just cleanup ever reports black missing, you ran uv sync without --extra dev.
• The shard_core.egg-info/ directory you may see at repo root is a setuptools build artifact, gitignored — ignore it.

Worktree discipline (mandatory for new work)

Convention in this repo: every feature/bugfix gets its own worktree under .worktrees/<slug> on its own branch, cut from freshly pulled origin/main. The overnight agent loop assumes this; parallel sessions must not share a checkout.

git fetch origin main
git worktree add .worktrees/<slug> -b <branch-name> origin/main
cd .worktrees/<slug>
uv sync --extra dev            # ALWAYS, before anything else
.venv/bin/pytest tests --collect-only -q   # sanity: env works

Rules:

• ALWAYS run uv sync --extra dev inside the new worktree. Never symlink or share .venv from the main checkout: branches can diverge on pyproject.toml/uv.lock, and a borrowed venv gives silently wrong test results and pollutes the other checkout with __pycache__ writes.
• .worktrees/ is ignored via .git/info/exclude (local-only; not committed in .gitignore). In a fresh clone, add it yourself: echo '.worktrees/' >> .git/info/exclude.
• Two worktrees cannot run the test suite at the same time — see traps (port 5433, compose project name, docker network portal are all host-global).
• local_config.toml is committed, so every worktree has it — but it is resolved relative to CWD (see traps). Run everything from the worktree root.

Repo layout orientation map

freeshard/                     repo root (package name: shard_core)
├── shard_core/                the Python package
│   ├── app.py                 entrypoint (target of `fastapi run/dev`)
│   ├── app_factory.py         create_app() + lifespan = startup/shutdown order
│   ├── settings.py            pydantic-settings: config.toml + local_config.toml + FREESHARD_* env
│   ├── web/                   FastAPI routers by auth level: public/ protected/ internal/ management/
│   ├── service/               business logic (app_installation/, backup.py, crypto.py, ...)
│   ├── database/              psycopg3 async access layer (yoyo migrations applied at startup)
│   ├── data_model/            pydantic models; backend/ = COPIED from controller — never hand-edit
│   └── util/
├── data/                      runtime templates: traefik.yml(+_no_ssl), welcome-log jinja, splash.html, EFF wordlist
├── migrations/                yoyo SQL migrations (shard-core-0001-init.sql)
├── tests/                     pytest suite; tests/config.toml = overlay; tests/docker-compose.yml = postgres:17 on :5433
├── scripts/                   generate_traefik_dyn_config_model.py — raises on purpose, do NOT run (hand-patched output)
├── config.toml                base config, committed (contains live ACR pull credentials — do not "clean up")
├── local_config.toml          dev overlay: path_root="run/", dns.zone="localhost" (CWD-relative, dockerignored)
├── docker-compose.yml         production stack: postgres + traefik + shard_core + web-terminal on network 'portal'
├── Dockerfile                 two-stage uv image build (see anatomy below)
├── justfile                   task recipes (see anatomy below)
├── uv.lock                    dependency source of truth (CI and image both build --frozen from it)
└── .worktrees/                per-task worktrees (ignored via .git/info/exclude)

justfile recipe anatomy

Verified against justfile as of 2026-07-03. just (no args) lists recipes.

Dockerfile anatomy (why the image looks like that)

Two-stage build, verified against Dockerfile as of 2026-07-03:

1. Build stage — ghcr.io/astral-sh/uv:python3.13-bookworm-slim. Runs uv sync --frozen --no-install-project --no-dev first (dependencies only, cached layer), then ADD . . + uv sync --frozen --no-dev (project itself). UV_COMPILE_BYTECODE=1, uv cache mounts. --frozen = build exactly from uv.lock.
2. Runtime stage — python:3.13-slim-bookworm plus:
   • curl — used by the container HEALTHCHECK (curl -f http://localhost/public/health).
   • rclone — the backup engine (rclone crypt → Azure Blob).
   • tini as PID 1 (ENTRYPOINT ["/usr/bin/tini","--"]) — reaps the zombie processes the healthcheck curl otherwise leaves behind (https://github.com/FreeshardBase/freeshard/pull/55).
   • Full Docker install via get.docker.com — shard_core drives its sibling app containers by shelling out to docker/docker compose against the docker.sock mounted from the host (docker-compose.yml mounts /var/run/docker.sock). Only the CLI + compose plugin are used; the daemon inside the image never runs.
   • CMD ["fastapi","run","--port","80","shard_core/app.py"].

CI parity rule

Install from the lockfile, exactly like CI and the image do. As of 2026-07-03, .github/workflows/test.yml on origin/main installs with uv sync --frozen --extra dev and runs uv run ruff check . / uv run pytest tests (changed in https://github.com/FreeshardBase/freeshard/pull/114). Before that, CI used pip install ".[dev]", which ignores uv.lock and resolves latest-at-runtime — a fresh upstream release broke the controller's CI that way with zero code changes, and this repo switched preemptively. If you see red CI with no plausible connection to the diff, first check whether the install step still honors the lock.

Match CI locally:

uv sync --frozen --extra dev   # exactly what CI installs
uv run ruff check .            # CI lint job
uv run pytest tests            # CI test job (needs Docker + internet)

uv sync without --frozen will also update uv.lock if pyproject.toml changed — fine during development, but a lockfile diff you didn't intend belongs out of your PR.

One workflow variable is dead but still present (do not cargo-cult it): TEST_ENV=full/sparse — its consumer was removed, so push and PR runs are identical. A companion dead CONFIG=tests/config.yml was removed by issue #128.

Known traps

15-minute smoke checklist (fresh environment)

Run top to bottom; each step has a pass condition. Stop and fix at the first failure.

# 1. Toolchain present (~1 min)
python3 --version && uv --version && just --version && gh auth status
docker info >/dev/null && docker compose version        # daemon up, compose v2

# 2. Clean, current source (~1 min)
git fetch origin main && git status                      # expect: clean tree
git log --oneline -1 origin/main                         # note the tip you're building against

# 3. Env from lockfile (~2 min)
uv sync --frozen --extra dev                             # pass: exits 0, no resolver output

# 4. Package imports (~10 s)
.venv/bin/python -c "import shard_core.app_factory; print('import ok')"
# pass: prints "import ok" (a CryptographyDeprecationWarning about OFB is known noise as of 2026-07-03)

# 5. Test collection (~30 s, no Docker yet)
.venv/bin/pytest tests --collect-only -q | tail -1       # pass: "N tests collected", zero errors (137 on origin/main as of 2026-07-03; the count drifts — what matters is zero collection errors)

# 6. Lint (~10 s)
.venv/bin/ruff check .                                   # pass: "All checks passed!"

# 7. One real test — exercises pytest-docker, postgres:17 spin-up on :5433,
#    internet, and the committed ACR docker login (~3-5 min)
.venv/bin/pytest tests/test_crypto.py -q                 # pass: all green

# 8. Formatting gate works (~30 s)
just cleanup && git status                               # pass: exits 0; tree still clean on untouched main

If step 7 fails at container startup: check for a leftover shard-core-test compose project or occupied port 5433 (traps table). If it fails at docker login or image pull: the committed ACR credential may be rotated — escalate rather than debug your own change.

Provenance and maintenance

Written 2026-07-03 against branch state origin/main @ 0a40684 (local checkout on fix-profile-billing-fields @ e55ce51). Primary sources: justfile, Dockerfile, pyproject.toml, uv.lock, shard_core/settings.py, tests/conftest.py, tests/docker-compose.yml, docker-compose.yml, .github/workflows/test.yml (origin/main), README.md, https://github.com/FreeshardBase/freeshard/pull/114, https://github.com/FreeshardBase/freeshard/pull/55. Tool versions quoted are the main dev VM's on the same date.

Drift-prone facts — re-verify before trusting: