kukeon

Imported from eminwux/kukeon (AGENTS.md). Install upstream with npx skills add eminwux/kukeon. Copyright stays with the author.

kukeon

Build, test, and smoke-test conventions for anyone — human or agent — working on this repo.

Conventions

• License header: Apache 2.0 + SPDX copyright header on every new Go file.
• Shared error sentinels: defined in internal/errdefs/. Don't declare new errors.New in package code when the concept is reusable — add it to errdefs instead.
• Tests colocated with source: foo.go + foo_test.go.

The two default realms

kuke init provisions two realms, each mapped to its own containerd namespace (see internal/consts/consts.go):

The kukeond daemon runs inside the kuke-system realm — specifically as a container inside the cell kuke-system / kukeon / kukeon / kukeond (realm / space / stack / cell). The default realm is deliberately left user-owned so kuke purge --cascade on it can never take down the daemon.

CLI use-case reference

The full set of operator workflows the kuke CLI must support — each with its command sequence and behavioral invariants (exit codes, side effects, idempotency, error paths) — lives in docs/cli-use-cases.md. That document is workflow-oriented (not alphabetical by command) and is the source of truth for UX expectations the smoke test below does not cover, including image management, cell teardown verbs (stop/kill/delete/purge), and the cascade-purge / divergent-spec edge paths.

Local smoke test: rebuild and re-run kuke init

When a change touches the build path, the daemon, or anything under /opt/kukeon, run this end-to-end before opening a PR.

Prerequisites

make dev-init requires one daemon to be running on the host:

• Standalone containerd at /run/containerd/containerd.sock — used by kuke build and kuke init. This is not the docker-private containerd at /var/run/docker/containerd/containerd.toml; pgrep containerd may show that one even when the system socket is missing. If ls /run/containerd/containerd.sock returns no such file, start it with service containerd start (or systemctl start containerd). On hosts with no init script and no systemd (e.g., the agent dev container), launch the binary directly: containerd > /tmp/containerd.log 2>&1 &.

No docker daemon is required. kuke build invokes the standalone kukebuild binary on the host — a BuildKit-as-library image builder that writes straight into the target realm's containerd namespace over the same /run/containerd/containerd.sock. make dev-init's build phase produces kukebuild alongside kuke and places it on PATH (/usr/local/bin) so the sudo ./kuke build step resolves it; a missing kukebuild fails fast with kuke build's "not found on PATH" message.

A containerd failure surfaces as a confusing error several phases into the script — failed to connect to containerd: ... dial unix:///run/containerd/containerd.sock: timeout. Bring it up before invoking make dev-init to skip the rabbit hole.

When run from inside a kukeon-dev-root cell (the canonical agent workflow), make dev-init automatically redirects the kukeond socket to /run/kukeon-dev/ so it doesn't clobber the parent host's kuke attach plumbing — see docs/dev-init.md for the full bare-host vs. nested-mode contract.

One-shot: make dev-init

make dev-init

scripts/dev-init.sh composes the full re-bootstrap loop: make kuke kukebuild + kukeond symlink, writes ./kukeond-dev.yaml + ./kuke-dev.yaml idempotently (guarded by [ ! -e ]) and threads --server-configuration (admin commands) + KUKE_CONFIGURATION (client + parity commands) through every invocation, kuke daemon reset of the prior cell, kuke build -t kukeon.internal/kukeond:v0.0.0-dev --realm kuke-system . building the image straight into the kuke-system realm's containerd namespace, kuke init --kukeond-image kukeon.internal/kukeond:v0.0.0-dev, and the daemon-parity check below. kukeon.internal is the ICANN-reserved, non-routable internal host the epic (#1063) adopts for every locally-built kukeon image — a stray pull of one fails fast against the reserved TLD instead of silently hitting Docker Hub. The published default (KukeondImageRepo = ghcr.io/eminwux/kukeon in cmd/config/version.go) is unchanged; only the local dev-loop ref moves. The script is idempotent — re-running on a healthy host produces a clean re-bootstrap.

The daemon-parity tail of the output (the regression guard) must read:

NAME         STATE  AGE  NAMESPACE
-----------  -----  ---  -------------------------
default      Ready  <age>  default.dev.kukeon.io
kuke-system  Ready  <age>  kuke-system.dev.kukeon.io

Both kuke get realms -o wide and kuke get realms -o wide --no-daemon must produce the same shape (same columns, same rows, same STATE, same NAMESPACE); AGE may differ by ms between the two invocations but renders at second granularity, so a single re-run normally produces byte-identical output. The tail above is the KUKEON_PROFILE=dev shape — export KUKEON_PROFILE=dev before make dev-init to get the dev profile (*.dev.kukeon.io namespaces, -o wide parity tail with the NAMESPACE column, cgroupPath: /kukeon-dev/...); this non-default profile is the canonical dev-init smoke since #285 phase 3. With KUKEON_PROFILE unset, the script deliberately keeps the historical default profile and the tail prints plain kuke get realms output (NAME STATE AGE, namespaces *.kukeon.io, cgroupPath: /kukeon/...) — same two-row parity contract, different columns and suffix. CGROUP no longer surfaces in either the default or wide table (epic:get retires it from both); the script's follow-up kuke get realms -o yaml | grep cgroupPath step prints cgroupPath: /kukeon-dev/default and cgroupPath: /kukeon-dev/kuke-system so the dev cgroup-root flip is visible too. If only the --no-daemon list is populated, the daemon's view of /opt/kukeon diverged from the in-process controller — that's the bind-mount regression this check catches.

If your change touches anything the daemon reads, this check must be in the PR's test plan. Reviewer agent will flag PRs that miss it.

After the daemon-parity tail, scripts/dev-init.sh also runs a PTY-driven kuke attach smoke against a kuketty-wrapped cell (dev-init-attach/ds/dks/cattach) — this exercises the attachable + kuketty socket pipeline end-to-end (post-create chown, 0o660 root:kukeon socket mode, JSON-RPC + SCM_RIGHTS handshake, clean ^]^] detach). Changes touching internal/controller/runner/attachable.go or any attach/kuketty socket path are validated by this phase; the no-op-StartCell wrong-mode reapply branch (#935/#936) is unreachable from CLI verbs (the higher-level controller.StartCell rejects already-running cells), so its specific correction behavior is covered exclusively by the unit tests in internal/controller/runner/attachable_test.go.

Post-init: kuke status

kuke status is the canonical equivalent of the manual diff ritual above — one command covering daemon liveness, host pre-flight, state consistency, and the daemon-parity walk across every resource kind. See docs/site/cli/kuke-status.md for the full reference (sections, flag table, exit codes, JSON shape).

The two-line kuke get realms diff above stays the minimal pinned regression guard the make dev-init tail prints; kuke status is the broader smoke an operator runs after kuke init to surface anything the two-realm tail won't catch (cgroup delegation gone, CNI binaries gone, divergent secrets/blueprints/configs, residual containerd namespaces from a half-cleaned kuke uninstall).

Post-reboot cgroup recovery

A host reboot wipes the cgroup tmpfs while cell metadata under /opt/kukeon/data/... survives. The background reconciler ((*Exec).ReconcileCell) re-runs ensureCellCgroup on the next tick for any cell whose cgroup is absent AND whose persisted Status.ReadyObserved is true — re-creating the cgroup directory and re-asserting subtree controllers in the same pass.

Half-CreateCell cells (a kuke create cell that crashed before markCellReady could close the ReadyObserved latch) are deliberately excluded from the heal: the in-flight CreateCell will finish its own ensureCellCgroup path under the per-cell lock, and the reconciler stepping in would race that flow.

Consequence: after a reboot, no operator-driven kuke start <name> is required per cell — the next reconcile tick re-creates each previously-Ready cell's cgroup automatically. The make dev-init smoke does not exercise this path; reboot recovery is covered by the unit tests under internal/controller/runner/reconcile_heal_test.go.

Manual phases (fallback)

To run individual phases by hand — e.g. while debugging a single phase — invoke them in order:

1. Tear down the existing runtime. sudo ./kuke daemon reset stops + deletes the prior kukeond cell and clears /run/kukeon/kukeond.{sock,pid}. User-realm data under /opt/kukeon/data/default is left intact, so kuke purge --cascade on default can never take down the daemon. Pass --purge-system to additionally wipe /opt/kukeon/data/kuke-system for a fully clean re-bootstrap.

2. Build the binaries.

   rm -f kuke kukeond kukebuild
   make kuke kukebuild # produces ./kuke and ./kukebuild
   ln -sf kuke kukeond # kukeond is argv[0]-dispatched from the same binary
   sudo ln -sf "$(pwd)/kukebuild" /usr/local/bin/kukebuild # kuke build resolves kukebuild via PATH

3. Build the local kukeond image into the kuke-system realm (no registry push).

   kuke build invokes kukebuild (BuildKit as a library), which writes the image straight into the realm's containerd namespace — no docker daemon and no --from-docker loader hop. The kuke-system realm must already exist (created by an earlier kuke init pass).

   sudo ./kuke build --build-arg VERSION=v0.0.0-dev -t kukeon.internal/kukeond:v0.0.0-dev --realm kuke-system .
   sudo ctr -n kuke-system.kukeon.io images ls | grep kukeon.internal/kukeond

4. Run kuke init.

   sudo ./kuke init --kukeond-image kukeon.internal/kukeond:v0.0.0-dev

   Expected tail:

       - cell "kukeond": created (image kukeon.internal/kukeond:v0.0.0-dev)
       - cell cgroup: created
       - cell root container: created
       - cell containers: started
   kukeond is ready (unix:///run/kukeon/kukeond.sock)
   

Inspecting the running daemon

# Two bind mounts expected: /run/kukeon and /opt/kukeon (host→container, same path).
sudo ctr -n kuke-system.kukeon.io container info \
    kukeon_kukeon_kukeond_kukeond | \
    python3 -c "import sys,json; print(json.dumps([m for m in json.load(sys.stdin)['Spec']['mounts'] if m.get('type')=='bind'],indent=2))"

# Expected process line:
# /bin/kukeond serve --socket /run/kukeon/kukeond.sock --run-path /opt/kukeon
ps -ef | grep '[k]ukeond serve'

Recovering from a failed kuke uninstall

If kuke uninstall reports skipped (realm purge failed) and filesystem + user/group cleanup skipped: residual containerd namespace prevented teardown, see the Full per-host teardown section in docs/cli-use-cases.md for the half-cleaned-host gate explanation and the namespace-cleanup + re-run recovery procedure.