Imported from joelhooks/joelclaw (skills/system-architecture/SKILL.md). Install upstream with npx skills add joelhooks/joelclaw --skill system-architecture. Copyright stays with the author.

System Architecture (Canonical Topology)

This skill is the single source of truth for joelclaw system wiring.

Freshness notes:

• 2026-08-23 recall cutover: memory_observations and the system-log JSONL/slog path are retired. Production joelclaw recall composes lane-separated flowing reflections, flowing observations, and curated Brain pages. Every request has explicit scope and access. Chorus/Rhizomatic is parked with no briefing injection or live claims; service stop still needs steering sudo. Claude auto-memory is a pointer index, not a content store.

• 2026-07-10 topology: the old Panda-centric mental model is migration debt. Current responsibility lanes:

• Central: Flagg/Mac Studio. It is authoritative for agent-mail and Run capture ingress; verify each remaining service family during migration.

• Relay: Panda account-bound leftovers and explicit decommission blockers.

• Satellite: Blaine/other capture clients, connectors, outboxes, and transcript backup freshness.

Treat older “Panda is the whole system” wording as stale unless re-verified against live receipts.

Use it for:

• "why did this run / not run"
• "which worker handles this function"
• "what is listening on port X"
• "how does event Y flow"
• "where does this Run/capture/memory record go"
• "is this Central, Relay, satellite, or shadow runtime work"
• full-stack routing/debug across CLI → Inngest → workers → gateway → telemetry

Ground-Truth Scope + Evidence Snapshot

This document is grounded in direct reads of:

• apps/docs-api/src/index.ts
• packages/restate/Dockerfile
• packages/restate/src/index.ts
• packages/restate/src/workflows/dag-orchestrator.ts
• packages/agent-execution/src/microvm.ts
• packages/system-bus/src/serve.ts
• packages/system-bus/src/inngest/functions/index.host.ts
• packages/system-bus/src/inngest/functions/index.cluster.ts
• packages/system-bus/src/inngest/client.ts
• infra/worker-supervisor/src/main.rs
• ~/Library/LaunchAgents/com.joel*.plist
• k8s/* (all files)
• infra/pds/values.yaml
• packages/gateway/src/daemon.ts
• packages/gateway/src/channels/*.ts
• ~/.joelclaw/gateway/AGENTS.md
• ~/.joelclaw/gateway/.pi/settings.json
• ~/.local/caddy/Caddyfile
• ~/.colima/default/colima.yaml + colima status --json
• packages/cli/src/cli.ts, packages/cli/src/config.ts, packages/cli/src/inngest.ts
• packages/system-bus/src/observability/* (key files: emit.ts, otel-event.ts, store.ts)
• packages/telemetry/src/emitter.ts
• packages/system-bus/src/lib/pi-output.ts
• packages/inference-router/src/tracing.ts
• CONTEXT.md
• docs/gateway.md
• docs/inngest-functions.md
• docs/runbooks/satellite-rig-setup.md
• docs/runbooks/flagg-gate5-staged-migration.md
• infra/central/README.md
• docs/prd-rhizomatic-network-canary.md
• infra/central/launchd/*.plist.template
• scripts/joelclaw-capture-session.ts
• scripts/joelclaw-capture-codex-session.js
• ADRs in ~/Vault/docs/decisions/ (required + topology-adjacent)
• canonical OTel events plus durable Brain .svx receipts (the former system-log JSONL is archived and retired)

Related docs verified

• docs/architecture.md — Restate/Firecracker runtime + workload execution flow
• docs/deploy.md — Restate worker deploy + auth/identity/PVC procedures
• docs/cli.md — workload command tree + runtime bridge
• docs/observability.md — not inspected in this update

Refresh receipt: 2026-06-15

This refresh folds in work from:

• Central vocabulary + Project Thread docs (6b3a1b05, CONTEXT.md, docs/gateway.md)
• Flagg Central scaffold and Gate 5 migration runbooks (6e02a6cd, d36b52f2, infra/central/*)
• worker-hosted Run capture (f06501a8, docs/inngest-functions.md, packages/system-bus/src/serve.ts)
• Inngest SDK hardening + connect-mode recovery (d7dd7788, 6c5d2a8e)
• Talon paging/debounce hardening (c03edc5c, 1f086cfb, d26351cf)
• satellite rig setup for Blaine/Flagg (bc5738f3, 9cc02f6e)
• historical Rhizomatic/Chorus network canary (6bebf5b1, docs/prd-rhizomatic-network-canary.md), parked by the 2026-07-17 decision

0) Current Operator Map

The old mental model was "Panda is joelclaw." That is no longer precise enough.

Use these terms:

Current authority split (verified 2026-07-10):

• Flagg is authoritative for agent-mail and Run capture ingress. The agent-mail daemon binds Flagg loopback; Blaine and Panda use SSH connector LaunchAgents so every joelclaw mail client reaches the same mailbox without exposing the service on the tailnet.
• Panda is migration debt plus Relay responsibilities. Its independent agent-mail daemon and Talon are removed. A reboot-survivable SSH connector now binds Panda IPv4 loopback 127.0.0.1:3111 and forwards legacy /api/runs and /webhooks ingress to Flagg. The legacy system worker still owns the IPv6 listener until its system LaunchDaemon is booted out with sudo.
• Satellites stay thin. They run Pi/Codex/Claude, local capture hooks, and connectors to Central. Do not install independent stateful Central services on a satellite without a specific reason.
• SQLite indexes Runs. NAS/local Run blobs are the source of truth; sessions.db is the compact live FTS index. The retired Typesense runs_dev and run_chunks_dev collections must not be recreated.
• Flagg splits book search from the operational Typesense node. The system LaunchDaemon on 127.0.0.1:8108 holds operational projections. The user LaunchAgent com.joelclaw.typesense-books on 127.0.0.1:8110 holds docs and docs_chunks_v2. DOCS_TYPESENSE_URL routes book readers and writers. A tailnet-only TCP forward exposes 8110 to Blaine.
• The book node is not a replica. A Typesense replica would copy every collection and repeat the same memory and startup cost. The separate process gives book indexing its own failure and restart boundary.

Cutover rule: avoid split-brain. Panda and Flagg must not both accept authoritative writes for the same Central service family. Gate 5 permits shadow smoke tests and migration rehearsal, but authority flips only inside an approved freeze/cutover window.

1) Physical Topology

Legacy Central snapshot: Panda

Mac Mini "Panda" (host macOS)
├─ launchd services (gateway, worker supervisor, caddy, talon, agent-mail, etc.)
├─ Colima VM (driver: VZ, arch: aarch64, runtime: docker, VM IP: 192.168.64.2)
│  └─ Talos node: joelclaw-controlplane-1 (k8s v1.35.0, internal IP 10.5.0.2)
│     ├─ namespace: joelclaw
│     │  ├─ inngest (StatefulSet + NodePort 8288/8289)
│     │  ├─ redis (StatefulSet + NodePort 6379)
│     │  ├─ typesense (StatefulSet + ClusterIP 8108)
│     │  ├─ restate (StatefulSet + NodePort 8080/9070/9071)
│     │  ├─ system-bus-worker (Deployment + ClusterIP 3111)
│     │  ├─ restate-worker (Deployment + ClusterIP 9080; full agent image + Firecracker)
│     │  ├─ dkron (StatefulSet + ClusterIP 8080)
│     │  ├─ docs-api (Deployment + NodePort 3838)
│     │  ├─ livekit-server (Deployment + NodePort 7880/7881)
│     │  ├─ bluesky-pds (Deployment + NodePort 3000)
│     │  └─ minio (StatefulSet + NodePort 30900/30901)
│     └─ namespace: aistor
│        ├─ aistor operator (Deployments: adminjob-operator, object-store-operator)
│        └─ aistor-s3 object store (StatefulSet + NodePort 31000/31001)
├─ Caddy reverse proxy (tailnet HTTPS fan-in)
├─ Gateway daemon (embedded pi session)
├─ Firecracker substrate (requires Colima nestedVirtualization=true for /dev/kvm; OFF by default — unstable under load)
└─ NAS "three-body" (NFS tiers per ADR-0088)

Flagg shadow / next Central target

Mac Studio "Flagg" (host macOS; target Central host)
├─ system tailscaled path required for cutover
├─ Central Service Account: joelclaw:staff
├─ service root: /Users/Shared/joelclaw/
│  ├─ services/{redis,typesense,inngest,minio}/
│  ├─ backups/central/
│  ├─ logs/central/
│  └─ src/joelclaw/ (service-owned checkout)
├─ shadow Compose stack (not authoritative)
│  ├─ Redis 7-alpine
│  ├─ Typesense 30.1
│  ├─ Inngest self-hosted
│  ├─ Restate 1.6.2 (Docker named volume for data)
│  └─ MinIO smoke surface
├─ system LaunchDaemon templates
│  ├─ com.joelclaw.central.colima
│  ├─ com.joelclaw.central.compose
│  ├─ com.joelclaw.central.health
│  └─ com.joelclaw.central.nas-mounts
├─ Chorus/Rhizomatic (parked historical canary)
│  ├─ no session briefing injection and no live claims
│  ├─ com.joelclaw.chorus-rhizomatic may remain loaded only until steering completes the sudo stop
│  └─ old 4821/7331 endpoints are historical troubleshooting context, not active dependencies
└─ NAS "three-body" proof path
   ├─ /Volumes/nas-nvme -> three-body:/volume2/data
   └─ /Volumes/three-body -> three-body:/volume1/joelclaw

Flagg Gate 4 is complete: shadow Central recovered after hard reboot with no GUI login. Gate 5 is not complete until Flagg owns Central state, workers, endpoints, and verification while Panda is frozen as rollback-only.

Known runtime endpoints

• Colima VM IP: 192.168.64.2 (colima status --json)
• Kubernetes API (stable operator tunnel): https://127.0.0.1:16443
• Talos API (stable operator tunnel): 127.0.0.1:15000
• Tailnet hostnames seen in config:
  • panda.tail7af24.ts.net (Caddy routes)
  • pds.panda.tail7af24.ts.net (PDS values)
  • flagg.tail7af24.ts.net (Mac Studio shadow / target Central host)
  • blaine.tail7af24.ts.net (satellite)
• Current live Run capture URL for satellites:
  • https://panda.tail7af24.ts.net/api/runs
  • served by Panda host system-bus worker on localhost:3111
  • do not use http://panda:3000 or http://panda.tail7af24.ts.net:3000; Panda has no durable Central web listener there.

Tailscale mesh state

• tailscale status --json failed in this environment: UNKNOWN — needs manual verification

2) Process Inventory (Long-Running)

Herdr launch-domain split

• com.joelclaw.herdr-server is the default interactive server. It runs as a per-user LaunchAgent in gui/<uid> so pane descendants inherit the Aqua bootstrap namespace and can reach user Keychain services.
• com.joelclaw.herdr-system-server is the boot-safe automation server. It remains a system LaunchDaemon and owns only the named system Herdr session.
• Never install the default server as a system LaunchDaemon. A UserName on a LaunchDaemon changes the Unix identity, not the launchd bootstrap namespace. Native macOS clients in those panes cannot resolve the user's com.apple.securityd.xpc service.
• The installer and cutover contract lives in infra/install-herdr-default-launchagent.sh. The durable rationale is .brain/resources/herdr-launch-domain-contract.svx.

Host launchd inventory (Panda live Central snapshot)

Snapshot source: launchctl print gui/$(id -u)/<label> and plist inspection.

Flagg Central launchd scaffold

Source: infra/central/README.md, infra/central/launchd/*.plist.template, and docs/prd-rhizomatic-network-canary.md.

These labels are part of the Flagg Central shadow/cutover scaffold. They are not proof that Flagg is authoritative.

Flagg reboot acceptance rule: Central is not eligible for cutover until infra/central/scripts/reboot-proof.sh passes from another machine after hard reboot with no GUI login.

Process supervision behavior: worker-supervisor

Source: infra/worker-supervisor/src/main.rs

• Default config:
  • worker dir: ~/Code/joelhooks/joelclaw/packages/system-bus
  • command: bun run src/serve.ts
  • port: 3111
  • health endpoint: /api/inngest
  • sync endpoint: /api/inngest (PUT)
  • health interval: 30s
  • restart after 3 consecutive health failures
  • restart backoff: 1s → 30s max
• Pre-start kills stale process on port 3111.
• Runs host import preflight before spawn:
  • bun --eval "await import('./src/inngest/functions/index.host.ts');"
  • on failure, skips spawn and retries with exponential backoff
• Loads env from ~/.config/system-bus.env plus leased secrets.
• Forces WORKER_ROLE=host for the supervised host worker.
• Emits OTEL events via CLI on supervisor failures/restarts:
  • worker.supervisor.preflight.failed
  • worker.supervisor.worker_exit
  • worker.supervisor.health_check.restart

Worker supervision split note

• Talon is running (com.joel.talon), but host worker is still launched via com.joel.system-bus-worker -> worker-supervisor.
• ADR + system-log indicate Talon can defer worker supervision during coexistence.

Kubernetes process inventory

Node

• joelclaw-controlplane-1 (Talos v1.12.4, k8s v1.35.0, internal IP 10.5.0.2)

Core services

Restate / Firecracker runtime note

• deployment/restate-worker is the current durable execution worker. The image bundles Bun + Node + pi + codex, the full repo checkout, and 76 symlinked skills.
• Runtime auth/identity come from secret/pi-auth and configmap/agent-identity, which recreate /root/.pi/agent/auth.json plus the joelclaw identity chain inside the pod.
• Firecracker is enabled in-pod via privileged access to /dev/kvm on Colima VZ. The /dev/kvm hostPath mount uses type "" (optional) so the pod starts without it when nestedVirtualization is off.
• Persistent microVM assets live on PVC firecracker-images, mounted at /tmp/firecracker-test for kernel, rootfs, and snapshot files.
• Retry caps (2026-03-17): dagWorker maxAttempts=5, dagOrchestrator maxAttempts=3. Prevents Restate journal poisoning from infinite retries after code changes or infrastructure failures.
• Colima stability: nestedVirtualization is OFF by default (crashes VM under Docker build load). Toggle ON only for Firecracker testing sessions, then toggle OFF. See k8s skill for recovery procedures.

Control-plane access

• kube API exposed locally at 127.0.0.1:16443 via com.joel.kube-operator-access (ssh -S none -o ControlPath=none -L 16443:10.5.0.2:6443)
• Talos API exposed locally at 127.0.0.1:15000 via the same daemon (ssh -S none -o ControlPath=none -L 15000:10.5.0.2:50000)
• NodePort/runtime app ports still come from Colima/Lima forwarding; the operator daemon exists specifically because the direct host-published 6443 path was not boring after the rebuild

3) Worker Architecture (Role Split + Registration)

Source files:

• packages/system-bus/src/serve.ts
• packages/system-bus/src/inngest/functions/index.host.ts
• packages/system-bus/src/inngest/functions/index.cluster.ts
• packages/system-bus/src/inngest/client.ts

Role model

• WORKER_ROLE parsed as host (default) or cluster.
• Registered function set is role-dependent:
  • host uses hostFunctionDefinitions
  • cluster uses clusterFunctionDefinitions

Ground-truth counts

• Measured from the imported definition arrays on 2026-07-29:
  • Host function set: 163
  • Cluster function set: 14
• gateway/external-delivery-canary is host-only. It appends synthetic message.requested events and watches the canonical event log. It does not run in the gateway driver, transport daemon, or gateway session.
• Cluster subset functions:
  • agent/approval-request, agent/approval-resolve
  • todoist-comment-notify, todoist-task-completed-notify, todoist-task-created-notify
  • github-workflow-run-completed-notify, github-package-published-notify
  • webhook-subscription-dispatch-generic
  • webhook-subscription-dispatch-github-workflow-run-completed
  • memory/observe-session
  • queue/observer, queue/observer-requested
  • swarm-orchestrator, swarm-agent-exec

App registration isolation

From inngest/client.ts:

• app id resolves to:
  • system-bus-host when role is host
  • system-bus-cluster when role is cluster
• explicit INNGEST_APP_ID overrides role-derived id.

This prevents host and cluster workers from overwriting each other’s function graphs.

serveHost behavior

From serve.ts:

• host role default serveHost: http://host.docker.internal:3111
• cluster role default serveHost: unset (connect-mode default)
• INNGEST_SERVE_HOST overrides either role.
• INNGEST_CONNECT_MODE=1|true|yes starts inngest/connect with instanceId=system-bus-<role>-<hostname> and maxWorkerConcurrency=8.
• /api/inngest now explicitly allows only GET, POST, and PUT; PATCH, OPTIONS, and DELETE return 405 with Allow: GET, POST, PUT.
• /api/inngest logs bounded request summaries for failed or ambiguous POST/PUT callbacks, including safe query/debug keys and body shape, not raw huge payloads.
• Bun server idleTimeout=255 because registration PUTs can exceed the default 10s while the self-hosted runtime is under cron/backlog pressure.

Kubernetes cluster worker manifest sets:

• INNGEST_BASE_URL=http://inngest-svc:8288
• INNGEST_SERVE_HOST=http://system-bus-worker:3111
• TYPESENSE_URL=http://typesense:8108
• image: ghcr.io/joelhooks/system-bus-worker:d7dd7788 (recorded after Inngest SDK advisory hardening)

Panda host worker config should advertise an SDK callback URL the Inngest pod can actually reach. Current docs call out INNGEST_SERVE_HOST=http://100.93.201.72:3111 on Panda. Do not assume host.lima.internal or host.docker.internal works from inside Talos unless a live pod-to-host probe proves it.

Registration mechanics

• Worker exposes GET|POST|PUT /api/inngest.
• Worker sends a delayed self-sync PUT /api/inngest ~5s after startup.
• worker-supervisor also performs startup PUT sync.

Host is primary today

From index comments + function lists:

• ADR-0089 transition: host remains authoritative for broad function ownership.
• Cluster is intentionally limited to cluster-safe subset (12 functions).

4) Event Flow (CLI → Inngest → Worker → Completion)

Canonical flow: joelclaw send

1. CLI joelclaw send <event> calls Inngest.send().
2. Inngest.send() POSTs event JSON to:
   • ${INNGEST_URL}/e/${INNGEST_EVENT_KEY}
   • default: http://localhost:8288/e/<key>
   • 2026-06-18: Pi memory-enforcer observe events must follow this keyed Event API shape too; never POST to bare /e/. The extension resolves INNGEST_EVENT_KEY from env/local env files and skips observe emission when no key is available.
3. Inngest server persists the event and resolves matching function triggers.
4. Inngest dispatches function steps to the worker app graph that owns that function ID:
   • host app (system-bus-host) for 101-host set
   • cluster app (system-bus-cluster) for 12-cluster subset
5. Worker handles callbacks via /api/inngest (Hono + inngest/hono handler).
6. Each step.run result is memoized by Inngest; next step executes when prior completes.
7. Completion/failure is queryable via GraphQL (/v0/gql) and CLI commands (runs, run, event, events).

Run capture flow: Machine hook → Central /api/runs → memory/run.captured

1. Runtime-native hook captures only the new Run slice:
   • Pi extension for Pi
   • Claude Code Stop hook via scripts/joelclaw-capture-session.ts
   • Codex hook/helper via scripts/joelclaw-capture-codex-session.js
   • server-side runtimes can call capture inline instead of shelling through a hook.
2. The hook posts JSON to Central:
   • current live URL for satellites: POST https://panda.tail7af24.ts.net/api/runs
   • Panda serves this through the host system-bus worker on localhost:3111
   • auth: Authorization: Bearer <~/.joelclaw/auth.json token>
3. packages/system-bus/src/serve.ts validates the bearer token against the persistent local SQLite registry ~/.joelclaw/capture-auth.db. The worker refreshes this four-row registry from Typesense machines_dev in the background every five minutes. Known-token requests never wait for Typesense; an unknown token gets one bounded migration lookup.
4. Worker writes the raw source of truth through @joelclaw/memory#writeRunBlob:
   • default dev store: ~/.joelclaw/runs-dev/<user>/<yyyy-mm>/<run-id>.jsonl
   • companion metadata includes user_id, machine_id, agent_runtime, parent/conversation IDs, tags, byte count, and SHA-256.
5. Worker emits memory/run.captured to Inngest.
6. packages/system-bus/src/inngest/functions/memory/run-captured.ts appends Run metadata, chunks, and FTS rows to SQLite sessions.db in one transaction.
7. If POST fails from a Machine, the hook writes the POST body into ~/.joelclaw/outbox/; the Machine does not become Central just because capture is temporarily offline.
8. If raw blobs exist but SQLite is stale, recover Inngest/worker registration first, then backfill with scripts/backfill-session-index.ts. Do not replay thousands of memory/run.captured events casually.
9. The public Vercel app is not a Run ingress or search surface. Its POST /api/runs and POST /api/runs/search routes return 410. Machine hooks must use the Central system-bus endpoint, and operators must use joelclaw sessions search.

Recall flow: scoped request → CLI composition root → lane-preserving result

1. Interactive callers run in a trusted GitHub checkout or pass both --project and --workstream. Automatic callers send one exact ComposedRecallRequestV1 through joelclaw recall --request-file - on private stdin.
2. Every request names principal, purpose, allowed privacy tiers, supersession policy, and independent limits for reflections, observations, and curated pages. The deprecated --limit flag is ignored.
3. The CLI reads the pinned flowing-memory artifact and curated critical.db projection, then returns three separate lanes: flowing-reflections, flowing-observations, and curated-pages. Never merge their scores.
4. Unavailable lanes return typed envelopes and exit 3. The gateway shows a generic unavailable marker. Optional Inngest enrichment records redacted telemetry and returns no context. Restate returns a generic degraded marker.
5. Use joelclaw sessions search "<query>" --source both --extract for exact Run/session wording, commands, and transcript evidence. Composed recall is a broad memory surface, not raw transcript search.
6. typesense-recall is config-selected rollback only. Status reports it as a visible degraded warning without turning global health red.

Accepted cluster decision (2026-08-23): automatic recall inside the system-bus and Restate cluster images is intentionally disabled/degraded after retiring old observation recall. packages/recall is present for module resolution, but the images do not receive a local semantic store or compiled joelclaw recall binary. Callers must take their explicit optional-enrichment degrade path. This is not a hidden deployment dependency. Re-enable cluster recall only after a separately reviewed secure remote flowing-read substrate exists.

Queue flow: joelclaw queue emit → Restate drainer → durable dispatch

1. CLI joelclaw queue emit <event> persists a QueueEventEnvelope into Redis stream joelclaw:queue:events and indexes it in sorted set joelclaw:queue:priority.
2. The restate-worker k8s deployment (packages/restate/src/index.ts) starts a deterministic queue drainer beside the channel callback listener.
3. On startup, the drainer claims pending + never-delivered entries via @joelclaw/queue#getUnacked(), reindexes replayable entries, and emits OTEL replay evidence.
4. Each drain tick selects the next priority candidate from the sorted set, resolves its static registry target from packages/queue/src/registry.ts, and POSTs a one-node DAG request to Restate /dagOrchestrator/{workflowId}/run/send.
5. When backlog remains and a dispatch slot frees, the drainer self-pulses immediately instead of waiting for the next QUEUE_DRAIN_INTERVAL_MS heartbeat. That interval is now the idle poll / retry cadence, not a mandatory 2-second tax between successful sends.
6. The current Story-3 bridge re-emits the queue item to its registered Inngest event target inside that one-node DAG request. This is deliberate: the deterministic queue/drainer is proven first; per-family Restate cutovers remain Story 4 work.
7. On accepted Restate dispatch, the drainer acks the queue message; on failure it leaves the message in Redis, applies retry cooldown, and emits queue.dispatch.failed OTEL evidence.
8. If backlog remains in Redis but the drainer stops making progress past QUEUE_DRAIN_STALL_AFTER_MS, it emits queue.drainer.stalled and exits non-zero so k8s restarts deployment/restate-worker. That is the self-heal path for a wedged drainer inside an otherwise-running Bun process.
9. Crash recovery comes from the Redis stream + consumer-group replay path, not from vibes: restart the restate-worker pod, let getUnacked() reclaim the inflight entries, then drain resumes.

Workload flow: joelclaw workload run → Redis → Restate DAG → execution

1. joelclaw workload plan ... --stages-from <file> can load an explicit stage DAG, validate unknown deps/self-deps/duplicates/cycles, and preserve per-stage acceptance gates.
2. joelclaw workload run <plan-artifact> normalizes the selected stage into the canonical workload/requested runtime request.
3. Queue admission writes the request into Redis, where the deterministic drainer forwards it into Restate as a dagOrchestrator/{workflowId}/run/send request.
4. dagOrchestrator executes dependency waves: ready nodes in parallel, chained nodes only after every dependsOn node has terminal output.
5. dagWorker executes the node handler:
   • shell → subprocess work inside the restate-worker pod
   • infer → pi -p --no-session --no-extensions inside the pod, using the mounted auth + identity + skill set
   • microvm → Firecracker boot/restore through /dev/kvm with kernel/rootfs/snapshot files on PVC firecracker-images
6. Each node emits OTEL (dag.node.*), and the workflow emits dag.workflow.* so queue → Restate → execution remains observable.
7. Current truthful limit: the microVM runtime boots and restores snapshots in-cluster, but the broader exec-in-VM workspace drive protocol is still incomplete for general coding slices.

Webhook flow

1. External service posts to /webhooks/:provider.
2. Caddy routes /webhooks/* on localhost:8443 to worker localhost:3111.
3. webhookApp verifies signature, normalizes payload, emits Inngest events (provider/event).
4. Inngest executes subscribed functions.

"Why did this run / not run" trace recipe

1. Start with an existing event or run id. Send a probe only when its destination and effects are explicitly authorized.
2. joelclaw events --prefix <event-prefix> --hours 1
3. joelclaw event <event-id> (fan-out to function runs)
4. joelclaw run <run-id> (step trace + errors)
5. joelclaw runs --count 20 --hours 1
6. joelclaw otel search "<component/action>" --hours 1
7. Validate function ownership in index.host.ts / index.cluster.ts.

5) Port Map (Canonical)

Exposure sources: k8s service manifests, Caddyfile, kubectl get svc, lsof listeners.

Notes

• Host NodePort exposure appears through an ssh listener process (Colima portForwarder=ssh).
• Exact per-port ssh forward command line is UNKNOWN — needs manual verification (process introspection restricted in this environment).
• Numeric ports such as 6379, 8108, 8288, 8289, 8080, 9070, 9071, 9000, and 9001 can refer to Panda live k8s/NodePort surfaces or Flagg shadow Compose loopback surfaces depending on the host. Check hostname, CENTRAL_BIND_ADDR, JOELCLAW_CENTRAL_URL, and the current shell env before diagnosing.

6) Storage Topology

Run blobs / memory capture

• Current dev source of truth: ~/.joelclaw/runs-dev/<user>/<yyyy-mm>/<run-id>.jsonl plus .metadata.json.
• Capture identity is resolved locally by bearer-token hash from ~/.joelclaw/capture-auth.db.
• machines_dev is the enrollment/migration mirror. A five-minute background sync propagates rotation and revocation without putting Typesense on the capture request path.
• Search projection: SQLite sessions.db for Run metadata, compact chunks, and FTS5.
• Failed Machine POSTs spool to ~/.joelclaw/outbox/.
• Future/target contract from CONTEXT.md: Run blobs live on NAS and SQLite remains rebuildable from those blobs.

Redis

• Runtime: k8s StatefulSet (redis:7-alpine, appendonly enabled).
• Primary uses:
  • gateway queue/session keys (joelclaw:events:*, joelclaw:notify:*, joelclaw:gateway:sessions)
  • webhook subscriptions (joelclaw:webhook:*)
  • gateway health mute/streak keys (gateway:health:*)

Typesense

From current runtime code:

• The old observations and brain_graph_nodes recall collections are not production recall sources. typesense-recall remains registered only as the dated, visible degraded rollback adapter.
• machines_dev as the enrollment/migration mirror for the local capture-auth registry
• Flagg operational node: /Users/Shared/joelclaw/data/typesense on 127.0.0.1:8108, owned by the system LaunchDaemon.
• Flagg book node: ~/.joelclaw/typesense-books/data on 127.0.0.1:8110, owned by the user LaunchAgent.
• DOCS_TYPESENSE_URL selects the book node for docs-api, docs ingest, joelclaw docs, and the book subset of unified search. TYPESENSE_URL remains the operational node.
• The book node holds only docs and docs_chunks_v2. Do not recreate these collections on the operational node after cutover.
• The exported JSONL and collection schemas under ~/.joelclaw/backups/typesense-books/ are the migration rollback source. Never copy a live Typesense data directory as a backup.

Firecracker runtime storage

• PVC: firecracker-images
• Mounted in deployment/restate-worker at /tmp/firecracker-test
• Stores:
  • kernel (vmlinux)
  • rootfs (agent-rootfs.ext4)
  • snapshots (snapshots/vm.snap, snapshots/vm.mem)
• Firecracker snapshot restore is currently operator-proven at ~9ms on the Colima VZ nested-virt path.

Inngest state

• StatefulSet PVC mounted at /data
• INNGEST_SQLITE_DIR=/data

docs-api surface

• Deployment: docs-api on NodePort 3838
• Route count: 11 endpoints including /health
• Key routes:
  • GET /search — hybrid chunk search with concept, concepts, doc_id, expand, and assemble
  • GET /docs/search
  • GET /docs
  • GET /docs/:id
  • GET /docs/:id/toc
  • GET /docs/:id/chunks
  • GET /chunks/:id
  • GET /concepts
  • GET /concepts/:id
  • GET /concepts/:id/docs
• Taxonomy surface: 21-concept SKOS graph (10 parents + 11 sub-concepts) with broader, narrower, and related edges.

NAS (ADR-0088 + ADR-0187)

Tiering policy:

• Tier 1 local SSD (hot runtime state)
• Tier 2 NAS NVMe (/Volumes/nas-nvme ↔ /volume2/data)
• Tier 3 NAS HDD (/Volumes/three-body)

Access paths

k8s ↔ NAS networking

k8s pods reach the NAS via a LAN route through the Colima col0 bridge: Talos → Docker NAT → VM col0 → macOS (ip.forwarding=1) → LAN → NAS

The VZ NAT on eth0 does NOT forward LAN traffic. Route persisted in Colima provision + colima-tunnel script: ip route replace 192.168.1.0/24 via 192.168.64.1 dev col0

Always use IP 192.168.1.163, never hostname three-body — DNS doesn't resolve from k8s.

Degradation contract (ADR-0187):

• writes must fallback local -> remote -> queued
• queue spool default: /tmp/joelclaw/nas-queue

Flagg Central shadow storage

• Service root: /Users/Shared/joelclaw/
• Shadow service data:
  • Redis: /Users/Shared/joelclaw/services/redis
  • Typesense: /Users/Shared/joelclaw/services/typesense
  • Inngest: /Users/Shared/joelclaw/services/inngest
  • MinIO smoke data: /Users/Shared/joelclaw/services/minio
  • Restate: Docker named volume ${CENTRAL_RESTATE_VOLUME:-joelclaw-central-restate-data} because macOS/Colima bind mounts reject Restate's Unix socket writes under /restate-data
• Flagg NAS object-storage proof paths:
  • hot: /Volumes/nas-nvme/s3
  • cold: /Volumes/three-body/s3
• Current proved NAS route:
  • Flagg interface en0, IP 192.168.1.10, 10Gbase-T
  • three-body IP 192.168.1.163
  • MTU 1500 is the proved safe Gate 5 state; jumbo 9000 caused an NFS blackhole and is follow-up work.
• CENTRAL_REQUIRE_NAS=1 should only be set after mount proof passes.

Rhizomatic / Chorus (parked historical canary)

• Decision: parked on 2026-07-17. It does not inject session briefings and no real claims may land.
• The Brain is the only memory substrate. Any future claim graph must be a Brain-owned projection.
• Historical artifacts remain at the old Flagg service/checkout/store paths only for cleanup and provenance.
• com.joelclaw.chorus-rhizomatic shutdown still requires the steering sudo step; a loaded process does not make it an active dependency.
• Old 4821/7331 endpoints and tunnels are not live routing guidance.

Vault and durable receipts

• Obsidian vault at /Users/joel/Vault
• Durable project/system receipts belong in Brain .svx.
• The former /Users/joel/Vault/system/system-log.jsonl journal is archived and retired; runtime telemetry is canonical OTel in Typesense/ClickHouse.

7) Networking Topology

Central / Relay / Satellite routing rule

• Live Central URL for capture hooks today: https://panda.tail7af24.ts.net.
• Satellites post Run capture to /api/runs on that URL and may use JOELCLAW_TYPESENSE_URL=http://panda:8108 for direct search/admin helpers.
• Flagg shadow services bind to 127.0.0.1 by default and must not be exposed over Tailscale/LAN until cutover planning says so.
• There is no live Rhizomatic/Chorus routing exception. The old Flagg service and satellite tunnels are parked historical cleanup surfaces.

Caddy reverse proxy routes (from `~/.local/caddy

Truncated - read the full file at https://github.com/joelhooks/joelclaw/blob/HEAD/skills/system-architecture/SKILL.md.