oakestra

Imported from oakestra/oakestra (AGENTS.md). Install upstream with npx skills add oakestra/oakestra. Copyright stays with the author.

Oakestra — Developer Context for AI Assistants

What Oakestra Is

Oakestra is a lightweight orchestration platform for edge computing. Unlike Kubernetes or K3s, which were designed for cloud-grade machines, Oakestra targets heterogeneous, resource-constrained edge devices. The entire root + cluster stack runs in ~1 GB RAM. A worker node needs only 100 MB RAM and 50 MB disk.

The platform orchestrates containerised workloads (Docker, containerd) and unikernels across a two-level hierarchy: root → clusters → workers. Applications are described via SLA (Service Level Agreement) JSON documents that encode resource constraints, placement preferences, and networking requirements. The scheduler places each microservice on the best-fit worker using a pluggable algorithm (default: cpumemfit.Scheduler).

Current develop version: see version.txt.

Repository Layout

oakestra/
├── root_orchestrator/          # Root-level services (docker-compose)
│   ├── docker-compose.yml
│   ├── override-*.yml          # Composable overrides (network, addons, etc.)
│   └── system-manager-python/ # Python/Flask – main root API
├── cluster_orchestrator/       # Cluster-level services (docker-compose)
│   ├── docker-compose.yml
│   ├── override-*.yml
│   └── cluster-manager/       # Python/Flask – cluster API + MQTT client
├── scheduler/                  # Go – shared scheduler (used by both root & cluster)
├── resource-abstractor/        # Python/Flask – resource DB layer (used by both levels)
├── go_node_engine/             # Go – worker binary (NodeEngine)
├── libraries/                  # Shared Python packages
│   ├── oakestra_utils_library/ # Enums: statuses, scheduling states
│   └── resource_abstractor_client/ # HTTP client for resource-abstractor
├── addons_engine/              # Optional addons subsystem (root only)
├── addons_marketplace/         # Optional marketplace manager
├── csi/                        # Container Storage Interface driver
├── scripts/                    # Startup and install scripts
│   ├── StartOakestraFull.sh    # 1-DOC: root + cluster on one machine
│   ├── StartOakestraRoot.sh    # Root only
│   ├── StartOakestraCluster.sh # Cluster only
│   └── InstallOakestraWorker.sh# Worker binary installer
├── run-a-cluster/              # Compose-based 1-DOC deployment (1-DOC.yaml + shared config)
├── hack/                       # Platform-specific workarounds (e.g. rpi4b-mongo override)
└── SKILLS/
    └── troubleshoot-oakestra.md # AI troubleshooting skill (keep this in sync)

Architecture

Hierarchy

Root Orchestrator  (1 per deployment)
  └── Cluster Orchestrator  (1..N per root)
        └── Worker Node  (1..N per cluster)

Root Orchestrator — services

Cluster Orchestrator — services

Worker Node — binaries

Key Communication Flows

Cluster Registration

cluster_manager → gRPC (port 50052) → system_manager Two-step handshake (CS1/SC1, CS2/SC2). On success, cluster gets an ID and starts a 15s background loop to push resource aggregates.

Job Deployment

1. User POSTs SLA to system_manager (REST :10000)
2. system_manager enqueues a schedule:job task into root_redis (asynq)
3. root_scheduler dequeues, queries root_resource_abstractor for available clusters, runs placement algorithm, calls back system_manager with chosen cluster ID
4. system_manager forwards to cluster_manager (REST :10100)
5. cluster_manager enqueues into cluster_redis; cluster_scheduler picks a worker
6. cluster_manager sends deployment command to worker's NodeEngine via MQTT

Resource Aggregation

cluster_manager → every 15s → system_manager REST: pushes CPU/mem/node counts NodeEngine → MQTT → cluster_manager: pushes per-worker resource metrics

Deployment Modes and Compose Overrides

Startup scripts in scripts/ auto-detect IPs, validate Docker versions, download config files from GitHub, and support version pinning via OAKESTRA_VERSION (branch triggers source build; tag like v0.4.401 uses pre-built images).

Override files are composable. Set OVERRIDE_FILES before running a startup script:

export OVERRIDE_FILES="override-no-addons.yml,override-network-host.yml"

Tech Stack Summary

Development Commands

Python services (system_manager, cluster_manager, resource-abstractor, etc.)

# Install deps for a service (example: system_manager)
pip install -r root_orchestrator/system-manager-python/requirements.txt

# Run tests
pytest root_orchestrator/system-manager-python/tests/
pytest resource-abstractor/tests/
# Note: cluster_manager has no unit tests currently

# Lint (ruff is configured in pyproject.toml at repo root; line-length=100)
ruff check .

Go services (scheduler, NodeEngine)

# Build the scheduler
cd scheduler && go build ./...

# Build NodeEngine
cd go_node_engine && go build -o NodeEngine .

# Run Go tests
go test ./...

Local stack (full root + cluster on one machine)

# Start everything (detects IPs, pulls/builds images)
./scripts/StartOakestraFull.sh

# Start root only
./scripts/StartOakestraRoot.sh

# Start cluster only (set SYSTEM_MANAGER_URL first)
export SYSTEM_MANAGER_URL=<root-ip>
./scripts/StartOakestraCluster.sh

# Pin to a specific version
export OAKESTRA_VERSION=v0.4.411
./scripts/StartOakestraFull.sh

# Build from source branch
export OAKESTRA_VERSION=develop
./scripts/StartOakestraFull.sh

Shared Libraries

• libraries/oakestra_utils_library — Python enums for job statuses (DeploymentStatus, PositiveSchedulingStatus, NegativeSchedulingStatus). Imported by system_manager and cluster_manager.
• libraries/resource_abstractor_client — Python HTTP client for resource-abstractor. Reads RESOURCE_ABSTRACTOR_URL and RESOURCE_ABSTRACTOR_PORT from env.

These libraries are always built from this repo's local libraries/ folder — never fetched from a remote git repo. Consumers wire them in as follows:

• Docker builds — the service's docker-compose.yml exposes libraries/ as a named build context (additional_contexts: libraries=../libraries); the Dockerfile does COPY --from=libraries . /libraries and pip installs them. CI (docker/build-push-action) passes the same via build-contexts: libraries=./libraries.
• Local dev / tests — pip install ./libraries/oakestra_utils_library ./libraries/resource_abstractor_client (the VS Code install-*-dependencies tasks already do this). The requirements.txt files no longer list the libraries.

Gotchas

• Host networking breaks container DNS. When using override-network-host.yml, containers can't resolve each other by name — set all env vars to IPs, not container names.
• Shared libraries are always local. oakestra_utils_library and resource_abstractor_client are built from this repo's libraries/ folder (see the Shared Libraries section) — there is no LIB_BRANCH build arg any more, and nothing is fetched from a remote git repo. Edit the code in libraries/ and rebuild; changes take effect immediately, no push required. Docker builds need BuildKit/Buildx (named build contexts) — already the default in the compose files, CI, and VS Code tasks.
• eventlet monkey-patching. Both system_manager and cluster_manager use eventlet. Monkey-patching must happen before Flask/pymongo imports — don't reorder the top of entry-point files.
• Scheduler is the same binary for root and cluster — differentiated only by env vars (SCHEDULER_TYPE, Redis URL/password). Keep deployment-specific logic out of the binary.

Design Principles (keep these in mind when contributing)

1. Lightweight first. The whole orchestrator stack targets ~1 GB RAM. Avoid pulling in heavy dependencies or adding services without strong justification.
2. Two-level hierarchy is intentional. Root handles multi-cluster global placement; cluster handles per-worker placement. Keep concerns separated — root doesn't talk to workers directly, cluster doesn't bypass root for scheduling decisions.
3. Pluggable scheduling. The scheduler binary is shared between root and cluster, differing only via env vars. New algorithms should implement the placement.Algorithm interface in scheduler/calculate/schedulers/placement/.
4. Stateless services, stateful DBs. Services (schedulers, managers) are designed to restart cleanly. State lives in MongoDB and Redis. Don't add in-process state that survives restarts without an explicit persistence story.
5. Override-based configuration. Don't bake deployment-specific config into docker-compose.yml. Add a new override-*.yml file instead.
6. The network plugin is external. oakestra-net (root_service_manager, cluster_service_manager) is a separate repository. Changes to network behavior need PRs there, not here.

Updating the Troubleshooting Skill

The troubleshooting skill lives at SKILLS/troubleshoot-oakestra.md. It is used by an AI agent to diagnose a live Oakestra deployment. Keep it in sync with architecture changes.

When to update the skill

How to update the skill

The skill is plain markdown. Each STEP is self-contained. Make targeted edits:

• Keep bash commands copy-pasteable and tested.
• When adding new checks, follow the existing pattern: show the command, explain what to look for, describe what it means.
• Keep the error-pattern table in STEP 2.3 up to date — it is the fastest triage tool.
• The diagnosis report template in STEP 16 should reflect the current set of components.

What the skill does NOT need to track

• Implementation details (function names, internal algorithms) — those belong in code comments.
• Temporary debugging one-liners — only promote something to the skill if it is reusable across deployments.
• Anything already obvious from docker ps output.