Imported from scheeles/keelos (AGENTS.md). Install upstream with npx skills add scheeles/keelos. Copyright stays with the author.

AGENTS.md

Guidance for AI agents contributing to KeelOS.

Project Overview

KeelOS is an immutable, API-driven Linux distribution designed exclusively for hosting Kubernetes workloads. It replaces the traditional userspace (Shell, SSH, Systemd) with a single-binary PID 1 (keel-init) and a gRPC-based management API (keel-agent).

Key Concepts

• PID 1: keel-init — Custom init system (never use systemd terminology)
• API-Driven: All management via gRPC, no SSH/shell access
• Immutable: Read-only SquashFS root filesystem with atomic A/B partition updates
• Minimalist: <100MB total OS footprint; only essential components (kernel, init, agent, containerd, kubelet)
• Security: mTLS everywhere, kernel lockdown, no interpreters

Terminology

Use KeelOS-specific terms consistently:

Repository Structure

Critical Rules

1. No Panics in PID 1: The keel-init binary is PID 1 and must never panic. Always use Result<T, E> with proper error handling.
2. Memory Safety: All system components (PID 1, Agent) are written in Rust. Tooling may use Go.
3. Minimal Dependencies: Audit every crate. Prefer std where possible.
4. Static Linking: All OS image binaries target x86_64-unknown-linux-musl. osctl additionally ships the following targets via cross:
   • aarch64-unknown-linux-musl — Linux ARM64
   • armv7-unknown-linux-musleabihf — Linux ARMv7
   • x86_64-pc-windows-gnu — Windows x86_64
   • x86_64-apple-darwin — macOS Intel
   • aarch64-apple-darwin — macOS Apple Silicon
5. Containerized Builds: All builds must run inside a container for reproducibility:

   ./tools/builder/build.sh

6. Feature Branches: Never commit directly to main. Create feature branches using the naming convention below and submit PRs:
   • feat/<short-description> — New features
   • fix/<short-description> — Bug fixes
   • docs/<short-description> — Documentation-only changes
   • chore/<short-description> — Maintenance (dependency bumps, CI)
7. Tests Required: All new code requires unit tests. System changes require verification scripts in tools/testing/.

Build & Test

# Build the OS image (runs in container)
./tools/builder/build.sh

# Run unit tests
cargo test --workspace

# Run full lint suite (clippy + fmt, same checks as CI)
./tools/lint.sh

# Run in QEMU for testing
./tools/testing/run-qemu.sh

# Run boot tests
./tools/testing/test-boot.sh

# Run integration tests
./tools/testing/test-integration.sh

# Verify build artifacts
./tools/testing/verify-artifacts.sh

Coding Standards

Rust

• Formatter: rustfmt with default settings.
• Lints: Comprehensive linting configured in:
  • clippy.toml — Project-specific clippy configuration
  • Cargo.toml — Workspace-level lint rules ([workspace.lints])
  • Enabled lint groups:
    • clippy::all (deny) — All clippy lints
    • clippy::pedantic (warn) — Opinionated but helpful lints
    • clippy::nursery (warn) — Experimental but useful checks
    • clippy::cargo (warn) — Cargo manifest lints
  • Critical denials for PID 1 safety:
    • unwrap_used — Prevents .unwrap() calls
    • expect_used — Prevents .expect() calls
    • panic — Prevents explicit panic!() macros
    • todo!(), unimplemented!(), and unreachable!() are also prohibited in runtime code — all panic
    • These are allowed in test code only
  • Must pass cargo clippy --workspace -- -D warnings (enforced in CI)
• Unsafe Code: unsafe_code is set to warn. It is permitted only at syscall/FFI boundaries and must include a // SAFETY: comment explaining the invariants that make it correct.
• Error Handling:
  • ❌ unwrap(), expect(), todo!(), unimplemented!(), unreachable!() in runtime code
  • ✅ ? operator or explicit matching
  • ✅ Proper error documentation with /// # Errors
• Running Lints Locally:

  # Check all lints (same as CI)
  cargo clippy --workspace -- -D warnings
  
  # Auto-fix some issues
  cargo clippy --workspace --fix
  
  # Check formatting
  cargo fmt --all -- --check

• Async: Use tokio for the Agent. keel-init should remain synchronous where possible for simplicity, or use a minimal executor if needed.

Commit Messages

Follow Conventional Commits:

• feat: add network interface configuration
• fix(init): correctly reap zombie processes
• docs: update architecture spec
• chore: bump kernel version

Commit Strategy

1. Atomic Changes: Each commit must represent a single, logical unit of work.
2. Linear History: Maintain a linear git history. Rebase your changes on top of main.
3. PR Squashing: Feature branches should be squashed into descriptive, single commits before merging.

Testing Guidelines

1. Unit Tests: Mandatory for all new logic. In Rust, use #[cfg(test)] modules within the same file.
2. Verification Scripts: New features affecting system boot or artifact creation must be covered by scripts in tools/testing/.
3. E2E Tests: Significant system-wide changes (e.g., networking, container lifecycle) require end-to-end tests using the QEMU-based testing framework.
4. No Regressions: All existing tests and verification scripts must pass before merging.

PR Reviews

Every PR description must follow the PULL_REQUEST_TEMPLATE, including:

• A clear description and linked issue(s)
• The type of change checked off
• A release note (or NONE)
• A documentation link (or NONE/TBD)

All pull requests are reviewed by Claude AI to ensure:

• Code quality and lint compliance
• Proper error handling (especially for PID 1 components)
• Test coverage for new features
• Documentation updates when needed
• Compliance with commit message conventions

Claude will provide feedback on the PR and may request changes before approval.

Documentation Strategy

Writing Style

1. Be Concise: KeelOS users are system engineers, not beginners.
2. Use Code Examples: Show actual commands and output.
3. Explain "Why": Document design decisions, not just "what".
4. Security First: Always mention security implications.
5. No Assumptions: Don't assume systemd, traditional Linux tools.

Documentation Structure

All user-facing documentation lives in /docs:

• architecture.md — System design, boot sequence, update mechanism
• getting-started.md — Building from source, running locally
• installation.md — Installing from pre-built images
• using-osctl.md — Remote administration CLI reference

Component-specific documentation lives in component README.md files (e.g., /cmd/keel-init/README.md).

Code-Level Documentation

• Rust: Public items (pub) generally require /// doc comments. Complex internal logic needs // comments explaining why, not what.
• Shell: Functions require a header comment block explaining usage and arguments.

Component Documentation

• Every crate in pkg/ and cmd/, and significant tool in tools/, must have a README.md.
• Include: Overview, Build Instructions, and Usage Examples.

CLI User Experience

• All CLI tools must implement --help.
• Use distinct 'long' vs 'short' help where applicable.

Architecture Diagrams

When updating architecture.md, use Mermaid diagrams for clarity:

graph TD
    A[Kernel] --> B[keel-init PID 1]
    B --> C[keel-agent]
    B --> D[containerd]
    D --> E[kubelet]

When Code Changes Affect Docs

API Documentation Format

For keel-agent gRPC APIs:

### EndpointName

Description of the endpoint.

**Request:**
- `field` (type) - Description

**Response:**
- `field` (type) - Description

**Example (osctl):**
```bash
osctl command --flag value
```

GitHub Interactions

• Always use the GitHub CLI (gh) for all GitHub operations
• Use gh pr, gh run, gh api instead of browser interactions
• Examples:
  • View PR checks: gh pr checks <pr-number>
  • View run logs: gh run view <run-id> --log
  • Check run status: gh run list --branch <branch>
• Only use the browser for creating PRs when the CLI prompt is insufficient

Key References

• Architecture — System design and boot sequence
• Getting Started — Build from source
• CHANGELOG — Release notes

Questions?

If you encounter ambiguity:

1. Check existing documentation for patterns
2. Review similar OS projects (Talos Linux, Flatcar, Bottlerocket)
3. Default to security and simplicity
4. When in doubt, ask for clarification in the PR

Remember: KeelOS is opinionated software. Documentation should reflect that confidence while remaining helpful.