agents

Imported from autonomy-cloud/agents (AGENTS.md). Install upstream with npx skills add autonomy-cloud/agents. Copyright stays with the author.

AGENTS.md

Scope

This repository is an air-gapped realtime agent runtime. It retains only:

• the Python agent framework core;
• an OpenAI-compatible adapter for explicitly configured internal endpoints;
• local Silero VAD endpointing;
• absorbed WebRTC server, Python RTC, protocol, Rust FFI, and libyuv sources;
• the OpenAgents runtime supervisor and internal credential manager.

Do not add public-cloud defaults, automatic downloads, provider catalogs, or runtime package installation.

Environment

• Python 3.12+
• uv
• Go 1.26+
• Rust/Cargo matching the pinned toolchain
• the vendored libwebrtc archive, extracted and passed through LK_CUSTOM_WEBRTC

Commands

uv sync
uv run pytest --unit
uv run ruff check runtime/python openagents-core openagents-plugins
uv run mypy runtime/python/openagents

cd runtime/components/openagents-server
GOPROXY=off GOSUMDB=off go build -mod=vendor -o build/openagents-server ./cmd/server

cd ../openagents-python-sdks/openagents-rtc/rust-sdks
cargo metadata --offline --locked --no-deps --format-version 1

The RTC wheel build must use the absorbed Rust source and must not invoke download_ffi.py. The patched build fails if a locally built FFI library is not present.

Configuration

Model traffic must use OPENAGENTS_OPENAI_BASE_URL. Public OpenAI is explicitly rejected. Internal transport credentials are generated by Runtime and must not become user-facing configuration.

Verification

• Unit tests must require no credentials or network.
• Release verification runs with outbound traffic denied.
• Go builds use -mod=vendor, GOPROXY=off, and GOSUMDB=off.
• Rust builds use cargo --offline and an absorbed libwebrtc tree.
• Python installation uses the checked-in wheelhouse and --no-index.
• Every release includes hashes, an SBOM, and third-party license notices.

Local dev stack: server + coworker + workstation + console + teams-bridge

Five first-party components under runtime/components/ build a complete, runnable "digital coworker" stack on top of the air-gapped core above (see runtime/COMPONENTS.md for exact upstream provenance of each):

Unlike the core runtime above, these four have real, ordinary network dependencies at build time (Docker's apt-get, uv/pnpm package installs, go build -mod=mod for openagents-server — see the note below) — they are the demo/dev-facing product surface, not the air-gapped release artifact.

One command

export OPENAGENTS_OPENAI_BASE_URL=<your internal OpenAI-compatible endpoint>  # optional, see below
./scripts/dev-up.sh

This builds (first run only) and starts openagents-server natively in --dev mode (ws://127.0.0.1:7880, devkey/secret), starts openagents-coworker natively via uv run if OPENAGENTS_OPENAI_BASE_URL is set (skipped with a clear message otherwise — job dispatch/registration works fine without it; only actual STT/LLM/TTS calls need a real endpoint), and brings up openagents-workstation via docker compose. Ctrl+C stops everything it started.

Then, separately (kept out of dev-up.sh so it stays fast to iterate on):

cd runtime/components/openagents-console && pnpm install && pnpm dev

Defaults to the same ws://127.0.0.1:7880 / devkey / secret the script brings up — open it, hit Connect, and (with OPENAGENTS_OPENAI_BASE_URL set) the coworker joins automatically via LiveKit job dispatch. Once connected, speaking into your mic both publishes real audio and — via the console's browser-side speech recognition — sends each finalized phrase as a text message, which reaches the agent through openagents-core's existing room_io text-input handler (topic lk.chat → session.generate_reply()) with no STT model required; it shows up in the Chat panel like any other message.

Inviting the coworker to a Microsoft Teams meeting

Opt-in, off by default (heavy image: Chrome + Xvfb + gstreamer + its own Postgres/Redis):

ENABLE_TEAMS_BRIDGE=1 ./scripts/dev-up.sh

This generates .env.teams-bridge at the repo root (once, gitignored — a Django secret key, a Fernet credentials-encryption key, and a service API token) and starts openagents-teams-bridge's three services (a dedicated Postgres, the API app on 127.0.0.1:7000, and a Celery worker that runs the actual Selenium/Chrome bot). Copy TEAMS_BRIDGE_API_TOKEN out of that file into openagents-console/.env.local too (see .env.example) if you're running the console as a separate pnpm dev process, since it needs the token to call the bridge's internal API server-side.

Once connected in the console with an active session, the Settings panel's "Teams meeting" field accepts a teams.microsoft.com/teams.live.com meeting link and bridges the currently-connected agent's LiveKit audio track into that meeting as the bot's "microphone" (and the meeting's incoming audio back into the LiveKit room) — see openagents-teams-bridge/ABSORBED.md for how the audio bridging actually works, and its licensing caveat.

VS Code extensions: coworker-meet and copilot-llm-bridge

Two independent VS Code extensions live under runtime/components/, alongside everything else:

Each is a standalone npm package (own package.json, own build), not wired into dev-up.sh. See each package's own README.md for full architecture; the proposed-API stability caveat mentioned there (Insiders, or --enable-proposed-api autonomy-cloud.coworker-meet, needed for coworker-meet's chat-session registration specifically) still applies below.

Option A: F5 dev host (fastest iteration loop)

cd runtime/components/coworker-meet   # or copilot-llm-bridge
npm install
npm run watch   # keeps rebuilding on save

Then open that folder in VS Code and press F5 to launch an Extension Development Host with it loaded. Reload that window (Developer: Reload Window) to pick up rebuilds instead of restarting F5 each time.

Option B: real install, end to end (package → install → use)

This is what actually ends up in your normal VS Code, not a throwaway dev host window:

cd runtime/components/coworker-meet   # repeat the same for copilot-llm-bridge
npm install
npm run package:vsix        # tsc + esbuild --production, then vsce package
                             # -> dist/coworker-meet.vsix (~170KB; only
                             #    package.json/dist/media get packaged, see
                             #    .vscodeignore -- everything else is a
                             #    build input, not shipped)
code --install-extension dist/coworker-meet.vsix

Repeat for copilot-llm-bridge (dist/copilot-llm-bridge.vsix). Reinstalling after a change is the same two commands again — code --install-extension overwrites the previous version in place, no uninstall step needed.

Verify both actually installed:

code --list-extensions | grep autonomy-cloud
# autonomy-cloud.coworker-meet
# autonomy-cloud.copilot-llm-bridge

Then, to actually use them:

• copilot-llm-bridge: open the "Copilot LLM Bridge" activity-bar panel, click Start (first run triggers Copilot's own consent prompt), then Copy Base URL and point OPENAGENTS_OPENAI_BASE_URL at it (default http://127.0.0.1:4319/v1) -- see "Optional: giving Anika a Copilot-backed LLM" above.
• coworker-meet: run Coworker Meet: Join Meeting from the Command Palette to open the meeting webview panel directly, or Coworker Meet: Open Coworker Window for the chat-session version (Chat view / Agents Window). The chat-session registration specifically needs the proposed-API flag: quit VS Code fully and relaunch with code --enable-proposed-api autonomy-cloud.coworker-meet (a flag passed to an already-running instance's CLI is ignored -- it only takes effect at process launch) -- or use VS Code Insiders, which doesn't require the flag at all. Without either, coworker-meet still works fine for Join Meeting (the plain webview panel); only the chat session won't register.

Known environment gotchas (already worked around in dev-up.sh, documented here so they're not mistaken for new bugs)

• This checkout's path contains : (.../github.com:autonomy-cloud/), which breaks uv outright (path segment contains separator ':') — see memory/colon-path-npx-workaround.md. dev-up.sh runs the coworker's uv commands from an rsync'd colon-free copy instead of the repo path directly.
• openagents-server's own vendor/ tree is currently missing several transitive Go packages (a pre-existing gap), so -mod=vendor fails for it specifically; dev-up.sh builds it with -mod=mod (network-resolved) instead. openagents-workstation's vendor tree is complete and still builds with -mod=vendor normally.
• openagents-workstation and openagents-coworker must use distinct default identities (anika-coworker-desktop vs anika-coworker) — LiveKit disconnects the earlier connection when a second one joins under the same participant identity, so sharing one would make them evict each other on a loop.
• A macOS wheel-tag mismatch can break uv's build of openagents-rtc's editable wheel on newer macOS versions; dev-up.sh sets MACOSX_DEPLOYMENT_TARGET=11.0 around that build as a workaround.
• Real end-to-end WebRTC media (not just signaling) has been blocked in some sandboxed/VPN-heavy environments during development (ICE never completes) — this is host/network-specific, not a bug in any component here. Signaling, worker registration/job-dispatch, and room state are all independently verifiable via openagents-server's dev-mode /debug/rooms and /debug/agents endpoints even when media itself is blocked.