Imported from elizaOS/benchmarks (harnesses/eliza/AGENTS.md). Install upstream with npx skills add elizaOS/benchmarks --skill eliza. Copyright stays with the author.

eliza-adapter — Agent Guide

Python bridge that connects benchmark runners (Python) to the elizaOS agent runtime (TypeScript) over HTTP. Not a standalone benchmark — imported as a library by every benchmark that needs to talk to the eliza benchmark server. Not registered in the orchestrator registry; consumers depend on it directly.

Install (one-time)

pip install -e harnesses/eliza/

Or from within this directory:

pip install -e .

Use as a library

from eliza_adapter import ElizaServerManager

mgr = ElizaServerManager()
mgr.start()          # spawns node --import tsx suites/lifeops-bench/runner/src/server.ts
client = mgr.client  # ready-to-use ElizaClient (HTTP to localhost:3939)

resp = client.send_message("hello", context={"benchmark": "agentbench", "task_id": "1"})
print(resp.text, resp.params)

mgr.stop()

Or point ElizaClient at an already-running server:

# Start the TypeScript server manually (in the repo root)
node --import tsx suites/lifeops-bench/runner/src/server.ts

from eliza_adapter import ElizaClient
client = ElizaClient("http://localhost:3939")
client.wait_until_ready()

Smoke / mock mode

run_osworld_mock.py drives a single-turn OSWorld-style mock session using a real server subprocess (requires Node.js and compiled TS dependencies):

python run_osworld_mock.py

To suppress API calls server-side, set ELIZA_BENCH_MOCK=true before starting ElizaServerManager — the manager will blank all provider API keys in the subprocess environment.

Test the harness

pip install -e .
pytest harnesses/eliza/tests/ -v

Tests are pure-Python (no Node.js, no live server) — they monkeypatch HTTP and subprocess calls.

Layout

Notes

• The TypeScript server lives at suites/lifeops-bench/runner/src/server.ts. ElizaServerManager auto-locates it by walking up from __file__.
• Default port is 3939; override with ELIZA_BENCH_PORT.
• BENCHMARK_HARNESS / ELIZA_BENCH_HARNESS routes ElizaClient through Hermes, Smithers, or OpenClaw backends instead of the eliza HTTP server.
• The Vending-Bench bridge forwards the benchmark's bounded transcript and required action schema unchanged to every native harness. Missing usage, reset failures, and empty output fail or score invalid; no strategy hints or fallback business actions are injected.
• Per-turn telemetry writes to BENCHMARK_TELEMETRY_JSONL or $BENCHMARK_RUN_DIR/telemetry.jsonl (auto-fallback to a tmp dir).
• This package is not registered in registry/commands.py; no orchestrator run_command applies — see consumers in agentbench/, context-bench/, mind2web/, and tau-bench/.
• Full architecture: README.md.

⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests

The binding, repo-wide standard is AGENTS.md. Read it. Nothing in this package is done until it is proven done — a reviewer must confirm it works without reading the code, from the artifacts you attach. This applies to every feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof.

• Record AND read model trajectories. Capture the actual inputs and outputs of the model from a live LLM — not the deterministic proxy, not a mock: the prompt, the providers/context, the raw model output, every tool/action call, and the result. Then open the trajectory and review it by hand. A captured-but-unread trajectory is not evidence (packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out>).
• Real, full-featured E2E — no larp. Every feature ships detailed end-to-end tests that drive the real path end to end. Not the happy "front door" only: cover error paths, edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that asserts against a mock/stub/fixture standing in for the thing under test does not count. If the real model/device/chain/connector/account is hard to reach, make it reachable — that is the work, not an excuse to mock. If the existing tests here are shallow or mocked, fixing them is part of your change.
• Screenshots + logs at every phase, plus a complete walkthrough video/run-through of the entire feature or view, start to finish (bun run test:e2e:record).
• Manually review every artifact the change touches — never just the green check: client logs (console + network), server logs ([ClassName] …), the model trajectories in and out, before/after full-page screenshots, and the domain artifacts listed below for this package.
• No residuals. No shortcuts. The goal is not "done" — it is everything done. Clear every blocker by the hard path: build the real architecture, stand up the real model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a "follow-up." When unsure, research thoroughly, weigh the options, and ship the best, highest-effort, production-ready version. Keep going until every possibility is exhausted.

Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in <details>); attach each evidence type or explicitly mark it N/A with a reason — never leave it blank. If develop moved and changed behavior, re-capture evidence; stale proof is worse than none.

Capture & manually review for this package — benchmark / eval suite:

• A real-model run (not the mock/smoke fixture) producing the score-report JSON, with the numbers inspected and the provider/model recorded.
• The per-item trajectories the harness captured, spot-reviewed for correctness — a green harness run over mock fixtures is not a result.
• The provider matrix actually exercised, and the scoring math validated against a known case.
• Failure / timeout / partial-output handling in the harness itself.