Imported from chenmingtang830/ax-eval (SKILL.md). Install upstream with npx skills add chenmingtang830/ax-eval. Copyright stays with the author.

ax-eval — host-agent skill

You are the agent harness. The eval is a reviewed, frozen TargetPack; you run it against the live product surface and the CLI verifies success via programmatic outcome verification (API readback), not self-report.

Two things make this real:

• Discovery is Phase 0. You are NOT told the endpoint, base URL, request shape, or docs link. You must web-search to discover the API first, then do every task with what you found. Record your search funnel honestly — it's scored.
• Standard effort. New live evaluations use medium effort. Historical low/high artifacts remain readable, but they are not the current benchmark contract.

Prerequisites

Pick a target. The repo ships example packs under targets/examples/, or you can generate one for any SaaS via ingest → generate → review (see workflow below). Then:

npm install
npm run ax-eval -- init      --pack <pack.yaml>   # print the .env stub the pack needs
# Fill in .env with the credentials + sandbox ids the stub asks for, then:
npm run ax-eval -- check-env --pack <pack.yaml>   # verify env is set
# Testing non-API surfaces too? Add --surface all to see/stub each surface's auth:
npm run ax-eval -- init      --pack <pack.yaml> --surface all
npm run ax-eval -- check-env --pack <pack.yaml> --surface all

When you are working from a cloned ax-eval repository, use npm run ax-eval -- <command> for every command in this skill. That runs the local checkout, including unpublished changes. A globally installed ax-eval binary is fine for released workflows, but it will not see local source edits until the package is rebuilt and linked/installed.

Each pack declares its own auth (which env var holds the credential) and sandbox_scope (the isolation level — workspace, project, board, etc. — that the developer must provision). init and check-env read the declarations verbatim, so you never need to know a target's specifics in advance.

Per-surface auth. Each surface (api/cli/sdk/mcp) authenticates independently, declared in surfaces.<s>.auth: inherit (reuse the API token — SDKs/CLIs), token (its own key — Monday/Linear MCP), or oauth_app (registered OAuth app + refresh token — hosted MCP surfaces such as Asana). You don't pre-configure everything: gating is lazy and per-surface. When exec-plan --surface all hits a surface it can't authenticate, it skips the prompt, writes a blocked cube cell (run-<surface>-blocked.normalized.json, blocked: requires-oauth | missing-credential), and prints the exact env keys to add. When the OAuth env vars are present and the harness supports provisioning, ax-eval exchanges the refresh token at invoke time and passes a short-lived bearer token through an isolated harness config/env. The api surface (and any token surface you have creds for) still runs; the blocked surface shows as a distinct cell in the competitive report, never a 0%. To unblock, add the keys it names to .env (init --surface all stubs them) and re-run.

Workflow

AXArena / AXArena-Database v1 canonical benchmark path

AXArena-Database v1 is different from ordinary per-target authoring. It starts from one canonical suite, then compiles vendor adapters from public vendor cards and vendor-specific verification extracts:

evaluation suite -> vendor verification extraction -> TargetPack -> execution -> verification -> normalized records -> leaderboard

Use ax-arena/benchmark/axarena-database/v1/suite.yaml as the source of truth. The files under ax-arena/benchmark/axarena-database/v1/packs/<vendor>/pack.yaml are compiled execution artifacts, not separate benchmark definitions. They should keep the same task ids, titles, intents, difficulty labels, scoring contract, surfaces, and harness matrix; only auth, base URL, outcome-verifier checks, N/A mapping, and surface configuration vary by vendor.

Run AXArena-Database authoring commands as npm run ax-arena -- benchmark <command>. The old ax-eval authoring aliases are temporary one-minor compatibility launchers and emit a deprecation warning.

Current status: mutable AXArena-Database v1 authoring freeze is done for the 6-vendor core cohort (Neon, CockroachDB, Turso, Supabase, Insforge, Nile) — packs are approved and trace review is completed. Production 3-trial and publication freeze are deferred; do not run them as the default next step. Research-lane tasks stay out of the scored denominator. Use ax-arena/benchmark/axarena-database/v1/vendor-selection-ledger.yaml for core vs research vs excluded.

When production is unblocked and all vendor runs have been verified, freeze the publication bundle (core cohort only):

npm run ax-arena -- benchmark publication-bundle \
  --run-root results/runs/axarena-database-v1-production \
  --out results/runs/axarena-database-v1-production/publication-bundle \
  --benchmark-root ax-arena/benchmark/axarena-database

The bundle manifest is the handoff to the AXArena static website and the launch report. The command accepts only an attested pinned-oci + hosted-trusted production rerun and verifies the detached GitHub OIDC attestation; local, low-pass, missing-artifact, and missing-attestation inputs are blockers. It requires externally approved AX_ARENA_APPROVED_SIGNER_SHA rather than the signer's self-described SHA. It also reruns canonical aggregation/reporting from attested cells, while export and competitive readers re-verify the preserved detached bundle, signed source assets, and exact bundle inventory. For official aggregation, use the signed batch-completion.json completed_at value as the reporting --generated-at; publication rejects a free timestamp.

1. Generate the frozen task set (or use a committed example)

Already have an example pack at targets/examples/<name>/pack.yaml? Skip to step 2. To build one for a new target:

npm run ax-eval -- ingest   --openapi <spec-url>            # or --graphql <endpoint>
npm run ax-eval -- generate --from results/<name>-ingest.json
#    → writes results/<name>.generated.pack.yaml (auto-derives product, auth, sandbox scope)

Default generation is LLM-assisted: ax-eval builds a rule-derived seed from the spec, then asks a local generator harness (codex or claude-code) to improve it. Product presets can add authoring hints and surface-shaping rules, but code still validates the generated pack and will trigger one repair pass before giving up. Pass --deterministic for keyless/offline fixtures. Either way, the output is only a draft until the review gate approves it.

Generation is surface-aware at task-selection time too: if a declared CLI/SDK/MCP surface only covers part of the product, ax-eval narrows that surface to the tasks it can actually support instead of assuming it mirrors the full API.

For an automated report pass, use:

npm run ax-eval -- automate-report --company <name> \
  --openapi <spec-url> \
  --surface all --harness codex

automate-report never uses Exa. It prefers explicit official --openapi, --graphql, --site, or --docs inputs; if only a company name is provided it asks the configured local harness to find official candidates, then validates them with direct fetch/crawl before ingesting anything. It always runs an API low-effort smoke gate before the fuller requested report, but generated packs still stop at the review gate until a human approves them with ax-eval review.

This produces an L1–L4 ladder (L1 single create · L2 composed chain · L3 ambiguous goal-level comprehension · L4 state mutation) with goal-level prompts (no endpoints) + a discovery spec. Drop-a-link: a new SaaS is a new pack, not a code change.

2. Review gate — a human approves the generated set (required)

npm run ax-eval -- review --pack <pack.yaml>            # read the set
npm run ax-eval -- review --pack <pack.yaml> --approve --by <name>

Generated tasks/outcome verifiers are executable intent that will run write-ops against a sandbox, so nothing runs un-reviewed. The summary lists every task + prompt + outcome verifier (flagged by confidence tier: T1 round-trip = strong, T2 existence/2xx = weak) and the credential/sandbox surface it will touch. Approval is content-addressed — it records a hash of the reviewable fields, so any later edit re-closes the gate and forces re-approval (no AI-approves-AI). The committed example packs ship pre-approved (*.approval.json); exec-plan refuses an un-reviewed/changed pack unless you pass --skip-review.

Controller-owned one-cell execution

When a controller already owns the product/surface/harness/model/effort/trial matrix, use the stable one-cell process contract instead of exec-plan fan-out:

ax-eval cell run --input cell.json --output record.json

The ax.evaluation-cell/v1 input must carry an approved pack path and the full SHA-256 of that exact file, explicit evaluation-set and batch identity, model, effort, trial, full source commit SHA, required credential names, and runtime paths/timeouts. Supply credential values only through the process environment; never serialize them in the cell or record. The runner invokes exactly one harness and surface and returns one ax.normalized-cell-record/v1 record with task/oracle results. It never derives a batch id, chooses benchmark defaults, aggregates trials, publishes rankings, or runs cleanup. Verify/live read-back completes before the record is returned; cleanup is a separate explicit controller step.

Library controllers may pass verificationCredentials separately from the cell-allowlisted host credentials. Verifier values are available to health checks, target adapters, and read-back oracles but never enter the harness child. Use the returned execution_namespace for post-persistence cleanup; do not reconstruct a namespace from display fields.

Library controllers may attach one immutable RuntimeExtensionRegistry with versioned oracle, health-check, provisioning, reset, and target-adapter providers. Health checks run before provisioning; provider failures are redacted into the cell result. Persist the returned record before invoking a reset provider. runCell never calls reset itself, and reset-provider identity belongs with later cleanup evidence rather than the verified cell record. Supply controller-only read-back secrets through verificationCredentials; they reach health checks and oracle providers but never the harness child. Provisioning providers receive no verifier-only credentials, cannot replace PATH or existing environment keys, and every environment value they add is redacted as secret material. They may request additive pathEntries; core canonicalizes and prepends only real directories that resolve outside the writable cell workspace and artifact tree. Arena Postgres cleanup is non-cascading and includes exact namespace-derived roles. Turso CLI cells require a non-writable install path plus exact version and SHA-256 pins selected by the arena controller from the committed ax-arena/benchmark/trusted-runtime/runtime-lock.json; runtime identity has no environment override.

3. Emit the medium-effort prompt

npm run ax-eval -- exec-plan --pack <pack.yaml> --run-dir results/runs/<id>

Writes results/runs/<id>/prompt-medium-a<N>.txt for each attempt (default 1 attempt; --attempts N for pass@k), each a two-phase prompt (Phase 0 discovery → Phase 1 tasks) with a unique namespace per attempt.

4. Run each prompt (as the host agent / sub-agents)

For the medium-effort prompt, follow it exactly:

• Phase 0: actually web-search the target's official docs; find the base URL, auth scheme, request/response envelope, and how to create resources. Record every search query and URL you open.
• Phase 1: perform each task against the live sandbox using only what you discovered. Create resources with the exact namespaced names given.
• Write run-<profile>-a<N>.json (discovery funnel + per-task ids) and run-<profile>-a<N>.trace.json (every API call) to the paths in the prompt. Edit no other files.

Honor the standard medium effort: investigate prerequisites, recover from errors, and verify read-backs without changing the declared task scope.

Between repeated attempts on the same profile, run ax-eval reset --pack <pack> --ns <completed-run-namespace> only when the user explicitly asks you to prepare the next attempt. Do not clean up by default: verify reads live product state, so deleting resources before the report is rendered corrupts scores. This legacy helper covers the retained generic HTTP reset compatibility path; database cleanup must run through the arena cell lifecycle with an explicit ResetProvider.

5. Verify + report

npm run ax-eval -- verify-generated --pack <pack.yaml> \
  --results results/runs/<id>/run-*.json \
  --min-pass-rate 0.8 \
  --html results/runs/<id>/generated-eval.html

When exec-plan selected one task with --task <id>, retain the same flag on verify-generated; the generated follow-up command does this automatically. Otherwise the verifier intentionally treats the full pack as the denominator.

The CLI GETs every resource back, scores outcome verifiers + each profile's discovery funnel, gates on --min-pass-rate, and writes a self-contained HTML report. Then summarize for the user:

• Static discovery score (docs-site crawl) and agent discovery score (usability-suite Phase 0) as separate signals.
• Pass rate by config/profile, difficulty (L1–L4), and pass@k across attempts.
• If --min-pass-rate was used, call out both the overall gate and any per-surface subgate failures.
• Discovery scorecard per config/profile (reached source / canonical action / hops / misled / auth), using surface-relative wording for API vs MCP/SDK/CLI.
• Top recommendations, especially any MCP tool coverage gaps, as Target / Evidence / Fix rather than a raw failure dump.
• Keep MCP tool coverage separate from harness/approval failures: missing project-brief/archive/portfolio capabilities are product/tool gaps, while user cancelled MCP tool call on an existing update tool is a harness interaction signal.
• Attribution: separate genuine product/docs gaps from plan-limited (402, free tier) and discovery-blocked failures. The headline gap is high static discovery with low usability-suite success, and only applies directly on surfaces where the docs site is the agent's discovery path.
• Label a first live matrix as a directional draft, not a final benchmark, unless it has repeated attempts (--attempts N) and the product owner has sanity-checked attribution. Share the HTML together with result JSON, trace, transcript/stdout/stderr, and a manifest/artifact bundle.

Cross-harness / cross-surface (optional, CLI-driven)

The steps above make you the harness. To compare harnesses instead, let the CLI drive them as subprocesses: run one lane per harness so each receives a compatible model slug, for example exec-plan --invoke --harness claude-code --surface all --profile medium --effort medium --model sonnet --invoke-retries 0, then a separate Codex lane with --harness codex --profile medium --effort medium --model <gpt-model> --invoke-retries 0. An OpenCode lane uses --harness opencode --surface all --profile medium --model <provider/model> --invoke-retries 0 for every surface the pack supports, including MCP. OpenCode 1.18.3+ is required. Claude Code and Codex retain their native reported-model handling. OpenCode instead records the requested provider/model route, not an attestation of the model actually served, and passes the selected low/medium/high effort through its provider-specific --variant flag so the recorded execution identity was actually invoked. Pi has the equivalent API/CLI/SDK-only lane: --harness pi --surface api --profile medium --model <provider/model> --effort medium. Pi requires an explicit provider/model and fails closed for MCP. It runs in JSON mode without a session, extensions, skills, prompt templates, themes, or discovered context files; its isolated PI_CODING_AGENT_DIR and session directory are controller-owned. The CLI writes one normalized {surface, product, harness} record per cell. verify then renders them as a single neutral matrix (surface · harness · effort) — no cell is crowned "best". Codex needs its sandbox network opened and an OpenAI-strict output schema; the adapter handles both.

Harness stdout is decoded before AX surface semantics are applied. Keep native wire-shape handling in the harness decoder and scoring meaning in the shared transcript layer. When integrating a new harness, pass its known id instead of depending on auto-detection, preserve result bodies only in already-redacted raw artifacts, and inspect the content-free decoder diagnostics (decoderVersion, parsed/recognized/emitted/malformed counts). A transcript with zero recognized events is not objective evidence and must retain the labeled self-report fallback. The same normalized events drive tool_call_count. Retry metadata records a bounded outcome_reason per attempt, so an exit-0 process that omitted results is distinguishable from a timeout or nonzero exit without persisting raw error text. API cells that objectively invoke a vendor CLI fail the surface-honesty gate even if they also make successful HTTP calls.

For publication-grade lanes, prefer native binaries through AX_EVAL_CLAUDE_BIN and AX_EVAL_CODEX_BIN when PATH wrappers inject corporate/local defaults; tool-track OpenCode lanes have the equivalent AX_EVAL_OPENCODE_BIN override. Pi lanes have AX_EVAL_PI_BIN. API, CLI, and SDK Codex cells are invoked with an isolated Codex home plus mcp_servers={} so unrelated global MCP auth failures do not become benchmark failures. OpenCode requires an explicit provider/model and runs with --pure and --auto from a disposable cwd outside the checkout, with fresh config, data, cache, and state roots, with autoupdate, LSP downloads, and Claude-compatibility loading disabled. It never copies ambient auth.json; provider credentials must be explicitly scoped into the child environment. The per-run home is deleted after artifact recovery so the binary session database is not retained; pack env names cannot replace OpenCode/XDG isolation controls. Managed system/MDM config blocks the lane, and task subagents are denied because their actions are absent from root JSONL. Legacy exec-plan also uses a disposable secret-free cwd and exact credential-value artifact redaction; this is not an OS sandbox. Its self-reported dollar cost is not trusted, so runtime cost_usd remains null. Publication may add a separate API list-price estimate from the committed dated pricing snapshot and normalized token usage; never copy that estimate into cost_usd or use cost per success for pass/fail or ranking. MCP cells for every harness receive only their explicit pack-declared server. OpenCode provisions local stdio and remote HTTP servers in its isolated config; bearer tokens remain in the child environment, and OAuth-app credentials are exchanged headlessly from a refresh token. Interactive browser OAuth is not supported. Adding this generic core runner does not make it a canonical AXArena production harness. This is one product across harnesses/surfaces — competitive is reserved for cross-product comparison.

The OpenCode runner captures native JSONL and maps its tool calls through the shared transcript-decoder seam before AX surface semantics are applied.

Render that cross-product view with npm run ax-arena -- benchmark competitive --from <sealed-publication-bundle> --html <ignored-output.html>. The verified bundle fixes the expected structural-N/A matrix, models, exact harness pins, effort, three-trial sources, and completed-cell evidence; mixed, incomplete, or resealed-score inputs fail closed. Create the ignored output directory first. The deprecated ax-eval competitive name delegates to that arena command during the private-workspace compatibility period.

When consolidating a report for review, put every cell's artifacts in one run directory before rendering the HTML. Keep result JSON, trace JSON, transcript, stdout/stderr, invoke metadata, and a small manifest together so reviewers can deep-dive without hunting through prior scratch runs.

AXArena-Database v1 production lane

AXArena-Database v1 has a dedicated production rerun command for the benchmark-of-record matrix. It is deferred until after team review of the approved packs; authoring freeze (approvals + completed trace review) is already done for the 6-vendor core cohort.

ax-arena benchmark axarena-database-production-rerun \
  --suite ax-arena/benchmark/axarena-database/v1/suite.yaml

Direct arena runtime execution remains intentionally fail-closed. The protected workflow is the only entrypoint that supplies and attests the OS sandbox. In a source checkout, ax-eval axarena-database-low-pass and ax-eval axarena-database-production-rerun delegate to that fail-closed arena CLI. The former daeb-low-pass and daeb-production-rerun spellings remain deprecated compatibility aliases; none of them execute the core runtime implementation. The npm release gate keeps those aliases unpublished until the arena package is public, so their one-minor compatibility clock has not started.

For hosted execution, dispatch Trusted sandbox arena benchmark only. Select a full source SHA and a committed whole-benchmark configuration under ax-arena/benchmark/axarena-database/, then choose the GitHub-hosted pool unless the approved self-hosted runner group is required. Harness versions, the OCI digest, Bubblewrap, models, surfaces, trials, and the vendor roster come from committed locks and configuration, not dispatch inputs. The credential-free planner rejects SDK/MCP cells and more than 256 hosted cells before emitting a matrix.

Before dispatch, create each protected environment named trusted-sandbox-<vendor>-<surface>-<harness>-trial-<n> with required reviewers. Its sole workflow secret is AX_ARENA_CELL_CREDENTIALS_JSON, a JSON object whose keys exactly equal that planned cell's host, verification, reset, and sandbox scope credential names. Do not bind those credentials individually. The credential-free plan receives no secret. Before each protected step, the fresh matrix runner re-verifies and extracts the digest-pinned OCI image to a root-owned sysroot, builds from exact dependency locks, and verifies its runtime manifest. It then executes exactly one cell with the locked Node/tools and Bubblewrap, never native fallback. A separate credential-free job assembles the exact result and byte-identical runtime-manifest set; an isolated job OIDC-signs the detached subject. Only controller-owned batch, normalized-record, cleanup, cell-result, runtime-manifest, and declared evidence files are uploaded. Keyless PR workflows never receive these secrets.

This lane is intentionally scoped to api and cli: Codex runs gpt-5.6-terra, Claude Code runs claude-sonnet-5, both at high effort, with exactly three clean trials per supported vendor/surface/harness cell. It writes trial-1/2/3 directories plus an aggregate/ directory whose normalized record reports the three-trial mean and range. SDK and MCP should not be mixed into the AXArena-Database v1 leaderboard denominator; keep those runs as research evidence unless a later suite revision says otherwise.

The command recomposes a run-scoped pack only when it exactly matches the committed human-approved pack, stages that approval, and runs through the normal review gate. Each trial writes cleanup.json after verification. If invocation or reset fails and cleanup cannot be confirmed, the workflow stops before the next trial. Inspect the preserved artifacts and perform an explicit namespace reset before retrying; do not delete evidence before verification.

Normalized aggregates are public ax.normalized-result/v2 artifacts (schemas/normalized-result.v2.json). Their identity is product × surface × standard-set version × harness × model × effort, and aggregation fails closed when trial records cross an identity boundary. records-diff preserves that boundary and fails if a baseline identity disappears. They retain native harness version, run-batch identity, successful-attempt latency, retry-inclusive duration and consumption, and exact pass³ numerator/denominator. Provider-native cache-write usage is normalized; pricing that requires it remains unavailable when the harness omits that telemetry. The unchanged v1 schema remains readable for migration. Codex native dollar cost is null; do not synthesize a price table outside the sealed arena pricing workflow.

Before human publication freeze, regenerate into the same AXArena-Database v1 contract. Do not bump the suite version for authoring iterations; git SHAs and content hashes identify exact drafts, and any content change invalidates prior pack approvals. Use ax-arena/benchmark/axarena-database/v1/vendor-selection-ledger.yaml as the core cohort source; research/excluded vendors must not silently enter synthesis or production runs. When reviewing coverage, distinguish the broad 75% concept-selection bar from task applicability: only support-matrix cells whose ranked capability bundle satisfies every task requirement on that surface may be executed or scored. Do not publication-freeze while suite.trace-review.yaml is pending (it is already completed for the current authoring freeze).

After freezing and integrity-sealing a publication bundle, export website data with:

npm run ax-arena -- benchmark export-publication \
  --from results/runs/axarena-database-v1-production/publication-bundle-final \
  --out results/runs/axarena-database-v1-production/axarena-export

The seal must include canonical batch.json and batch-completion.json, every completed record/cleanup/artifact, every normalized source record, and every nested snapshot evidence path. Export rejects aggregates whose scores cannot be recomputed from the three completed trials.

Website handoff uses axarena-database as the stable machine id and AXArena-Database as the display name. The version stays in suite_version; do not expose frozen daeb-1-v1 provenance as the public benchmark name.

This keeps the repo boundary clean: ax-eval owns generic single-product execution and verification, while ax-arena owns benchmark aggregation, publication exports, and the eventual curated website handoff. The deprecated ax-eval publication-bundle, ax-eval export-publication, and ax-eval competitive names are shell-free launchers for their arena commands during the compatibility period, which begins only when the arena package is available to users.

Rules

• Only mutate the sandbox scope the pack declares. Never touch production data.
• Discovery is real: web-search, don't paste a known endpoint from memory. Never inject an endpoint the prompt didn't give you.
• Never run an un-reviewed pack: get human review --approve first (or --skip-review only for a committed/trusted pack). A changed pack must be re-approved.
• Do not skip verify — success requires verifier PASS against live state.
• Report harness: host-agent, the host model, and medium effort. The sonnet/gpt5 cross-model profiles only produce real cross-model data in Cursor Composer (where the Task tool spawns alternative-model sub-agents); a plain CLI host should use medium effort or pin a model with --model.

References

• Example packs: targets/examples/*/pack.yaml
• Every command + flag: src/cli.ts
• Expected behavior, documented: the tests/ suite (keyless/offline)
• Report rendering: src/generate/report.ts