Imported from WizisCool/BenchVerify (AGENTS.md). Install upstream with npx skills add WizisCool/BenchVerify. Copyright stays with the author.

BenchVerify — Agent Collaboration Guide

Core validator for the Real-time LiveBench Evaluation & LLM Downgrade Detector

1. What This Project Is

BenchVerify detects "downgraded" AI API access: resellers silently substituting the premium models users pay for (e.g. GPT-5.5, Claude Opus) with cheaper ones (e.g. Luna, open-weight small models, distilled models).

The open core (this repo) is a validator microservice: given an API base URL, key, and claimed model, it runs a LiveBench evaluation in real time (quick subset or full suite), scores it with the official pipeline, runs model-spoofing fingerprint probes, and returns a machine-readable verdict (green / yellow / red) with i18n keys.

The closed website (BenchVerify-Web, full-stack TypeScript) consumes this validator over HTTP and owns everything user-facing: orchestration, queue, database, quotas, report pages, OAuth (GitHub / Google / Linux.do), ads, and all bilingual copy.

2. Open-Source Boundary (important — violations must be reworked)

Hard rules:

• This repo must never contain: frontend code, orchestration logic (task queues, databases, quotas), user-facing copy catalogs (i18n). The validator returns machine-readable keys + params; BenchVerify-Web renders all copy with next-intl.
• Self-hosters and integrators use the validator via curl / Swagger (/docs).
• AGPL-3.0 intent: if a reseller turns the validator into a public service, they must open-source their changes, creating a commercial moat.
• The two repos are coupled through the API contract in docs/api.md; contract changes must be backward compatible or upgraded in lockstep.

3. Tech Stack (decided — do not change without approval)

4. Repository Layout

BenchVerify/
├── AGENTS.md                 # This file
├── README.md
├── LICENSE                   # AGPL-3.0
├── docker-compose.yml        # validator (single service)
├── .env.example
├── services/
│   └── validator/            # The open core validator
│       ├── validator/
│       │   ├── main.py           # FastAPI entry (evaluate/jobs/models/estimate)
│       │   ├── core/             # Config (pydantic-settings), logging (key masking)
│       │   ├── data/             # Baseline + question-pack loaders
│       │   ├── evaluation/       # LiveBench runner (official CLI subprocess),
│       │   │                     #   pipeline, in-process job engine
│       │   ├── fingerprint/      # Spoofing probes + aggregation + registry
│       │   ├── verdict.py        # Three-color downgrade rules
│       │   ├── cost.py           # Cost estimation utility
│       │   └── schemas.py        # Pydantic request/response models
│       ├── tests/            # pytest (verdict, fingerprints, pipeline, API)
│       └── pyproject.toml
├── data/
│   ├── baseline/             # Official leaderboard baseline JSON + pricing
│   └── questions/            # Quick-mode fixed question subset
├── scripts/                  # Sync official data, build subsets, validate data
└── docs/                     # Documentation system (index below)

5. Documentation Index (multi-agent entry points)

6. Development Commands

# Local development (validator) — run from the REPOSITORY ROOT so that the
# default data/ paths resolve (data/baseline, data/questions):
cd services/validator
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"        # livebench git dep only for the worker extra:
pip install -e ".[dev,worker]" #   full pipeline needs the official checkout
cd ../..                       # back to the repo root
PYTHONPATH=services/validator .venv-fallback/bin/uvicorn \
    validator.main:app --reload    # API at :8000, Swagger /docs
# (or, from the repo root: PYTHONPATH=services/validator \
#    services/validator/.venv/bin/uvicorn validator.main:app --reload)

# Tests and checks (inside services/validator)
pytest
isort --check-only .
black --check .
ruff check .
mypy validator

# Run the official livebench pipeline requires a checkout; the Dockerfile
# clones it into /opt/livebench. Locally, set LIVEBENCH_REPO_DIR to a clone:
#   git clone --depth 1 https://github.com/LiveBench/livebench.git /tmp/livebench
#   LIVEBENCH_REPO_DIR=/tmp/livebench/livebench <uvicorn command above>

# Full stack one command
docker compose up --build

7. Engineering Conventions

7.1 Language

• All code identifiers, code comments, commit messages, README and all documentation (docs/) are written in English.
• User-facing copy is not in this repo: the validator emits key + params and BenchVerify-Web renders it (next-intl, zh/en).

7.2 Python code style — Google Python Style Guide

• Follow the Google Python Style Guide (https://google.github.io/styleguide/pyguide.html):
  • 4-space indentation; double quotes for strings.
  • Maximum line length: 80 characters (checked by black + ruff).
  • Naming: module_name, ClassName (CapWords), function_name, variable_name, CONSTANT_NAME (UPPER_SNAKE), protected _private, dunder __special__ only for operators.
  • Type annotations required on all public functions (PEP 484/526); mypy strict must pass.
  • Docstrings: Google style with Args: / Returns: / Raises: sections where applicable; every public module, class and function gets one.
• Import sorting: isort (profile black, line_length=80). Imports are grouped: stdlib → third-party → first-party, sorted alphabetically within each group.
• Formatting: black with line_length=80; ruff for linting.
• Run before every commit: isort . → black . → ruff check . → mypy validator.

7.3 API key security (red line)

api_key exists only in the job's process memory. It is never persisted, never logged, never part of error payloads. Logs always show the sk-*** mask. Any plaintext key in a logger, error response, or job record is a critical bug.

7.4 Types and tests

• Full type annotations + pydantic validation everywhere; mypy strict.
• The verdict and fingerprint library must have unit tests; API tests use httpx + TestClient. Never mock the official livebench scoring functions (they are the golden standard) — only mock the user API's HTTP calls and the runner wrapper's subprocess I/O.

7.5 Progress reporting

Every long-running evaluation updates the job record and exposes progress via GET /jobs/{id} (see docs/workflow.md); BenchVerify-Web polls it and owns its own SSE for the frontend.

7.6 API contract changes

Change docs/api.md before changing code — the doc is the contract.

8. Working Tips for Agents

• When delegating subtasks, name the doc files and file-path ranges to read; avoid full-repo scans.
• Changes to evaluation caliber (scoring, question sets, thresholds) must be reviewed against docs/benchmark.md and the baseline data.
• New fingerprint probes: read the probe interface and registry in docs/fingerprint.md.
• Never create frontend directories or orchestration code (apps/, pages/, components/, DB models, task queues) in this repo.
• Do not add scoring dependencies beyond official livebench; check the official package first (it ships download_questions.py / download_leaderboard.py that can be reused).
• The official livebench repo evolves its run_livebench.py CLI; the wrapper layer (validator/evaluation/runner.py) adapts to its invocation and output parsing. Unit tests pin the official output format via fixtures.