Imported from younissk/LLMSearchBench (
AGENTS.md). Install upstream withnpx skills add younissk/LLMSearchBench. Copyright stays with the author.
Working in this repository
A benchmark is a claim about other people's software. The value of everything here rests on the numbers being reproducible and the sources being credited, so most of the conventions below exist to protect one of those two things.
Layout
Two halves. The Python harness produces numbers; the Docusaurus site publishes
them. They meet at exactly one file shape — summary.json.
src/llmsearchbench/
types/ the records, as Pydantic models
providers/ who serves which model, and at what price
datasets/ source data: spec, registry, download, attribution
scoring/ metrics, aggregation, reproduction diffing
harness/ the run loop and the seams a model plugs into
storage/ artefacts on disk
ui/ the Rich consoles
cli/ the Typer app, one module per command group
tests/ mirrors the layout above
data/ source datasets — fetched, never committed
tasks/ benchmark task sets, one JSONL per release
results/ published run artefacts
attribution/ licences and citations for every source
docs/ the documentation site
src/llmsearchbench/ must stay a package directory. Flattening it would make
every module a top-level import name, and types would shadow stdlib types —
which breaks contextlib, and therefore pathlib, at interpreter start.
Commands
make install # uv sync + npm install
make check # lint, types, tests, and the site's typecheck — the CI gate
make test # pytest alone
make format # ruff format + ruff check --fix
make data # download source datasets (~38 MB), skipping what verifies
make docs # docs dev server with hot reload
make help # everything else
Run make check before calling any change done. It is what CI runs.
Run Python from the repository root, not from inside src/llmsearchbench/ —
from in there, types/ shadows the stdlib module of the same name.
Invariants
These are the things that quietly destroy a benchmark. Each one has a test.
Numbers are fractions, not percentages. accuracy, citationF1, and
hallucinationRate are 0..1. The Pydantic types reject anything outside that
range, because 87.1 renders as 8710% on the site.
Source data is pinned to a commit and a SHA-256. main is not a pin. A
checksum mismatch is a decision to make, not a warning to skip — make data
fails rather than accepting the new bytes.
data/raw/ is read-only and never committed. Fixing a bad record means
writing a derivation into data/processed/, not editing the source. Otherwise
nobody can tell whether a number came from the published dataset or from our
patch of it.
Credit is generated, not maintained. attribution/SOURCES.md,
attribution/citations.bib, and docs/content/data-sources.md are rendered
from the registry in src/llmsearchbench/datasets/registry.py. Edit the
registry and run make attribution; never edit the outputs. Save the licence
text under attribution/licenses/ whenever you add a dataset.
Failing loudly beats substituting quietly. An unknown model id, an unconfigured adapter, and a missing API key all raise with a pointer to the file to edit. A benchmark that silently runs a different model than it reports produces numbers nobody can place.
The judge is part of a release's identity. Changing the judge model is a MAJOR version bump even if the task set is untouched.
A missing measurement is not a zero. A failed request, a reply with no text and no tool call, and a reply no parser could read are all recorded and left out of the score. Counting them as answers put a model that failed all 360 items into mid-table once, and scored 478 silent replies across 17 models as deliberate decisions. A rerun retries them; scoring never guesses at them.
A measure that does not apply is not a zero either. A no_answer item has
nothing to rank and nothing to recall. Those columns are None and the
averages skip them. Filling them with zeros once cost a model 28 points of
precision for getting the items right.
Label provenance travels with the item. label_source says who judged
relevance (human or derived); answer_source says where a gold answer came
from (dataset, annotated, none). Validators refuse an item that carries
an answer without saying where it came from. Never average the two kinds into
one number without saying so.
A model-written label has to be checkable. An annotated answer must be a verbatim span of the passage it came from, and the script verifies that rather than trusting it. Same rule for a judge: a verdict must quote its evidence, and a quote that is not in the cited document is discarded.
Only what may be redistributed is committed. MS MARCO grants
non-commercial research use and extends no licence, so its passages are built
locally and git-ignored while a manifest of ids and checksums is committed.
Check a source's terms before adding its text to tasks/.
Two schemas, one shape
summary.json is written by Python and read by TypeScript, and they disagree on
naming. Run artefacts (raw.jsonl, manifest.json) are snake_case; the files
the site reads are camelCase.
Rather than spell camelCase in Python, the site-facing models in
types/results.py keep snake_case attributes and carry an alias. Pydantic
serialises by alias, so the JSON is unchanged.
tests/types/test_site_contract.py compares those aliases against
docs/src/data/types.ts and fails if the two drift. If you add a field to
ResultRow, add it to the TypeScript too — and note that to_camel turns
cost_per_1k into costPer1K, which is not what the site reads, so that one
field carries an explicit alias.
Versioning
Two things version independently, and conflating them produces numbers nobody can place:
- the benchmark — tasks, judge, harness, and the numbers they produce —
versioned by a release tag, frozen in
results/<version>/anddocs/src/data/releases/; - the documentation prose — versioned by Docusaurus, frozen with
make docs-version VERSION=v0.1.0.
You do both at the same moment, but they are separate commands. The policy is
written out in docs/content/versioning.md.
Conventions
Tests state the behaviour, not the implementation. A test name should say what breaks if it fails. Fakes over mocks: the run loop is driven through a fake Messages API, so no test needs a network or an API key.
Rich and Typer stay in ui/ and cli/. Everything below them is
terminal-free, so the library can be driven from a notebook or another program.
The downloader takes an on_bytes hook rather than knowing about a progress bar.
The search tool is offered, never executed. The task measures whether a model reaches for it, and that decision happens before any result comes back. There is no search backend and no second API to configure. Do not add one without a reason that survives the question "what does running the search tell us that the decision does not?"
Exit codes are a contract, because CI gates on them: 0 success, 1 a real
failure, 2 bad input, 3 not wired up yet.
Committed JSON is machine-written. save_summary is the only thing that
writes a release file; hand-aligning one makes make publish reformat it.
Commit messages
Conventional Commits, with the package as the scope where one applies:
feat(scoring): add citation F1 over gold source sets
fix(datasets): reject a checksum mismatch instead of warning
test(harness): cover an interrupted run keeping earlier records
docs: explain why raw data is never committed
chore(deps): add pydantic, rich, and typer
Explain why in the body when the diff does not make it obvious. Do not mention the tooling used to write the change.
State of play
Kept short on purpose. The full picture — every number, every known-broken
provider, every decision and why — is in docs/content/status.mdx, which
renders as Where it stands on the site. Update that page, not this section.
- Task 1, tool-use correctness: done. 360 items, 36 models, published.
- Task 2, search-result discrimination: built. 331 items, 199 scorable, one model run. Scores the answer and nothing else.
- Task 3, search quality: design only.
- No judge runs yet. The design is written up and tested; the groundedness column is absent rather than zero.
Provider gotchas that will waste an afternoon otherwise: Moonshot rejects any
temperature but 1; Avey serves tools on /llm/responses only and is rate
limited to roughly one item per seven minutes; two of the three free OpenRouter
endpoints return nothing at all.
Running things
npm run buildis the only check that catches broken links — the dev server serves an SPA fallback for every path, so a broken link returns 200 there.- The docs dev server needs
NODE_OPTIONS=--max-old-space-size=8192. - Never
pkill -fon a pattern that matches the command you are running from. It kills the shell, and anything later in the same compound command never happens. This has cost two shells and one half-written script. - Run each model as its own process. A saturated endpoint held up a working one for an hour when they shared a sequential loop.