workspace-bench

Imported from camilablank/workspace-bench (AGENTS.md). Install upstream with npx skills add camilablank/workspace-bench. Copyright stays with the author.

Working in this repo

This repo holds the banks, the judges and the baselines of WorkspaceBench. It does not hold inference code: producing readouts (capturing activations, running a lens) happens elsewhere and arrives here as a readouts file. CLAUDE.md has the code conventions and how to add a family; this file is about using the benchmark on a model other than the one it was built for, and about the checks to run before you trust a number.

What the banks assume

Every bank was written and gated against Qwen3.6-27B. A gate means the model actually does the task: greedy-correct AND correct in at least 8 samples of 10 (chain and brew were gated at 10 of 10), or copies the carrier sentence without mentioning the held concept, or names the user attribute when asked outright. Each bank header records its own rule. If the model under test fails a bank's gate, its readouts on that bank cannot be interpreted: a lens that says nothing about an intermediate the model never computed is not wrong.

Run the gate yourself before porting:

wsbench capable model=<openrouter-model> families=poetry,moral_rationale draws=10
wsbench capable model=<openrouter-model>                 # all 21 families that have a question

It asks each bank's own question and grades the answers with the repo judge. An item passes when the greedy answer is right AND the sampled rate reaches the family's threshold (8/10, or 10/10 for chain and brew); the rate is over the draws that came back, and an item with fewer than threshold x draws decided is undecided rather than a pass. Reported per family: gate (items passing), accuracy (mean per-item rate), greedy, and bank — the rate the bank recorded for Qwen3.6-27B on the same items, which is what the new number should be read against. Results land in outputs/capable/<model with slashes as underscores>/<family>/capable.json, cached, so a re-run resumes. dry_run=True prints the first question and the grade prompt and makes no call.

Two limits to know. The answering call goes through OpenRouter with a JSON schema and reasoning effort minimal by default (reasoning_effort= to change it), so it is not a raw continuation, and a model that reasons in hidden tokens can pass the no-chain-of-thought families (chain, brew, arithmetic) in a way the banks' own gate would not allow. And the check covers the answerable half of some gates only: capable.json carries a partial line saying what it misses.

Which families port, and which need work

Three buckets, then the detail.

Usable as they are, once capable says the model does the task (no bank edits): association, basic_readout (minus its implicit third), multihop, multilingual, typo, the six multi-token families, chain_intermediates, brew_intermediates, arithmetic_intermediates, user_modeling, conjunctive_association, relational_multihop, role_bound_association, buggy_code.

The bank itself has to be edited for the new model: poetry (the rhyme target is whatever word that model commits to), moral_rationale (each item's YES/NO reasons are written for the side the model takes), and basic_readout's 32 implicit items (the gold is the model's own favourite).

Nothing to port until you re-run something on the new model: hallucination (its bank IS the model's own responses), agentic_misalignment (the rollouts), jlens_concept_pr (the activations and the J-lens reference), directed_modulation and multi_concept_directed_modulation (the compliance screen), and jailbreak_recognition, whose items carry over but whose read positions are token indices and need re-capturing.

Two things are model-specific no matter what the gate says. Read positions are token indices, so any re-capture needs the new model's tokenizer. And the prompt-only and lucky-guessing floors in evals/baselines/ were measured with Qwen3.6-27B and Gemini 3.8 Flash; re-measure them if you change either.

Measured: two other models on the two model-specific families

wsbench capable draws=5, graded by Gemini 3.8 Flash. bank is the rate the bank recorded for Qwen3.6-27B on the same items.

Poetry ports better than expected. Both models commit to the bank's rhyme on 98 of 100 items. The exceptions are the interesting part: on b3-po-tune-june both answer June where Qwen commits noon, and each model drifts on one more item (couplet-hold-old for Gemini, couplet-star-far for DeepSeek, both answering a near-rhyme). Those items score the wrong target on the new model and need their rhyme re-gated; the rest of the bank is usable.

Moral rationale does not port. Gemini takes the other side of the ethical question on 68 of 200 items and DeepSeek on 50. Every one of those items' YES-side and NO-side reasons was written for the side Qwen took, so the family cannot be read on either model until the reasons are rewritten per item. This is the family to fix first when porting.

Sanity checks before you trust a number

1. wsbench list — families, item counts, calls per arm.
2. wsbench judge family=<f> readouts=examples/readouts/<f>.jsonl out=... dry_run=True — prints the first prompt and the grid. Read the prompt before spending.
3. Smoke limit=3 live. Check the verdicts by eye against the readout text.
4. Missing cells are fatal by design. If a run demands allow_missing=True, find out why the cells are missing before setting it.
5. Check n_unjudged_cells and n_empty_cells in results.json. An all-zero family is usually a broken key or an empty gen dir, not a finding.
6. Read the family's floor beside its rate: wsbench report dir=<out> draws the frozen lucky guessing and prompt-only columns, and marks the floors that saturate.
7. Prefer the family's own null (permutation, decoy, role-swap, derangement foil) to any analytic chance line. Any-of-grid floors saturate and must not be quoted.
8. If the judge changed, the floors stamped with the old prompt_version stop being drawn. That is the intended behaviour, not a bug: re-measure them.

Making the evals harder

The banks here are frozen. The generators that built them live in the source repo, so a harder bank is authored, gated and then frozen into evals/<family>/items.json. What "harder" has meant so far, and what a new item has to satisfy:

• Multi-token targets. A top-10 token bag can hold iron but not Simula 67, so the hard tier moved to multi-token bridges and the judge became a forced choice among five confusables.
• No leak. The target must not appear in the prompt, in any script or spelling. The arithmetic and chain banks check that no intermediate appears as a numeral in the prompt.
• A real gate. Greedy-correct, then at least 8 of 10 samples at temperature 0.7.
• Distractors drawn from the same pool. Same kind, same specificity, seeded per item so every arm and every subset sees the same list. Then measure the lucky-guessing floor: blind guessing above chance means the options leak (multihop_mt blind is 0.658 against a measured uniform 0.126 and an analytic 0.149).
• A null you can compute. A permutation over other items' answers, a decoy set matched in magnitude, a role swap, a derangement foil. Families without one are the weakest here.
• Controls in the bank, not just baselines. Clean twins, don't-think twins, ab/ba direction pairs, off-trajectory colours: a control that shares the item's surface but not its answer is what separates reading from echoing.

The two prompts capable sends

Verbatim, so the grading rubric is not invisible. System prompt for the model under test:

You are answering benchmark questions. Answer the question as asked, following any format it specifies, with no preamble, no explanation and no restatement of the question. If the question asks you to continue a text, reply with the continuation only.

System prompt for the grader:

You grade a model's ANSWER against the EXPECTED answer to the same question. Mark it correct when the answer gives the expected one: the same word or value, an inflection of it, an established synonym or alias, a faithful translation into another language, or the same quantity written differently. A longer answer that contains the expected one is correct; an answer that merely mentions the topic, or names something related but different, is not. Judge only the answer in front of you.

The per-family question text is pinned by tests/golden/capable_questions.json: editing it is an instrument change and the test will say so.

What this repo will not do for you

There is no capture code, no lens implementation and no model serving here. A readouts file is the contract: {"id", "layer", "pos", "samples"} for a prose lens, tokens (+ scores) for a token lens. wsbench convert-gen-dir and wsbench convert-read-json turn the two common producer layouts into it.