Published runs - opensmartroute 0.5.0
The 0.5.0 run on MT-Bench human, PPE human and WebDev Arena: learned router vs declarative, static task table and baselines, with robustness and held-out calibration.
Produced with the recipe in ../README.md on 2026-09-06, osr 0.5.0, seed 0,
--limit 1000 per suite. One JSON per suite: <suite>.json is the learned router
(--slm slm.json --baselines --calibration --robustness), <suite>.declarative.json is the same
catalogue without the SLM (--baselines only). The per-row arrays calibration.conformal_pairs and
calibration.temperature_samples were removed from the committed files; every aggregate they feed
(reliability, held_out) is kept.
Data#
| Role | Cache name | Hugging Face dataset | Rows after --tier relabelling |
|---|---|---|---|
| train | arena-55k-tiers | lmarena-ai/arena-human-preference-55k | 1 986 |
| train | routellm-battles-tiers | routellm/gpt4_judge_battles | 4 000 |
| held out | mt-bench-human-tiers | lmsys/mt_bench_human_judgments (human) | 2 228 |
| held out | ppe-human-tiers | lmarena-ai/PPE-Human-Preference-V1 (test) | 2 360 |
| held out | webdev-arena-tiers | lmarena-ai/webdev-arena-preference-10k (test) | 1 070 |
Catalogue: ../targets.yaml (llm-small / llm-mid / llm-frontier). SLM bundle:
osr -t examples/leaderboard/targets.yaml slm train --out slm.json --cache-dir data --source arena-55k-tiers --source routellm-battles-tiers --report --seed 0 -> 5 956 rows, 10 % holdout
accuracy 0.721, ECE 0.056, sha256 8faf8466902b753a9f22382de24bfd0e1d5271c7fe5d6564dec8105310dcbb1e
(4.0 MB, not committed; the command is deterministic for the same cache).
Results#
| Suite (n=1000) | Router (learned) | Declarative | Static task table | Random | Cheapest | Best prior | Paraphrase | Repeat flips | ECE raw | ECE held-out temp. | ECE held-out isotonic |
|---|---|---|---|---|---|---|---|---|---|---|---|
| MT-Bench human | 0.523 | 0.421 | 0.541 | 0.441 | 0.372 | 0.343 | 1.000 | 0.000 | 0.328 | 0.056 | 0.048 |
| PPE human | 0.552 | 0.418 | 0.651 | 0.445 | 0.174 | 0.651 | 0.898 | 0.000 | 0.298 | 0.121 | 0.035 |
| WebDev Arena | 0.573 | 0.192 | 0.670 | 0.391 | 0.000 | 0.670 | 0.883 | 0.000 | 0.283 | 0.032 | 0.047 |
Cost and latency per 1k tokens for the learned router: MT-Bench $0.00363 / 968 ms, PPE $0.00390 / 1005 ms, WebDev $0.00324 / 925 ms (static task table: $0.003 / 900 ms on MT-Bench, $0.015 / 2500 ms on PPE and WebDev, since it degenerates to "always the best single tier").
Reading: the learned router beats the declarative router by +10 to +38 pp and random / cheapest everywhere, at a fifth of the cost of "always frontier" on PPE and WebDev. It does not beat the best single tier on any suite (-2 to -10 pp): a pairwise battle between two named models is a weak label for a three-tier decision, and the label-noise floor cannot be measured on these rows (one sample per prompt). Raw confidence is over-confident (ECE 0.28-0.33); a calibrator fitted on the even rows and scored on the odd rows brings held-out ECE to 0.03-0.05 (isotonic). These are the numbers a RouterArena / LLMRouterBench / xRouteBench entry should quote; the gated multi-sample suites (step 5 of the recipe) are what would let the router be scored against the noise floor.
Listings#
Where this run has been submitted. python scripts/release.py readiness counts rows whose status is
accepted for the v1.0 adoption item "Leaderboard listings accepted" and reports it as open until one
listing is accepted. A submission links the exact commit, the SLM SHA-256 above and this page.
| Leaderboard | Entry | Status | Notes |
|---|---|---|---|
| RouterArena | - | not submitted | submission = a PR to the RouterArena repository with runs/*.json |
| LLMRouterBench | - | not submitted | needs the gated multi-sample suites (step 5) for the noise-floor comparison |
| xRouteBench | - | not submitted | component ablation report from osr eval --ablation |