osr-evaluation

Dataset format (JSONL, one row per request)#

{"text": "Hi there! How's it going?", "acceptable": ["llm-small"]}
{"text": "Prove sqrt(2) is irrational", "expected": "llm-frontier", "context": {"tier": "pro"}}
{"prompt": "Summarise this PDF", "scores": {"llm-small": 0.4, "llm-mid": 0.8}, "task_type": "summarize"}

Fields: text (or prompt), expected (single id), acceptable (list of ids), context, profile, task_type, scores {id: quality} (RouterBench style), samples {id: [floats]}, tokens {id: tokens spent} (effort / elastic siblings). Unknown keys are ignored. A row is correct when the decision is expected or in acceptable; with scores, mean_quality reports the quality of the chosen target.

Agentic tasks (--agentic) use a different JSONL: one multi-step task per line with a per-agent, per-step success / latency table (eval.agentic.AgentTask):

{"task_id": "t1", "steps": ["look up the order", "reason through the refund conflict step by step"],
 "success": {"fast-agent": [0.97, 0.45], "strong-agent": [0.94, 0.90]},
 "latency_ms": {"fast-agent": [400, 400], "strong-agent": [2000, 2000]}, "domain": "customer_support"}

Presets: --preset routerbench|llmrouterbench|routereval|xroutebench|routerxbench maps public benchmark files onto your catalogue (eval.datasets.load_benchmark(path, preset, model_map, limit)).

CLI#

osr -t targets.yaml -r rules.yaml eval data.jsonl [--limit N] [--json]
    --frontier       2-D cost/quality frontier + area under it
    --frontier3      quality/cost/latency Pareto frontier + hypervolume
    --baselines      random / cheapest / best-prior / task-table / oracle (+ noise floor); --quality-floor 0.7
    --robustness     repeat flip-rate, paraphrase stability, profile-swap fairness (first 200 rows)
    --calibration    ECE, Brier, reliability diagram, conformal set coverage / size; plus held-out ECE after
                     fitting a temperature / isotonic calibrator on the even rows (>= 20 rows)
    --ablation       leave-one-strategy-out and leave-one-extractor-out deltas
    --multi-round    route-execute-judge-refine loop vs best single target (rows with scores); --threshold --max-rounds
    --effort         effort routing vs always-highest-effort (rows with scores + tokens; targets declare effort)
    --agentic        DATASET is agentic tasks; per-step routing vs best single agent on accuracy-latency; --retries N
    --min-accuracy X exit status 2 when accuracy < X (CI gate)
osr -t targets.yaml ope logs.jsonl [--max-weight 20]     # IPS / SNIPS / DR off-policy evaluation
osr -t targets.yaml safety [--learned-guard] [--json]    # red-team safety-routing suite
osr collect --cache-dir data --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small
    # ten public sources (routellm-battles, arena-55k/100k/140k, ppe-human, webdev-arena, mt-bench-human,
    # reward-bench, ultrafeedback, routellm-gpt4), resumable after a 429; --source NAME picks, --refresh refetches
osr -t targets.yaml slm train --out slm.json --cache-dir data --report   # every *-tiers cache; numpy when installed
    # judge --report against the constant baseline: "always mid" is acceptable ~64 % of the time on the arena corpora
OSR_SLM_ENCODER=attention osr ... slm train ... --report  # compare the transformer-block encoder on the holdout
osr slm eval slm.json data.jsonl --min-accuracy 0.8       # CI gate for the routing SLM itself

Python#

from opensmartroute.eval import evaluate, load_dataset, cost_quality_frontier, area_under_frontier, calibration_report
rows = load_dataset("data.jsonl")
r = evaluate(router, rows)             # EvalResult
print(r.summary())                     # n, accuracy, mean_cost_per_1k, mean_latency_ms, mean_confidence,
                                       # mean_quality, ece, brier, abstain_rate, set_coverage, mean_set_size
r.confusion                            # {expected: {chosen: count}}
r.errors                               # rows that missed, with the decision trace

Also: eval.baselines.baseline_suite, eval.robustness.{repeat_flip_rate, paraphrase_robustness, profile_swap_fairness}, eval.frontier.hypervolume, eval.ope.off_policy_evaluate. Roadmap exit criteria as runnable measurements: eval.criteria (cold_start_ratio, effort_token_savings, multi_round_vs_best_single, match_at_1_at_scale, conformal_coverage, knapsack_never_exceeds_cap, ope_within_live_ci, run_all) and eval.agentic.task_routing_frontier; python scripts/exit_criteria.py [--full] prints the table quoted in docs/ROADMAP.md.

Interpreting results#

• Accuracy below baselines' best-prior means the catalogue (examples / complexity bands) is wrong, not the weights. Fix data first.
• ece > 0.1 -> confidence is miscalibrated: run router.calibrate() on logged outcomes or lower OSR_ROUTING_SOFTMAX_TEMPERATURE. calibration_report()["held_out"] shows what a fitted TemperatureScaler / IsotonicCalibrator reaches out of sample (on the public human-preference suites raw ECE 0.25-0.32 drops to 0.04-0.06 isotonic); ship the fitted calibrator rather than tuning weights.
• --ablation deltas near zero identify strategies you can drop to cut latency.
• --robustness flip-rate > 0 with identical inputs means a stochastic strategy (bandit) is dominating; seed it or lower its weight.
• Use --frontier when the goal is "same quality, less cost": pick the objective weights on the knee.

CI gate pattern#

- run: osr -t examples/targets.yaml -r examples/rules.yaml eval examples/eval_dataset.jsonl --min-accuracy 0.9 --json

Keep the dataset in the repo, deterministic (seed=0 strategies), and add a row for every routing bug you fix.