Imported from pedapudi/zicato (skills/zicato-tune-scoring/SKILL.md). Install upstream with npx skills add pedapudi/zicato --skill zicato-tune-scoring. Copyright stays with the author.

Tuning scoring.json

scoring.json turns each generation's LossProfile into a single scalar and defines the tournament promotion gate. Lower scalar = better. It is one of the three live contract files (board.jsonl, brief.md, scoring.json) next to the workspace; evolve reads it there and freezes a per-epoch copy under .zicato/epochs/{epoch_id}/scoring.json. Sibling skills: zicato-author-board, zicato-write-brief. See SCORING.md.

The on-disk keys (canonical, from ScoringWeights) are PLURAL. Use: per_kind_weights, per_judge_weights, severity_weights, plan_revision_weight, namespace_weights, promote_margin, pass_rate_monotonicity — NOT per_kind_weight, w_drift, tournament_margin. The dataclass field names are the truth.

The shape of the scalar

Two kinds of term, and only two:

1. One bounded correctness term — pass_weight * (1 - mean_score), over the board entries that carry an expectation. Nothing else uses that denominator, which is why correctness is not a channel.

2. One signed coefficient per measured CHANNEL — every measured signal is a namespace on namespace_weights, summed in sorted order:

   channel	what it measures	within-channel shape
   drift:	drift events + plan revisions	severity_weights × per_kind_weights
   judge:	custom process judges	per_judge_weights / default_judge_weight
   failure:	task failures + a run that did not complete	task_failure_weight / not_completed_weight
   runtime:	wall-clock seconds	—
   cost: latency: rubric: output: schema:	whatever the adapter reports under the prefix	—

scalar = pass_weight * (1 - mean_score) + Σ over channels of coefficient * mean

No channel is privileged. Turning one off turns off exactly that one: namespace_weights: {"drift:": 0.0} stops scoring drift and leaves judges, failures and cost scoring untouched. The one coefficient you may not zero is "failure:" — the loader rejects it, because a contract must not be able to make crashing free.

Tuning is three layers of escalating power, neutral by default — adding none of the lower layers leaves scoring byte-identical to the linear weights:

1. Linear weights (below) — the common 90%: which kinds/judges/severities matter, and how the two halves combine.
2. Transform registry (the "Non-linear shapes" section) — a non-linear shape (a quadratic recall curve, a diminishing-returns aggregation, a cap) without writing operator code. Declarative, serializable.
3. Plugin escape hatch (the "Plugins" section) — arbitrary pure logic (F-beta, cost-aware) as a dotted-spec function in the operator package.

Reach for the lowest layer that expresses the change. Do NOT add a transform or plugin to do something linear weights already cover.

A real scoring.json

The shipped example (examples/zicato_examples/target_1_presentation/scoring.json):

{
  "pass_weight": 1.0,
  "severity_weights": {
    "info": 1.0,
    "warning": 3.0,
    "critical": 10.0
  },
  "per_kind_weights": {
    "confabulation_risk": 2.0,
    "looping_reasoning": 1.5
  },
  "plan_revision_weight": 0.5,
  "promote_margin": 0.01,
  "pass_rate_monotonicity": true,
  "pass_rate_monotonicity_scope": "per_entry"
}

Every key (defaults from ScoringWeights)

(The dataclass also carries an optional regression_gate_enabled / regression_test_command test-suite gate; leave it off unless the snapshot ships its own suite. The train/holdout split, tournament structure, and proposer-quality knobs also live on ScoringWeights — see zicato-configure-tournament and OVERFITTING.md for those.)

per_judge_weights and default_judge_weight

A custom judge (a board entry's judges, authored with name/mode/body/ severity — see zicato-author-board) emits the single custom drift kind on violation. per_kind_weights["custom"] therefore weights every custom judge identically. When one judge's violation should count for more, key per_judge_weights on the judge name:

"per_judge_weights": {
  "no_fabricated_numbers": 3.0,
  "incorporates_feedback": 1.0
},
"default_judge_weight": 1.0

A custom judge with no entry here scores at default_judge_weight. The names MUST match the name field on the board's judges exactly.

The promotion gate (the rules, in the order they fire)

A child replaces the parent only when EVERY applicable rule holds. Rule 1 is unconditional, Rules 2 and 3 are default-on, Rule 0 is opt-in:

• Rule 0 — diff-complexity ceiling (opt-in, diff_complexity_ceiling > 0): a structural admissibility veto applied before the scoring rules, so an over-budget edit is rejected naming the ceiling rather than a scoring near-miss (diff_complexity_ceiling: diff complexity 14 exceeds ceiling 10).

• Rule 1 — drift margin: child.scalar <= parent.scalar - promote_margin. A larger promote_margin demands a more convincing win and absorbs LLM run-to-run noise.

• Rule 2 — pass-rate monotonicity: with pass_rate_monotonicity: true, the gate guards pass-rate. The granularity is pass_rate_monotonicity_scope:

  • "per_entry" (default) — if any entry the parent passed comes back failing on the child, the child is rejected regardless of drift gains (rejection reason lists the regressing entries). Best for invariant/regression-suite boards where every entry must not regress.
  • "aggregate" — reject only when the child's overall pass-rate drops below the parent's (modulo float noise). A challenger may trade which entries pass as long as the net holds or improves. Best for sampled evaluation boards where individual pass/fail is noisy and a strictly-better challenger should not be vetoed by a single entry flip.

  Both guard against the proposer reducing drift by refusing to attempt hard entries. Flip pass_rate_monotonicity to false only for experimental epochs that expect non-monotone exploration (there is no "off" scope value).

• Rule 3 — per-namespace monotonicity (default-on for rubric: and schema:): a child that moved a guarded namespace in its worse direction is rejected even when the combined scalar improved (monotonicity_regression on namespace=rubric:). This is the rule operators most often forget they have on — a quality drop or a new schema failure vetoes an otherwise-winning challenger. Turn a namespace off in namespace_monotonicity if you mean to allow that trade.

One more veto sits after those four rules: with the default-on train/holdout split (overfitting.enabled, boards of >= 6 entries), a win the train slice measured must also not regress on the holdout, or it flips to holdout_not_confirmed. The holdout is never asked to clear promote_margin in the improving direction — merely holding flat confirms.

Non-linear shapes — the transform registry

When a linear weight can't express the shape you want (you need a curve, not just a coefficient), reach for a declarative transform. Each is a single {"op": "<name>", ...params} spec from zicato.scoring.transforms:

Two slots take a transform:

"pass_transform":  { "op": "pow", "exponent": 2.0 },
"drift_kind_aggregation": {
  "looping_reasoning": { "op": "harmonic" },
  "off_topic":         { "op": "cap", "max": 5 }
}

• pass_transform reshapes the pass/miss term (1 - mean_score). {"op":"pow","exponent":2.0} is the replacement for the retired pass_exponent field — express pass_exponent=2 as this (a stray pass_exponent key is rejected at load rather than silently dropped). Absent / linear = today's plain linear miss.
• drift_kind_aggregation reshapes, per drift KIND, how that kind's count aggregates into drift loss. An absent kind = linear = today's severity × kind_weight × count. {"looping_reasoning":{"op":"harmonic"}} opts THIS contract — and no other — into the harmonic looping curve (it used to be an unconditional core special-case for everyone).

One op per slot — no pipelines. Specs are validated fail-fast at contract load: an unknown op, a missing/non-finite/typo'd param, or a clip with lo > hi is rejected loudly at evolve time, never producing a NaN mid-scoring.

Plugins — the escape hatch for arbitrary logic

For anything the registry can't express (an F-beta recall/precision blend, a cost-aware penalty), name a dotted-spec plugin in the operator package — resolved by the SAME importer predicates/judges use:

"drift_reducer": "mypkg.contract.scoring:my_drift_reducer",   // Seam 1: per-run drift loss
"scalar_fn":     "mypkg.contract.scoring:my_scalar"           // Seam 2: per-gen scalar

Each is a pure, deterministic, NO-LLM, no-I/O function over a frozen typed context (zicato.scoring.api — DriftContext / ScalarContext). The context carries the post-transform builtin_loss / builtin_scalar, so a plugin wraps/adjusts the built-in rather than reimplementing it:

# mypkg/contract/scoring.py — immutable to the proposer, like predicates/judges
def my_scalar(ctx) -> float:
    cost = ctx.namespace_aggregates.get("cost:", 0.0)
    return ctx.builtin_scalar + 0.001 * cost   # cost-aware penalty on top of the built-in

Rules:

• Pure only — no LLM, no I/O, no wall-clock; re-scoring must be reproducible.
• Fail-open — a plugin that raises / returns NaN/inf falls back to the pre-plugin (built-in / transformed) value, logged + recorded in provenance. Never crashes the run. Watch the dashboard for fail-open flags (next section).
• Immutable to the proposer — plugins live in the operator package, never enumerated as mutation points.
• drift_reducer runs inside the killable worker; scalar_fn in the orchestrator. Both fold into the contract hash, AND the plugin module's source is hashed — editing the plugin BODY rolls the epoch.

Provenance — explaining a scalar

Each seam records a parseable provenance token: per-run loss.json::scoring_provenance (Seam 1) and per-generation gen_score.json::scalar_provenance (Seam 2). The dashboard's promote-gate breakdown decomposes them into a per-side "which transform/plugin shaped this" view. Token shapes: builtin, transform:pass=pow(2.0), transform:drift{looping_reasoning=harmonic}, plugin:scalar_fn=<spec>, and the fail-open form <token> (fallback: <reason>) — surfaced prominently (caution-colored) so a silently-degraded plugin is obvious rather than buried in a log.

scoring.json is part of the evaluation contract

Weights — AND the transforms and plugin specs — are frozen per epoch. Editing scoring.json (or a referenced plugin's body) changes the contract hash, and the next evolve (default auto-epoching) closes the current epoch and opens a fresh one. Tune between epochs rather than mid-epoch.

Calibrating promote_margin against measurement

promote_margin is the one weight you do not have to guess. Two commands measure the window it must sit inside — both are live runs that spend budget (they execute the harness), so get the operator's go-ahead first:

.venv/bin/zicato board audit     --runs 5 --harness-call-llm … --auxiliary-call-llm …
.venv/bin/zicato board preflight --runs 5 --harness-call-llm … --auxiliary-call-llm …

board audit duels the champion against ITSELF and reports the A/A noise floor (persisted onto the epoch as noise_floor). board preflight adds the degradation signal — the champion versus knowingly degraded copies of itself — and places promote_margin against the floor and that signal, naming the side it fell outside of. Every one of these is a WARNING; none stops a run:

• margin_below_floor (WARN) — promotions cannot be told from re-rolls of the same generation. Raise the margin.
• margin_above_achievable (WARN) — the margin exceeds the only movement the probe demonstrated. Read this one carefully. The probe measures how far the scalar moved when a mutation point was DESTROYED — degradation headroom, how much the champion has left to LOSE — while a promotion needs movement the other way. The two are unrelated in general, and a champion sitting near the failing end has little left to break and plenty to gain, so improvement headroom is UNMEASURED (issue #119). Worth checking the margin against what a real fix is worth; NOT evidence the run is null. The signal is also a single-point lower bound (one mutation point per probe), so a margin set above single-point reach on purpose — recombination unions two sub-margin fixes — is expected.
• empty_window (EMPTY) — the measured signal does not clear the noise floor, so no margin is defensible. Do not tune the margin; reduce evaluation noise (more replicates, steadier judges) or strengthen the board.
• holdout_note (prose, when the board is split) — the HOLDOUT confirmation's own bounds. See below.

The recommended margin is 2.5 × delta_std — the standard deviation of the A/A delta_scalar, the exact quantity the gate thresholds. Do NOT scale max_abs_delta instead: it is a range statistic that grows without bound in the draw count, so more calibration draws would inflate the recommendation (issue #112). delta_std sharpens with more draws, which is the direction a recommendation should move.

The holdout has its own margin (issue #118)

promote_margin is calibrated against the TRAIN slice. When the split is active, a promotion must also survive the holdout confirmation on a smaller slice, whose scalar moves in coarser 1/N steps — and sharing one knob left real board shapes (the default 12-train / 6-holdout split with one holdout entry flipping) with no promotable margin at all. Two contract fields split the bounds; both default to exactly today's behaviour and neither moves the contract hash at its default:

The TRAIN side keeps its zero-tolerance rule either way, so neither knob loosens the gate's primary decision.

Calibration workflow

Good weights are unknown until the loop has run real epochs.

1. Start near the defaults (the shipped channel coefficients, the shipped severity_weights).
2. Run an epoch.
3. Inspect the journal/analysis: do promoted generations match what you would have promoted by eye?
4. On disagreement, tune — a promoted generation whose idea you'd reject ⇒ drift weights are off; a rejected generation whose patches you'd accept ⇒ pass_weight too high / promote_margin too strict. Edit, start a new epoch, repeat.

What good looks like

• Up-weight the drift kinds and judges that matter for this harness (per_kind_weights / per_judge_weights); leave the rest at default.
• Keep pass_rate_monotonicity: true for serious epochs.
• A promote_margin inside the measured window — above the A/A noise floor so spurious deltas don't flip promotions, below the achievable signal so something can still win.