Name: ai-eval-harness
Author: Matrixx0070

About

Imported from Matrixx0070/sudo-skills (docs/skills/ai-eval-harness/SKILL.md). Install upstream with npx skills add Matrixx0070/sudo-skills --skill ai-eval-harness. Copyright stays with the author.

When to use

Use this when an LLM feature is real enough that a silent quality regression would hurt users, and "it looked fine when I tried it" is no longer acceptable proof.

Not for: the first prompt sketch (iterate by hand first, see ai-prompt-design), or pure latency/cost tuning (see ai-cost-latency).

Method

Collect a dataset from real traffic and known failures, not invented cases. Aim for 50-200 items; label each with the expected outcome or a rubric.
Split into a dev set (you tune against) and a frozen holdout (you only measure). Never tune on the holdout.
Pick a grader per case type. Decision point: exact/enum/JSON → programmatic assertion; open-ended → LLM-as-judge with a rubric; factual → check against a reference. Prefer the cheapest grader that is trustworthy.
Validate the LLM judge itself: have it grade 20 human-labeled items; if agreement is low, tighten the rubric before trusting it.
Compute aggregate metrics (accuracy, pass rate, per-category breakdown) plus a list of individual failures for inspection.
Set a regression gate: a threshold that must hold in CI before a prompt/model change merges. Decision point: block on holdout pass-rate drop OR any critical-category failure.
Re-run on every prompt, model, or retrieval change; archive scores per version.

Example

Feature: invoice field extraction. Dataset: 120 real invoices, expected JSON per invoice.

grader = exact_match(pred["total"], gold["total"])
        and iso_date(pred["due_date"]) == gold["due_date"]

Baseline (prompt v3): 108/120. Candidate (v4, added a few-shot example): 116/120, but 2 NEW failures in the "foreign currency" category. Gate rule "no category may regress" blocks the merge until v5 fixes currency, reaching 119/120.

Pitfalls

Vibes as eval. Eyeballing a few outputs. If it is not scored on a frozen set, it is not an eval.
Trusting an unvalidated judge. LLM judges have biases (length, position). Calibrate against human labels first.
Aggregate blindness. A steady average hides a tanked subcategory. Always break down by category.
Leaky holdout. Tuning until the holdout passes turns it into a training set. Touch it only to measure.

Output format

# Eval Suite: <feature>
DATASET: <n> items | source | dev/holdout split
LABELS: <expected outcome or rubric>
GRADERS:
- <case type> -> <programmatic | llm-judge | reference>
JUDGE CALIBRATION: agreement=<x> on <n> human labels
METRICS: overall=<x> | per-category={...}
GATE: block merge if holdout<<t> OR any category regresses
RESULTS: v<n> <score> | failures=[...]

Use it

Copy one of these into your project. Installing also returns the manifest and these snippets.

yaml

targets:
  - https://api.opensmartroute.ai/api/v1/registry/matrixx0070-sudo-skills-ai-eval-harness/manifest   # or paste the manifest below

Manifest

An Open Capability Manifest: the router reads it to know what this does, what it costs and when to pick it.

matrixx0070-sudo-skills-ai-eval-harness.ocm.jsonjson

{
  "ocm": "1",
  "id": "matrixx0070-sudo-skills-ai-eval-harness",
  "kind": "skill",
  "name": "ai-eval-harness",
  "description": "Build an eval suite for an LLM feature with datasets, graders, and regression gates that block bad releases.",
  "publisher": "Matrixx0070",
  "version": "1.0.0",
  "capabilities": {
    "domains": [
      "general"
    ],
    "tags": [
      "skill-md",
      "eval",
      "testing",
      "llm-judge",
      "regression",
      "ci",
      "github"
    ],
    "languages": [
      "en"
    ]
  },
  "quality_prior": 0.6,
  "examples": [
    "Build an eval suite for an LLM feature with datasets, graders, and regression gates that block bad releases."
  ],
  "primary": false,
  "metadata": {
    "source": {
      "provider": "github",
      "repository": "https://github.com/Matrixx0070/sudo-skills",
      "path": "docs/skills/ai-eval-harness/SKILL.md",
      "ref": "6bc5cc2f77b1ae38553e590c7511fbfdd5e1317e",
      "url": "https://github.com/Matrixx0070/sudo-skills/blob/6bc5cc2f77b1ae38553e590c7511fbfdd5e1317e/docs/skills/ai-eval-harness/SKILL.md",
      "key": "Matrixx0070/sudo-skills/docs/skills/ai-eval-harness/SKILL.md"
    }
  },
  "instructions": "## When to use\n\nUse this when an LLM feature is real enough that a silent quality regression would hurt users, and \"it looked fine when I tried it\" is no longer acceptable proof.\n\n**Not for:** the first prompt sketch (iterate by hand first, see `ai-prompt-design`), or pure latency/cost tuning (see `ai-cost-latency`).\n\n## Method\n\n1. Collect a dataset from real traffic and known failures, not invented cases. Aim for 50-200 items; label each with the expected outcome or a rubric.\n2. Split into a dev set (you tune against) and a frozen holdout (you only measure). Never tune on the holdout.\n3. Pick",
  "cost": {
    "context_tokens": 656
  }
}

Fetch it by URL: GET /api/v1/registry/matrixx0070-sudo-skills-ai-eval-harness/manifest?version=1.0.0

Reviews

Star ratings from people who tried it. One review per account; edit yours any time.

No reviews yet. Install it, try it, and be the first to rate it.