Imported from justsml/ExploitHunter.app (.agents/skills/tune-model-evals/SKILL.md). Install upstream with npx skills add justsml/ExploitHunter.app --skill tune-model-evals. Copyright stays with the author.

Tune Model Evals

Use this skill when changing anything that might affect model or eval behavior: prompts, product skills, system prompts, model refs, temperature/token settings, tool availability, runtime options, Ollama model size, or Ollama environment variables.

The discipline is: baseline, hypothesis, isolate, run, compare, keep or revert.

For paid production-app or remote-GPU rows, read docs/eval-production-operations.md before admission. It contains the authoritative target-routing, host-contention, remote-transport, resume, and teardown procedures; keep provider- or campaign-specific workarounds there rather than expanding this skill.

Tuning Subcommands

Use these subcommands to run a tuning loop when the user asks to improve a specific or custom dimension such as quality, cost, speed, tool-use reliability, refusal rate, evidence grounding, or loop avoidance. Treat explicit constraints from the user as hard constraints. If constraints are not given, use the common constraints below.

Subcommands can be requested as plain language or exact commands:

• eval:tune research or eval:tune analyze: inspect the current eval, target dimension, constraints, baselines, recent runs, and failure modes before proposing changes.
• eval:tune hypothesis: write one falsifiable hypothesis and acceptance criteria.
• eval:tune experiment: apply the smallest change needed to test the active hypothesis and run the narrowest useful eval matrix.
• eval:tune analyze-result or eval:tune analyze: compare baseline and experiment outputs, including raw artifacts and failure modes.
• eval:tune research-failure: investigate a concrete model/provider/harness failure against current docs, changelogs, source, GitHub issues, and comparable agent projects before spending more live-eval budget.
• eval:tune conclude: decide keep, revert, combine later, or repeat with a sharper hypothesis.
• eval:tune repeat: start the next loop from the conclusion, preserving the same target dimension and constraints unless the user changes them.

When command names collide, infer from context: before an experiment, analyze means research/analyze; after a run, analyze means analyze-result.

Common Constraints

Use these defaults unless the user provides stricter or different constraints:

• No benchmark-answer, hidden-gold, vulnerable-route, payload, prior-finding, or target-specific hint leakage into candidate-visible prompts, skills, memory, RAG, tool descriptions, screenshots, or config.
• Keep improvements generic across targets in the same class.
• Preserve approval gating, target scope, and Evidence requirements.
• Match candidate tool and turn budgets to the product usage being claimed. Unless the hypothesis explicitly studies constrained execution, use the normal product envelope; label artificial-cap rows as stress tests and never use them to declare a model broadly capable or borked.
• Match task capabilities as well as budgets. Before treating a web-research miss as model quality, verify that the candidate received the same configured and preflighted browser, HTTP, command, approval, and evidence capabilities the product would provide. A candidate statement that a needed capability is unavailable is a harness diagnostic until the effective tool inventory proves otherwise.
• Keep public-objective resolution separate from optional hidden-fact recall. A stronger artifact-backed resolution must not become an incorrect final answer merely because it omits unrelated hidden inventory. If better target evidence lowers the primary success result, close paid admissions, repair the scorer generically for every model, bump its provenance version, and preserve both resolution and extended-recall metrics.
• Never synthesize a user-authored encouragement turn after an accepted terminal transition. It falsifies the interaction, can race completion review and teardown, and turns the benchmark into a coached system. If premature stopping remains after capability and scorer repairs, test at most one clearly server-owned, pre-terminal resolution checkpoint as a separately labeled matched ablation; do not add tools, bypass approvals, reset budgets, or reopen accepted terminal state.
• Do not regress toolCalls/maxToolCalls reporting, artifact capture, diagnostics, or refusal/blocked-state visibility.
• Grade recovery by outcome. Models commonly try commands that the runtime rejects before execution, including sudo-prefixed commands, then adapt to an allowed equivalent. A safely rejected attempt is not by itself a failed row when the model recovers within the existing task budget, obtains real evidence, and reports it honestly. Record blocked attempts, repetition, wasted calls, and recovery latency as efficiency evidence. Fail the outcome when a forbidden action executes, rejected output is presented as successful evidence, or the run ends without recovery; do not invent a stricter mistake-count cutoff merely to make scoring easier.
• Prefer cheaper and faster runs while tuning, but do not keep a change that only improves cost or speed by hollowing out useful target work.
• For remote-GPU offer selection, apply explicit user constraints as hard filters first. Among qualifying offers, prefer the cheapest viable hardware newer than four years, with enough VRAM/runtime support for the exact model configuration, adequate disk and network throughput, and datacenter or strong reliability history. When practical, require a representative canary to demonstrate at least two to three parallel inference requests; keep that capacity proof separate from browser-row concurrency. Persist the compared offer ids, GPU model/count and age, VRAM, full hourly/storage price, location/datacenter status, reliability, bandwidth, disk, compatibility, and requested versus measured parallelism, including the reason for any exception.
• Use only the audited remote-inference transports in docs/eval-production-operations.md: an authenticated provider endpoint or the repository-managed supervised direct-SSH forward. Keep admissions closed when neither is available. Host relay binaries and custom TCP proxy snippets are operator-side execution outside the eval containment boundary; never improvise one to recover a campaign, and never expose an unauthenticated inference HTTP port. This does not restrict a scenario-authorized transport tool running inside its isolated target or workspace.
• Treat max output tokens and total context size as separate quality dimensions. Max output tokens cap one generated response; total context size covers the complete prompt, retained/replayed history, tool traffic, reasoning, and response admitted by the runtime. Never substitute one for the other or infer either value from the other. Cap independent judges at 2,048 output tokens. Start capable local candidates on capable hardware at 32,768 output tokens; use less only for a measured concise task or a documented hardware/runtime constraint. Require at least 32K total context for local candidates and prefer 64K or 128K where supported. Record both effective values and never change either silently to reduce cost, latency, VRAM, or provider credit reservation.
• Treat reasoning effort as a monotonic promotion experiment. Start with thinking off when the exact route has a documented, canary-verified off control; otherwise use the lowest supported tier. Increase one tier at a time only while repeated quality measurements roughly improve enough to warrant additional latency and cost, and stop on flat, noisy, or negative results. Record requested and effective reasoning settings, reasoning tokens, quality, latency, and cost for every tier. Never assume an omitted or ignored option means thinking is disabled.
• Detect context overflow as its own infrastructure class on the first provider/engine signature, including explicit requested-versus-available token counts. Do not repeatedly retry the same row or immediately shrink maxTokens. First inspect actual per-slot context, slot division, model metadata, steady-state free VRAM, and the sum of representative prompt plus planned output reservation. Prefer increasing per-request context, reducing slot concurrency, applying a documented context override on the trusted engine, or selecting a larger-memory instance. Prove the repair with a canary beyond the old failure boundary.
• Internet research is encouraged for new model/runtime failures: start with publisher and engine documentation/source, then active upstream issues, then well-corroborated community reports. Never execute random scripts or replace trusted inference binaries from a community suggestion. Record promising flag-only or configuration-only findings in the final report even when they were not adopted.
• Product tests should prove behavior, not exact prompt text or tuned model defaults.
• For every new or repeated model/provider/harness failure, run an external failure-research pass before more expensive reruns unless the next rerun is only to reproduce the signature cheaply.
• Treat harness, setup, configuration, environment, target, and observability failures as repair work in the active eval campaign. Stop admissions, preserve the invalid evidence, diagnose and fix the cause, repeat the zero-cost gates, and resume with one replacement canary. Acknowledging, excluding, or working around broken rows does not complete the requested matrix.
• Make judges as deterministic as the provider permits: use temperature 0 and a fixed seed when supported, disable thinking when supported, otherwise use the lowest supported reasoning tier. Persist the effective settings and verify them with the production-sized structured schema before paid admission.

Dimension Contract

Every loop must name:

• Primary dimension: the thing being improved, for example quality, cost, speed, tool-call validity, evidence grounding, coverage breadth, timeout rate, refusal accuracy, or onboarding drift.
• Guardrail dimensions: metrics that must not regress, such as authorization safety, contamination risk, cost ceiling, runtime ceiling, evidence count, toolCalls/maxToolCalls, diagnostic preservation, or finding precision.
• Constraints: explicit user constraints plus the common constraints above.
• Budget: max runs, max wall time, cost ceiling, model set, target set, or "stop after one loop" when not specified.

eval:tune research / eval:tune analyze

Use this before changing anything.

Inputs:

• user-requested primary dimension and constraints,
• eval scripts, scenario ids, target, model set, current prompt/config/skill state,
• latest baseline and experiment outputs if available,
• dirty worktree and staged changes.

Actions:

1. Identify the exact eval path and command that measures the primary dimension.
2. Check whether a baseline exists for the same commit, target, model, runtime, scorer, timeout, tool budget, and reset mode.
3. Inspect raw responses, artifacts, scorer notes, tool traces, and errors for the top failure mode.
4. Decide whether the next step is rerun baseline, write a hypothesis, or ask for missing constraints.

Output:

Tuning target:
- Primary dimension:
- Guardrail dimensions:
- Constraints:
- Budget:

Current evidence:
- Baseline:
- Recent runs:
- Dominant failure mode:
- Missing data:

Next command:
- eval:tune hypothesis | eval:tune research-failure | rerun baseline | ask for constraints

eval:tune research-failure

Use this whenever a run exposes provider errors, malformed or dropped tool calls, reasoning/usage/cost metadata anomalies, repeated tool loops, timeouts, missing assistant text, strict real-LLM evidence failures, stream projection mismatches, package-version churn, or adapter-specific behavior. The goal is to discover credible fixes, workarounds, upstream confirmations, or cross-project patterns before burning more model budget.

Inputs:

• exact failure signature and error text,
• model URI, actual provider/model identity, package versions, AI SDK/Mastra/adapter versions,
• result path, raw stream path, Mastra/app DB evidence, trace/span ids,
• current code path and any recently changed config or package versions.

Search priority:

1. Current official docs and changelogs for the package or provider involved.
2. Source and issues/discussions in the primary upstream repo, for example AI SDK, OpenRouter provider, Mastra, model-provider SDKs, or the relevant MCP/browser adapter.
3. Similar reports in active agent projects and harnesses, especially LangChain/LangGraph, Pydantic AI, OpenHands, opencode, Roo Code, TanStack AI, Hermes/OpenFang, ScienceAgentBench/SciBench-style harnesses, SWE-bench/OpenHands benchmarks, and lm-evaluation-harness.
4. Community posts only as weak leads. Never treat blogs, Reddit, or one-off snippets as sufficient evidence without repo/source confirmation or a local repro.

Search query templates:

"<exact error>" "<package or provider>"
"<model/provider>" "tool call" "AI SDK" GitHub issue
"<adapter package>" changelog "<symptom>"
site:github.com <project> "<symptom>" "tool" "retry"
site:github.com <project> "<symptom>" "max iterations" OR "loop"
site:github.com <project> "<provider>" "<model>" "tools"

Evidence grading:

• confirmed-upstream: primary repo issue, merged PR, changelog, official doc, or source code clearly matches our signature/version.
• probable-upstream: multiple independent repo issues or comparable projects report the same package/provider/model signature.
• local-harness-bug: our stream, trace, DB, or code path contradicts the provider behavior or drops valid events.
• model-behavior: provider delivered valid events, but the model repeatedly chooses invalid/no-op behavior.
• unverified-lead: interesting but not enough to act on without a repro.

Allowed outcomes:

• small, reversible patch such as version upgrade/downgrade, provider-option change, try/catch around flaky parsing, stricter diagnostics, loop guard, cost fallback, tool-schema simplification, or adapter swap behind a flag;
• stop the eval batch and put the finding at the top of the report when a credible upstream/package issue could invalidate more rows or inflate cost;
• rerun one cheap reproduction after patching or after pinning a suspected version;
• ask the user before adopting a new package, replacing an adapter, disabling a provider feature, or changing a user-visible model path.

Safety rules:

• Do not run random scripts, install unreviewed packages, paste opaque patches, or trust issue comments without reading the code path.
• Verify installed versions locally before applying online advice.
• Prefer official package upgrades/downgrades, documented options, and minimal local guards over large dependency swaps.
• Preserve raw diagnostics and cost data; do not hide provider errors as model-quality failures.
• Do not contaminate candidate-visible prompts with target-specific failure notes or benchmark answers. Failure research belongs in harness analysis, reports, tests, docs, or scorer diagnostics.

Output:

Failure research:
- Signature:
- Local evidence:
- Classification: confirmed-upstream | probable-upstream | local-harness-bug | model-behavior | unverified-lead
- Sources:
- Candidate fixes/workarounds:
- Risk:
- Decision: stop evals | patch then rerun | keep running with warning | ask user
- Report headline:

eval:tune hypothesis

Write exactly one testable hypothesis. Do not bundle unrelated changes.

Output:

Hypothesis:
If we change <one thing>, then <primary dimension metric> should improve by <margin> because <observed failure mode>.

Change surface:
prompt | skill | system prompt | tool schema | model config | runtime | Ollama env/model size | scorer/reporting

Baseline:
<path/id/command and key metrics>

Acceptance:
Keep only if <primary metric> improves by <margin> within <constraints> and guardrail dimensions do not regress beyond <margin>.

Experiment command:
<exact command(s)>

Regression checks:
<adjacent scenario/model/target or "none for one-loop exploration">

eval:tune experiment

Apply only the active hypothesis and run the narrowest useful eval matrix.

Actions:

1. Confirm the baseline is fresh or rerun it first.
2. For a paid production-app run after runtime source changes, stop the service, run pnpm build, record .next/BUILD_ID, restart, exercise real HTTP paths, and inspect a fresh error-free service log window. A restart alone still serves the old compiled code.
3. Run the repository's isolated PostgreSQL production smoke when available. Treat it as complementary direct-source coverage, not proof of the deployed .next bundle.
4. Make the minimal code, prompt, skill, config, runtime, or env change.
5. Run one zero-cost integration batch before admitting a paid replacement canary. Run paid eval services with immediate Langfuse export so native candidate spans become queryable without shutting down the long-lived app; retain bounded polling for asynchronous ingestion visibility.
6. Inspect every effective candidate and judge URI before admission. Require a 32,768 output allowance for capable local candidates on capable hardware unless a measured concise-task or documented hardware/runtime exception applies. Require independent judges to use no more than 2,048 output tokens. Reject local candidate context below 32K without measured evidence; prefer 64K or 128K.
7. Verify command-scoped model settings survive dotenv loading. Compare every effective recorded candidate and ancillary URI with the launch intent; close admissions if stale repository configuration replaced an explicit route or output cap.
8. Run the exact experiment command.
9. Save the command, environment, production build ID, result path, raw output path, effective candidate/judge output caps, and exit status.
10. Stop new admissions on the first server, database, provider, browser, harness, judge, or telemetry error, or if cost, runtime, target scope, approval policy, contamination constraints, or output-cap policy are breached.
11. After a transport or long-context repair, exercise the failed production request shape—not only a small health completion—before admitting the replacement row.

Output:

Experiment run:
- Change:
- Command:
- Result path:
- Exit status:
- Runtime:
- Cost/tokens:
- toolCalls/maxToolCalls:
- Constraint breaches:

eval:tune analyze-result / post-run eval:tune analyze

Compare the experiment against the baseline using like-for-like fields.

Required checks:

• score and normalized score delta,
• primary dimension delta,
• guardrail dimension deltas,
• findings/signals and whether they are Evidence-backed,
• artifact count and key paths,
• toolCalls/maxToolCalls,
• invalid tool calls, loops, repeated calls, dead-end circling, and generic onboarding drift,
• elapsed runtime, timeout status, token counts, cost,
• errors, refusals, blocked states, and diagnostics,
• raw response excerpt and whether the answer was synthesized from Evidence.
• external failure-research status for every new or repeated failure signature.

Output:

Then list:

• strongest improvement,
• strongest regression,
• contamination or generality risk,
• raw artifact evidence,
• upstream/docs/GitHub findings and whether they justify stopping more evals,
• recommended conclusion.

eval:tune conclude

Make one decision:

• Keep when the primary dimension improved enough and guardrail dimensions stayed within constraints.
• Revert when it failed, leaked target-specific hints, reduced generality, worsened safety, or only improved a weak proxy.
• Combine later when the effect is promising but should be tested with another isolated change.
• Repeat when data is inconclusive, noisy, underpowered, or reveals a sharper failure mode.

Output:

Decision:
keep | revert | combine later | repeat

Reason:
<one paragraph tied to metrics, raw artifacts, cost, runtime, and constraints>

Changes kept/reverted:
<files/config/env>

Next loop:
<next primary dimension or next hypothesis>

eval:tune repeat

Start the next loop without losing scientific control.

Actions:

1. Carry forward the primary dimension and constraints unless the user changes them.
2. Promote the previous conclusion into the new baseline only if the decision was Keep.
3. If the previous result was noisy, repeat baseline and experiment before changing another variable.
4. Choose the next highest-value failure mode from raw outputs, not from aggregate score alone.
5. Return to eval:tune hypothesis.

Guardrails

• Do not leak benchmark answers, prior findings, target-specific routes, payloads, account names, hidden scorer labels, or tuned hints into candidate-visible prompts, skills, memory, RAG, tool descriptions, screenshots, or config.
• Tune generic behavior: instruction clarity, evidence contracts, stop conditions, tool schemas, tool budgets, model settings, runtime limits, and local model environment.
• Separate model-comparison evals from prompt-improvement evals. Semi-directed strategy prompts belong in prompt-improvement/judge evals, not direct model comparisons.
• Every result summary must include toolCalls/maxToolCalls.
• Prefer path-independent examples and category lists over specific strings, routes, packages, products, payloads, or CVE names.
• Do not add low-value product tests that lock exact tuning knobs such as temperature, max tokens, reasoning effort, model id, or query-string ordering. Put tuning values in eval metadata and experiment reports. Product tests should assert outcomes: artifacts saved/vectorized, memory loaded, tools called or gated, streams projected, Render JSON emitted, findings grounded, and diagnostics preserved.
• Make LLM calls mockable at the model/provider boundary. In Mastra chat-model tests, use EXPLOIT_HUNTER_MOCK_LLM=true or EXPLOIT_HUNTER_LIVE_LLM=false with setMastraChatModelFactoryForTests. Use mocks mainly for response parsing, tool-call handling, stream behavior, refusals/errors, and persistence side effects. Live LLM evals must be opt-in and should default to cheap models such as local gemma4, DeepSeek V4 Flash, GPT OSS 120B, or Gemini Flash/Flash Lite-class models.

External Failure Research

Treat live eval failures as research tasks, not just rows in a table. If a model run fails or behaves strangely, first reduce the local signature, then search for current upstream evidence and known workarounds.

High-value signatures include:

• OpenRouter or provider warnings about reasoning details, signatures, encrypted reasoning, missing usage, or malformed stream chunks.
• AI SDK errors such as no-such-tool, invalid tool input, tool-call repair failures, tool-call validation failures, missing tool results, dropped tool-call events, or finish_reason=tool_calls with no parsed tool calls.
• AgentController/Mastra projection issues such as unmatched tool-call ids, missing assistant text, missing model-inference spans, or background-task chunks not projected.
• Provider/model-specific failures: Gemini/Gemma tool loops, Kimi/Qwen tool-call drift, GPT OSS provider validation errors, or DeepSeek reasoning-only outputs.
• Harness failures: target cleanup, max-iteration stops, stale workspaces, repeated identical tool calls, cost fallback gaps, stream files missing, or strict real-LLM evidence failures.

Research sources to check:

• official docs/changelogs for AI SDK, Mastra, OpenRouter provider, and direct model-provider SDKs;
• primary GitHub issues/discussions and merged PRs for the exact package/version;
• comparable agent/harness projects such as LangChain/LangGraph, Pydantic AI, OpenHands, opencode, Roo Code, TanStack AI, Hermes/OpenFang, ScienceAgentBench/SciBench-style harnesses, SWE-bench/OpenHands benchmarks, and lm-evaluation-harness;
• community posts only as pointers to verify elsewhere.

Examples of credible patterns to look for:

• AI SDK documents NoSuchToolError, InvalidToolInputError, ToolCallRepairError, try/catch handling, stream error parts, and experimental tool-call repair. Check whether a local repair hook, clearer tool-error surface, or schema simplification is a smaller fix than changing prompts.
• AI SDK loop control distinguishes step limits from individual tool-call limits. If one step can emit many tool calls, add a real call budget or repeated-call guard instead of assuming maxSteps is enough.
• LangChain's middleware model includes model-call and tool-call limits with run-level and tool-specific caps. Use that as a design reference when our evals show runaway tool loops.
• OpenHands-style evaluation loops repeat actions until completion or max iterations, with simulated user responses for non-executable actions. If our agent asks for help repeatedly, add a benchmark-mode simulated response or stop condition rather than letting the run drift.
• Pydantic AI issues show that increasing retries alone can repeat the same bad behavior; malformed tool-call recovery needs actionable error context and bounded retry accounting.
• Hermes/OpenFang-style loop guards hash tool name plus serialized args to detect repeated patterns before the hard iteration cap.
• OpenRouter provider issues and changelogs can identify version-specific reasoning-signature, structured-output-plus-tools, stream lifecycle, and adapter incompatibilities. Check whether upgrading, downgrading, disabling a provider option, or using a direct provider/OpenAI-compatible adapter is safer for the next experiment.
• Treat signed/encrypted reasoning metadata as provider protocol state within the active replay window. Managed context may replace complete older turns with an explicit compacted observation, and a narrowly scoped provider adapter may remove details known to be invalid for replay, such as unsigned text in a signature-required format. Preserve every retained opaque payload and its ordering; never generically edit, merge, relocate, duplicate, expose, or invent signatures. A valid production regression must traverse the same managed-context, adapter, and durable-store backend as the paid run, because a SQLite-only sanitizer proves nothing about PostgreSQL history replay.
• When investigating a signature failure, compare structural metadata only (message/part order, tool-call grouping, detail type/format/id presence, and payload hashes). Never print opaque signature or encrypted payload contents into logs or reports.

Never blindly apply a workaround found online. Convert it into a hypothesis, verify it against local code and installed versions, then run the narrowest reproduction.

Top-of-Report Findings

Every eval report or summary with credible external failure research must put a short External Research Findings or Run-Blocking Findings block immediately after the top cost/token lines and before aggregate score tables.

Include one row per important finding:

Severity:

• blocker: likely invalidates more live eval rows, risks major cost inflation, loses incurred cost data, drops tool calls, or hides provider/harness failures.
• warning: credible upstream/provider issue or workaround exists, but current rows remain interpretable with caveats.
• note: weak lead or non-blocking improvement idea.

If severity is blocker, stop or pause further expensive eval batches until the finding is patched, explicitly waived by the user, or isolated to a non-selected model/provider path. Do not bury this information in footnotes, raw logs, or final paragraphs.

Baseline First

Terminal and harness experiments

Use the research terminal protocol as the operating contract for stopping experiments.

• Keep provider finish, controller result, terminal transition, and deterministic oracle result as separate fields.
• Compare only matched harness configurations. Model URI, prompt hash, target snapshot, tool profile, context, output cap, tool-call cap, and scorer version must match unless one is the declared treatment.
• Use the versioned exploit-hunter-current, typed-finish, typed-finish-gate, and typed-finish-gate-wrapup arms. Keep the legacy arm untouched.
• External Qwen Code, Mastra Code, OpenCode, and vendor CLI adapters run through pnpm eval:harness:matched -- --manifest=<path> and remain harness-smoke-only until their sandbox, permissions, staging, target lifecycle, and accounting contracts are equivalent and recorded. Adapter commands are argv arrays, never shell strings; normalized results must report observed configuration and isolation.
• Score exact success and highest deterministic checkpoint separately. A judge cannot promote a checkpoint or exact solve.
• Run the offline provider-protocol conformance fixtures before spending on adapter changes.
• Use --isolation-mode=benchmark-grade for rows claiming publication-grade isolation. Static claims only open pre-admission. Require manual approval mode plus observed durable authorization/approval rows, egress profile, fresh target identity, confined workspace, separated install phase, unchanged grader snapshot, randomized-secret digest attestation, positive exploit-attribution events, target teardown, and non-empty MP4. Any missing observation closes admission rather than becoming a model failure.
• Record Observer and Reflector route, tokens, duration, cost provenance, and failures separately from candidate usage.

For matrix runs, use a constant work queue and dynamically admit 1-4 concurrent rows according to current CPU, free memory, GPU memory, and isolated-target capacity. Keep the queue populated so the next row begins as soon as a worker finishes. Scale down by withholding new admissions, not by interrupting active paid rows. Force production mode after loading repository environment, give every target-backed row a fresh isolated target, require a verified non-empty MP4 for browser/Electron rows, and tear down every per-row target in a finally/equivalent cleanup path. When browser rows use Docker-backed targets, pre-stage admitted targets before launching an MCP-owned isolated Chrome and before paid submission; avoid container lifecycle changes while another submitted Chromium row is active. In Linux environments where the MCP-owned browser otherwise exits with Target closed, pass Chrome's --no-sandbox argument explicitly and prove the exact launch configuration with a zero-target list_pages plus representative navigation canary. Prefer this deferred MCP-owned browser lifecycle over keeping an external Chrome alive across Docker target changes, which can surface ERR_NETWORK_CHANGED. Ensure page-state parsing accepts the MCP provider's titled page-list format and verify dialog handling before admitting an alert-triggering paid row. A post-submission browser lifecycle or network-state error closes admissions; preserve it as infrastructure evidence and resume at concurrency one only after target staging, browser preflight, and deferred teardown are narrowly verified.

Before opening paid admissions, distinguish a provider key's configured spending ceiling from its funded credits. After zero-cost service/database/telemetry gates pass, send one tiny direct request through the exact provider, model, and planned output cap. HTTP 402/insufficient-credit closes the provider gate even if a key-status endpoint reports unused limit. Preserve any already-admitted failures, do not silently lower output/context limits, and resume only after the authenticated account's usable balance changes and one exact-config replacement canary passes.

For asynchronous completion signoffs, candidate completion is not admission evidence. Poll the durable review to a terminal state before classifying the row. Missing, pending, malformed, or failed reviews are infrastructure failures; low but schema-valid completed scores remain honest model-quality evidence. Judge-only structured calls should not inherit candidate tools or tool choice, and must preserve the full structured object rather than accepting a streamed terminal field as the result.

Before each experiment, establish a fresh baseline for the exact scenario, model, target, runtime config, and commit state you will compare against.

Record:

• git commit and dirty worktree state,
• eval id, scenario ids, target, run mode, prompt mode,
• model URI, model size/tag, provider, temperature, max tokens, reasoning settings, context size,
• tool budget and toolCalls/maxToolCalls,
• runtime limits, timeout, concurrency, reset mode, network profile,
• Ollama/local env when relevant: OLLAMA_CONTEXT, OLLAMA_FLASH_ATTENTION, OLLAMA_KV_CACHE_TYPE, OLLAMA_NUM_PARALLEL, OLLAMA_MAX_LOADED_MODELS, OLLAMA_KEEP_ALIVE, selected model tag,
• score, findings/signals, evidence artifact count, URL/surface coverage,
• cost, token counts, elapsed runtime,
• errors, timeouts, invalid tool calls, loop/cycle/dead-end detections.

If the baseline is stale, missing, from a different model/config, or contaminated by a different prompt, rerun it.

Experiment Design

Write a short experiment log before making changes:

Hypothesis:
If we change <one thing>, then <metric> should improve by <margin> because <reason>.

Change surface:
prompt | skill | system prompt | tool schema | model config | runtime | Ollama env/model size

Baseline:
eval id/path, model, target, scenario, metrics

Acceptance:
keep only if <primary metric> improves by <margin> and no guardrail metric regresses beyond <margin>.

Regression checks:
evals/scenarios/models that must not get worse.

Test one hypothesis at a time. Do not bundle changes unless the experiment is explicitly testing a combined hypothesis; even then, run individual-effect experiments first when feasible.

Tuning Surfaces

Prefer changes in this order unless evidence suggests otherwise:

1. Candidate-visible prompt/skill/system prompt wording

   • Make instructions more generic, concrete, and action-oriented.
   • Add evidence contract, output contract, stop condition, and approval boundary.
   • Remove ambiguity that causes onboarding, refusal, or generic help text.
   • Do not add target-specific hints.

2. Tool and runtime shape

   • Reduce tool clutter for small models.
   • Make the first useful action obvious.
   • Tighten tool descriptions and schemas.
   • Tune max steps, timeout, concurrency, reset behavior, and seeded evidence policy.
   • Treat a quiet client stream as ambiguous while server/model/tool work is active. Warn or require an explicit retry; do not automatically abort and replace a paid turn, and serialize an approved replacement after the prior runtime is fully idle.

3. Model configuration

   • Tune temperature, maxTokens, context size, reasoning options, provider options, and model URI.
   • Keep llm://provider/model refs and use existing model config helpers.

4. Local/Ollama configuration

   • Select model size/tag based on available RAM and target latency.
   • Tune env vars such as context size, flash attention, KV cache type, parallelism, loaded-model count, and keep-alive.
   • Run local models sequentially when memory pressure or contention would confound results.

Small Model Hypothesis Seeds

When local or small models produce generic onboarding, weak findings, invalid tool calls, too few tool calls, or mostly baseline-derived signals, start with hypotheses like these. Treat them as seeds, not a fixed list; inspect the observed outputs and invent sharper hypotheses from the actual failure mode.

• Stronger target-work framing: If the prompt explicitly suppresses onboarding and requires immediate target work, then generic-help drift should decrease and first useful action rate should improve.
• Simpler tool surface: If the run exposes fewer tools or simpler tool schemas, then valid tool-call rate should improve and malformed tool calls or tool-schema repair loops should decrease.
• Required first read/probe step: If the prompt or runner requires one concrete read/probe before final answer, then zero-tool runs should decrease and evidence-backed findings should increase.
• Pre-digested local evidence: If local models receive a compact, generic URL inventory or evidence artifact summary, then target-specific reasoning should improve without leaking hidden answers.
• Tool-like text recovery: If a local wrapper detects tool-like text emitted as plain text and retries with a repair instruction, then valid tool calls should increase and dead-end conversational replies should decrease.

For each seed, run it as its own experiment first. Only test combinations after individual effects are measured.

Scientific Loop

For each experiment:

1. Capture or rerun the baseline.
2. State one falsifiable hypothesis and expected movement.
3. Apply only the intended change.
4. Run the same eval matrix needed to prove improvement and guard against regression.
5. Compare against baseline using the same scorer and output fields.
6. Inspect raw responses and artifacts, not only aggregate scores.
7. Decide:
   • Keep when improvement clears the margin and no important regression appears.
   • Revert when it fails, creates contamination risk, reduces generality, or only improves a weak/proxy metric.
   • Combine later only after individual effects are understood.

Metrics To Collect

Required for every run:

• score and normalized score,
• findings/signals and whether they are evidence-backed,
• evidence artifact count and relevant artifact paths,
• toolCalls/maxToolCalls,
• invalid tool calls, tool-schema repair attempts, tool loops, repeated calls, dead-end circling,
• elapsed runtime and timeout status,
• token counts: input, output, reasoning, cached input when available,
• estimated cost and budget spent,
• errors, blocked states, refusal/onboarding/generic-help drift,
• raw response excerpt and whether the answer was synthesized from evidence.

Loop/dead-end indicators include repeated identical tool calls, repeated malformed tool calls, repeated requests for already-provided target context, generic onboarding after a concrete target prompt, and long runtime with zero tool use.

Comparison Rules

• Compare like with like: same target, scenario, run mode, model, timeout, tool budget, reset mode, and scorer unless the experiment is changing one of those fields.
• Use absolute and relative deltas. A tiny score bump with doubled runtime or worse tool behavior may not be worth keeping.
• Treat a finding as weak if it is only inferred from baseline/seed evidence and the model did not produce target-specific reasoning or valid tool use.
• Run at least one adjacent regression scenario when changing shared prompts, skills, system prompt, tool descriptions, or runtime defaults.
• If results are noisy, repeat baseline and experiment runs before deciding.

Reporting Format

End with a compact experiment table:

Then summarize:

• what improved,
• what regressed,
• whether local/small models used tools or only baseline evidence,
• loop/dead-end patterns,
• kept changes and reverted changes,
• next highest-value hypothesis.

Do not claim success from aggregate score alone. Tie the decision to raw responses, evidence artifacts, tool behavior, cost, and regressions.