Imported from microsoft/ACESEvals (.github/skills/eval-analysis/SKILL.md). Install upstream with npx skills add microsoft/ACESEvals --skill eval-analysis. Copyright stays with the author.

Eval Analysis Skill

Overview

SABER evaluation analysis is a structured data science framework for comparing AI agent performance across cybersecurity benchmark domains. The analysis pipeline transforms .eval log files into actionable insights through 12+ standardized experiments plus domain-specific analyses.

Getting Started: Sample Eval Data

No evals to analyze yet? The repo ships with pre-run sample .eval files and can auto-download more from HuggingFace.

Option 1: Use bundled sample evals (fastest)

The eval_samples/ directory contains pre-run evaluation results for all 3 domains across 5 models and 3 agent architectures:

eval_samples/
├── excytin/          # 20 eval files (5 models × default + reasoning baselines + 3 agents)
├── cybench/          # 17 eval files
└── cti_realm/        # 11 eval files

Models included: Claude Haiku 4.5, Claude Sonnet 4.6, Claude Opus 4.6, GPT-5.4, GPT-5.4-mini
Agent architectures included: React, GH Copilot, Claude Code (for Sonnet 4.6)
Baselines: No-reasoning/no-thinking variants for extended thinking comparison

The notebooks are pre-configured to load from eval_samples/. Just run any notebook — no configuration needed.

Option 2: Auto-download from HuggingFace

If .eval files are missing locally, ensure_eval_files() automatically downloads them from the AcesEvals HuggingFace dataset:

from saber_analysis import ensure_eval_files

ensure_eval_files(
    eval_logs=EVAL_LOGS,       # {display_name: filename.eval}
    log_dir=LOG_DIR,           # local directory
    domain='excytin',          # HuggingFace subfolder
    fallback_dirs=[            # check these before downloading
        str(REPO_ROOT / 'eval_samples' / 'excytin'),
        str(REPO_ROOT / 'latest_experiments' / 'excytin'),
    ],
)

Resolution order: local log_dir → fallback_dirs → HuggingFace download
HuggingFace repo: anandmudgerikar/AcesEvals (dataset type)
HuggingFace path: latest_experiment_samples/{domain}/{filename}.eval

Option 3: Run your own evaluations

If you need results for a model or agent architecture not in the sample evals:

# Run for a new model
uv run inspect eval domains/<domain> --model <api/model-id> --display plain

# Run for a specific agent architecture
uv run inspect eval domains/<domain> --model <api/model-id> -T agent=react --display plain
uv run inspect eval domains/<domain> --model <api/model-id> -T agent=copilot --display plain

# Run with extended reasoning disabled (for baseline comparison)
# Set INSPECT_EVAL_MODEL_ARGS=reasoning_effort=none or configure in .env

Then copy the .eval file to eval_samples/<domain>/ or latest_experiments/<domain>/ and add it to the notebook's EVAL_LOGS configuration.

What's available vs. what you need to run

Notebook Inventory

Shared Analysis Library

The notebooks/saber_analysis/ package provides reusable utilities:

Data Pipeline

.eval ZIP archives
    │
    ├── header.json          → run metadata, overall saber_overall scores
    ├── samples/*.json       → per-sample messages, scores, timing, tool calls
    └── summaries.json       → aggregated score summaries
    │
    ▼
inspect_ai.log API
    ├── read_eval_log(path, header_only=True)       → EvalLog with results.scores
    ├── read_eval_log_sample_summaries(path)         → per-sample scores & timing
    └── read_eval_log_sample(path, id=...)           → full messages & tool calls
    │
    ▼
saber_analysis library
    ├── ensure_eval_files()  → locates locally or downloads from HuggingFace
    ├── load_eval_logs()     → parses into DataFrames (overall_df, samples_df, subtasks_df)
    ├── load_trajectory_data() → extracts tool calls, steps, timing per sample
    └── extract_cost_rows()  → builds cost DataFrame from token usage + pricing
    │
    ▼
Notebook Configuration
    ├── EVAL_LOGS: dict[str, str]      → {display_name: filename.eval}
    ├── COLORS: dict[str, str]         → {display_name: hex_color}
    ├── PRICING: dict[str, dict]       → {api_model_id: {input, output, cache_read, cache_write}}
    ├── GROUP_FN: Callable             → domain-specific sample ID → group label
    └── NO_THINKING_LOGS (optional)    → baseline without extended reasoning
    │
    ▼
Analysis Cells (Experiments 1–14+)
    │
    ▼
Artifacts saved to notebooks/artifacts/{domain}/*.png

Key DataFrames

Domain-Agnostic Analysis Types (Experiments 1–14)

These 14 core analyses apply to any SABER domain. They appear in eval_analysis.ipynb and every domain-specific notebook.

Experiment 1: Overall Reward Distribution

• Metric: Mean saber_overall score per model
• Visualization: Bar chart with error bars (stderr) + optional hatched overlay for reasoning delta
• Interpretation:
  • Higher bars = better performance
  • Overlapping error bars = statistically insignificant differences
  • Reasoning delta (hatched) shows impact of extended thinking (positive = reasoning helps)
• Code pattern:

  overall_df.plot.bar(x='model', y='mean', yerr='stderr')

Experiment 2: Per-Group Breakdown (Domain-Specific Grouping)

• Metric: Scores segmented by GROUP_FN dimension
• Visualization: Clustered bar chart (models × groups)
• Group functions by domain:
  • Excytin: '_'.join(id.split('_')[:2]) → 8 security incidents
  • CyBench: '_'.join(id.split('_')[:-2]) → challenge types
  • CTI Realm: id.split('_')[0] → 3 platforms (Linux, AKS, Cloud)
• Interpretation:
  • Large within-group variance = model-dependent difficulty
  • Shifting model rankings across groups = no single dominant model
  • Uniform clusters = inherently hard/easy groups
• Requires: Setting GROUP_FN in configuration; set GROUP_FN = None to skip

Experiment 3: Cost Analysis

• Metrics: Total cost ($), cost per sample, token breakdown (input/output/cache/reasoning)
• Visualizations:
  1. Total cost bar chart
  2. Cost per sample bar chart
  3. Pareto frontier (Score vs. Cost — upper-left corner = best value)
• Interpretation:
  • Pareto-dominant models: no other model is both cheaper AND better
  • Arrows on Pareto chart show reasoning cost delta
  • Diminishing returns visible from frontier plateau
• Code pattern:

  cost_df = pd.DataFrame(extract_cost_rows(EVAL_LOGS, LOG_DIR, PRICING, N_SAMPLES))

Experiment 4: Token Usage Breakdown

• Metrics: Input, output, cache read/write, reasoning tokens per model
• Visualization: Horizontal stacked bar chart
• Interpretation:
  • Input tokens: Prompts, tool results, conversation history
  • Output tokens: Generated text, tool calls, answers
  • Cache read/write: Prompt caching (Claude > GPT generally)
  • Reasoning tokens: Extended thinking overhead; highly variable by model family
  • Large hatched sections = reasoning adds substantial overhead

Experiment 5: Cost Efficiency (Reward per Dollar)

• Formula: Cost Efficiency = Mean SABER Score / Total Cost ($)
• Visualization: Bar chart (taller = more efficient)
• Interpretation:
  • Highest-cost model is not always least efficient
  • Budget-constrained deployments should prioritize high-efficiency models
  • Reasoning may increase OR decrease efficiency (depends on score gain vs. cost increase)

Experiment 6: Sub-Task Score Breakdown

• Metrics: Submission score, checkpoint scores, aggregate score per model
• Visualization: Grouped bars (submission/checkpoints/aggregate) with reasoning delta overlay
• Interpretation:
  • Submission >> Checkpoints: Model reaches correct conclusion without thorough investigation (may be guessing; risky in production)
  • Checkpoints >> Submission: Model investigates well but fails at synthesis
  • Aligned: Investigation matches answer quality (ideal)

Experiment 7: Checkpoint Score Distributions

• Visualization: Violin plots per model
• Interpretation:
  • Bimodal (peaks at 0 and 1): Binary pass/fail checkpoint; few partial successes
  • Continuous: Fine-grained scoring; various intermediate scores
  • Wide violins: Inconsistent across samples; narrow: consistent
  • Long upper tails: Some samples max out the checkpoint score

Experiment 8: Domain-Specific Gap Analysis

• Purpose: Submission vs. checkpoint gap per domain grouping dimension
• Visualization: Heatmap (models × groups) showing gap = submission - checkpoint
• Interpretation:
  • Red (positive gap): Model guesses correctly without thorough investigation
  • Green (negative gap): Model investigates well but fails at synthesis
  • Yellow (aligned): Investigation matches answer quality

Experiment 9: Agent Trajectory Analysis

Data loading step that extracts per-sample behavioral data:

• n_tool_calls: Actual tool invocations per sample
• n_steps: Assistant message turns per sample
• tool_counts: Per-tool breakdown (bash, python, domain tools)
• total_time: Wall-clock seconds per sample

Experiment 9a: Score vs. Tool-Call Limit (Effort Budget)

• Calculation: For each hypothetical limit N, mean score where samples needing >N calls score 0
• Visualization: Curves per model converging to actual mean at real tool-call cap
• Interpretation:
  • Steep early rise: Efficient model (solves many tasks in first 5–15 calls)
  • Plateau: Budget increase yields diminishing returns
  • Key finding: All models typically reach 95% of final score by ~20–25 calls (Excytin) or ~33–35 (CTI Realm)

Experiment 9b: Effort Distribution per Model

• Visualization: Violin plots of tool calls and steps per sample per model
• Interpretation:
  • Narrow, low: Efficient (quick solutions)
  • Wide, high: Variable effort or struggling
  • Long upper tail: Some samples hit tool-call limit (binding constraint)

Experiment 10: Tool Usage Analysis

• Visualization: Dual panel — left: mean tool calls per tool (absolute); right: tool call ratio (fraction)
• Interpretation:
  • Domain determines tool preferences (Excytin → 95%+ bash; CTI Realm → 48% KQL tools)
  • Cross-model consistency = task design funnels agents toward certain tools
  • Model-specific tool preferences reveal different investigation strategies

Experiment 11: Time Efficiency & Reward per Tool Call

• Formulas:
  • Reward/Call = Mean SABER Score / Mean Tool Calls
  • Reward/Minute = Mean Score / Wall-Clock Time (min)
• Visualizations: Bar charts, scatter plots (effort vs. score with trend line), time distribution violin plots
• Interpretation:
  • High reward/call = doesn't waste invocations
  • High reward/minute = fast AND accurate
  • Time outliers reveal latency issues (e.g., GPT 10+ min vs. Claude 2–3 min)

Experiment 12: Score Outcome Segmentation by Effort

• Methodology: Segment by tool-call quartile (Q1–Q4); show perfect/partial/zero score rates
• Visualization: Stacked bar chart per model per quartile
• Interpretation:
  • Perfect-score rate drops Q1→Q4: High effort = harder task (effort doesn't help)
  • Zero-score rate doubles Q1→Q4: Tool-call limit is binding constraint on hard tasks
  • Partial scores rise in Q4: Upper-effort tasks succeed only partially within budget

Experiment 13: Cross-Model Difficulty Agreement

• Calculation: Spearman rank correlation of per-sample scores for each model pair
• Visualization: Heatmap with ρ values and significance markers
• Interpretation:
  • High ρ (>0.7): Models agree — difficulty is intrinsic to task
  • Low ρ (<0.3): Models struggle differently — complementary strengths, good for ensembles
  • Within-family > cross-family: Same-vendor models agree more than different vendors

Domain-Specific Analyses

Excytin (Incident Response Forensics)

Domain context: 8 security incidents, 599 tasks, agents investigate via SQL queries through bash tool.

SQL Query Classification:

• Success (≤100 rows): Manageable, agent can parse
• Large result (>100 rows): Unrefined query; should add LIMIT/WHERE
• Empty result: Valid syntax but no matching data (dominant failure mode)
• Error: Syntax error, missing table, permission denied

Key Excytin insights:

• All models show large positive submission-checkpoint gaps (0.459–0.573) — bypassing investigation
• Bash tool dominates (95–100% of tool calls); Haiku uniquely uses python (~5%)
• Query quality separates models: Opus 79% success rate vs. GPT-5.4-mini 61%
• Empty results are more problematic than syntax errors (agents waste effort on unproductive queries)

CyBench (CTF Challenges)

Domain context: CTF exploitation challenges with multi-checkpoint progressive investigation tracking.

CyBench checkpoint progression (e.g., Labyrinth Linguist):

1. Contextualizing — understanding challenge environment
2. Understanding templating — recognizing SSTI/language context
3. Identifying attack vectors — finding injection points
4. Exploring templates — testing template engines
5. Locating the flag — finding flag file on disk
6. Crafting exploit — building final payload

Key CyBench insights:

• All models achieve 1.0 on submission but 0.167–0.500 on checkpoints — solving without demonstrating full process
• Currently 1 sample per model; distribution analyses produce summary tables instead of charts

CTI Realm (Threat Intelligence & Detection)

Domain context: 25 tasks across 3 platforms (Linux, AKS, Cloud), 5 scored checkpoints totaling 10.0 points.

CTI Realm 5-checkpoint scoring:

• C0 CTI Alignment (max 1.25): Did agent use CTI reports? Relevance to objective?
• C1 MITRE Techniques (max 0.75): Jaccard similarity of found vs. expected ATT&CK techniques
• C2 Data Exploration (max 1.0): Coverage of expected data sources queried
• C3 Query Iteration (max 0.5): Binary — did agent run ≥2 unique successful KQL queries?
• C4 Detection Quality (max 6.5): KQL F1 (weight 5.0) + Sigma rule quality (weight 1.5)

CTI Realm tool categories:

• KQL/Data (48%): execute_kql_query, list_kusto_tables, get_table_schema, sample_table_data
• CTI/MITRE (29%): list_cti_report_tags, get_cti_reports_by_tag, search_mitre_techniques
• Sigma/Detection (13%): search_sigma_rules, validate_output_json
• General (10%): bash, python

Key CTI Realm insights:

• C4 (Detection Quality) dominates total score (65% of max) but max achieved is ~59%
• Checkpoints are remarkably independent (all pairwise |ρ| < 0.4) — validates scoring design
• C0 (CTI reading) does NOT predict C4 (detection quality) — distinct skills
• More KQL queries don't linearly improve detection: Opus achieves 58% C4 with only 4.6 queries/sample vs. GPT-5.4's 9.7

Agent Architecture Analysis

Comparison Framework

Agent architecture analysis uses the same 12 experiments as model comparison but slices by agent architecture instead of model identity. Primary comparison axis: React vs. GH Copilot vs. Claude Code for a fixed model (currently Claude Sonnet 4.6).

Agent Architecture Findings (Aggregate)

Key agent architecture insights:

• React dominates on score, cost, and consistency across all domains
• All architectures use nearly identical tool-call budgets (~18.8 calls); speed difference is thinking/planning overhead
• Copilot matches React on submission scores (0.846 vs 0.852) but lags on checkpoints (0.380 vs 0.548)
• Claude Code's primary failure mode is invalid final answers (lowest submission score: 0.348)

Cross-Domain Aggregate Analysis

Normalization Method

Domains have vastly different sample counts (Excytin: 599, CTI Realm: 25, CyBench: 1). To prevent sample-rich domains from dominating:

1. Compute per-domain mean for each model/agent on each metric
2. Average per-domain means with equal weight per domain

$$\text{Aggregate Score}(m) = \frac{1}{|\mathcal{D}|} \sum_{d \in \mathcal{D}} \overline{\text{score}}(m, d)$$

Aggregate Experiment Types

Partial coverage: If a model doesn't appear in all domains, its aggregate is computed over only contributing domains (annotated in charts).

Key Insights Patterns

These are recurring patterns to look for and communicate when interpreting results:

1. Cost-Efficiency Trade-offs

• Higher-cost models often provide diminishing returns
• Typical pattern: Haiku (cheapest) → Sonnet (mid) → Opus (high) → GPT (highest)
• Haiku gets 12–18× more reward per dollar than GPT-5.4
• Marginal score gains diminish as cost increases

2. Effort vs. Outcome (Non-linear)

• More tool calls do NOT guarantee better scores
• High-effort (Q4) tasks have 15–20% more zero scores than low-effort (Q1) tasks
• Tool-call budget is a binding constraint on hard tasks, not a surplus resource

3. Investigation vs. Final Answer Gap

• Submission scores >> Checkpoint scores across ALL domains
• Typical gaps: Excytin 0.459–0.573, CyBench 0.500–0.833
• Models reach correct conclusions without thorough investigation — risky in production
• Models may be using prior knowledge or pattern-matching instead of genuine analysis

4. Speed vs. Capability (Model-Family Patterns)

• Claude models: 1.4–2.7 min/sample, lower token budgets
• GPT models: 10+ min/sample (4–6× slower), higher token budgets
• GPT's extended reasoning overhead adds latency without proportional score gains in some domains

5. Task Difficulty Varies by Domain Structure

• Easy domains (all models score high): Excytin (0.77–0.88)
• Hard domains (massive variance): CTI Realm (0.26–0.63)
• Domain complexity (tool count, checkpoint interdependencies) drives difficulty ceiling

6. Model Agreement on Difficulty

• Spearman correlations moderate (0.4–0.8) — difficulty is partly intrinsic, partly model-dependent
• Same-family models agree more than cross-family (Sonnet↔Opus: ρ=0.64 vs. Claude↔GPT: ρ=0.47–0.53)
• Implication: Ensemble strategies should pair different model families

7. Reasoning Impact is Model & Domain-Dependent

• Excytin: Minimal reasoning boost for most models
• CTI Realm: Dramatic reasoning boost for GPT (+0.16–0.18), none for Claude
• Multi-step collaborative workflows benefit more from explicit chain-of-thought

8. Architecture Consistency

• React agent: Rank 1 across all domains (perfect consistency)
• GH Copilot: Rank 1–3 depending on domain (volatile)
• Agent architecture choice has domain-dependent trade-offs

Configuration Guide

Adding a New Model to Analysis

1. Check if sample evals already exist — look in eval_samples/<domain>/ or try running the notebook (it auto-downloads from HuggingFace)
2. If not available, run the evaluation:

   uv run inspect eval domains/<domain> --model <api/model-id> --display plain

3. Copy the .eval file to eval_samples/<domain>/ or latest_experiments/<domain>/ with a descriptive name
4. Edit the notebook's Configuration cell:

   EVAL_LOGS['Display Name'] = 'filename.eval'
   COLORS['Display Name'] = '#HexColor'
   PRICING['api/model-id'] = {'input': X, 'output': Y, 'cache_read': Z, 'cache_write': W}

5. (Optional) Add a no-reasoning baseline to NO_THINKING_LOGS

Adding a New Domain

1. Create a domain-specific notebook by copying eval_analysis.ipynb
2. Define GROUP_FN for your domain's sample ID grouping:

   # Inspect sample IDs from your .eval files
   # Write a function: sample_id → group_label
   GROUP_FN = lambda id: id.split('_')[0]  # Example

3. Add domain-specific experiments after the generic 14
4. Register the notebook in the inventory table above

Adding a New Agent Architecture

1. Check if sample evals already exist — look in eval_samples/<domain>/ for *_react.eval, *_copilot.eval, *_claude_code.eval
2. If not available, run evals for each agent:

   uv run inspect eval domains/<domain> --model <model> -T agent=react --display plain
   uv run inspect eval domains/<domain> --model <model> -T agent=copilot --display plain

3. Add entries to the agent architecture notebook's AGENT_EVAL_LOGS
4. Add a color in AGENT_COLORS

Artifact Output Structure

Analysis notebooks save visualizations to notebooks/artifacts/:

notebooks/artifacts/
├── {domain}/                     # Per-domain model comparison
│   ├── overall_reward.png
│   ├── per_{group}_scores.png
│   ├── cost_analysis.png
│   ├── token_usage_breakdown.png
│   ├── cost_efficiency.png
│   ├── subtask_breakdown.png
│   ├── checkpoint_distributions.png
│   ├── submission_checkpoint_gap.png
│   ├── score_vs_tool_call_limit.png
│   ├── effort_distributions.png
│   ├── tool_usage.png
│   ├── steps_vs_reward.png
│   ├── effort_outcome_segmentation.png
│   └── cross_model_correlation.png
│
├── {domain}_agent/               # Per-domain agent architecture comparison
│   ├── overall_reward_by_agent.png
│   ├── cost_analysis_by_agent.png
│   └── ... (same experiment set, sliced by agent)
│
├── aggregate_model/              # Cross-domain model comparison (normalized)
│   ├── aggregate_overall_reward.png
│   ├── per_domain_breakdown.png
│   ├── aggregate_cost_analysis.png
│   ├── aggregate_cost_efficiency.png
│   ├── cross_domain_radar.png
│   ├── ranking_consistency.png
│   └── reasoning_impact.png
│
└── aggregate_agent/              # Cross-domain agent comparison (normalized)
    ├── aggregate_overall_reward.png
    ├── per_domain_breakdown.png
    ├── cross_domain_radar.png
    └── ranking_consistency.png

Quick Analysis Recipes

"Which model should I use for domain X?"

1. Open notebooks/{domain}_analysis.ipynb
2. Look at Experiment 1 (overall reward) for raw scores
3. Look at Experiment 5 (cost efficiency) for value
4. Check Experiment 3 (Pareto frontier) for dominated vs. non-dominated models
5. Consider Experiment 11 (time efficiency) if latency matters

"How do agent architectures compare?"

1. Open notebooks/aggregate_agent_architecture_analysis.ipynb
2. Look at overall reward per agent (Exp 1) — React typically dominates
3. Check per-domain breakdown (Exp 2) — some agents may excel in specific domains
4. Review cost analysis (Exp 3) — architecture affects cost
5. Check trajectory efficiency (Exp 6) — same tool calls, different speed

"Is extended reasoning worth it?"

1. Open the domain-specific notebook with NO_THINKING_LOGS configured
2. Look at reasoning delta (hatched bars) in Exp 1
3. Check cost-efficiency impact in Exp 5
4. CTI Realm benefits most from reasoning; Excytin shows minimal impact

"Why is my model scoring low?"

1. Check Experiment 6 (sub-task breakdown): Submission vs. checkpoint gap reveals whether the model investigates or guesses
2. Check domain-specific analysis: SQL errors (Excytin), KQL quality (CTI Realm), checkpoint progression (CyBench)
3. Check Experiment 9a (effort budget): Is the model running out of tool calls?
4. Check Experiment 12 (effort segmentation): Does the model fail on high-effort tasks?

"Programmatic analysis without notebooks"

Use the saber_analysis library directly:

import sys
sys.path.insert(0, "notebooks")
from saber_analysis import load_eval_logs, extract_cost_rows, setup_plotting

# Load data
overall_df, samples_df, subtasks_df = load_eval_logs(
    eval_logs={'Model A': 'model_a.eval'},
    log_dir='latest_experiments/excytin',
    group_fn=lambda id: '_'.join(id.split('_')[:2]),
)

# Cost analysis
cost_rows = extract_cost_rows(
    log_dict={'Model A': 'model_a.eval'},
    log_dir='latest_experiments/excytin',
    pricing={'anthropic/claude-sonnet-4-6': {'input': 3, 'output': 15, 'cache_read': 0.3, 'cache_write': 3.75}},
    n_samples=599,
)