Imported from TechNickAI/heartcentered.ai (model-benchmarks/AGENTS.md). Install upstream with npx skills add TechNickAI/heartcentered.ai --skill model-benchmarks. Copyright stays with the author.

Model Benchmarks — Project Context

Comparative LLM benchmark page at heartcentered.ai/model-benchmarks/ emphasizing emotional intelligence (EQ-Bench) as the differentiating metric alongside reasoning, coding, agentic, speed, and cost.

Architecture

Static HTML/CSS/JS page — no build step, no framework.

• index.html — page structure, Tailwind CDN, Alpine.js (pinned @3.14.9), AOS
• js/app.js — vanilla JS IIFE: fetch data, sort, filter, search, render table + mobile cards
• styles.css — extends Organic Flow Design System CSS vars
• data/model-data.json — all model data (scores, benchmarks, pricing, traits)
• data/eqbench-raw.json — raw EQ-Bench scrape data (reference only)
• scripts/fetch-model.py — data pipeline: fetch, enrich, merge, generate llms.txt
• llms.txt — plain-text model summary for LLM consumption (auto-generated)

Data Pipeline

Python environment

Python 3.13 via uv. The repo pins it in .python-version; create the venv once with uv venv --python 3.13 from the repo root, then prefix every command with uv run --no-project python. Do NOT use bare python3 — on macOS that resolves to Apple's 3.9 and the scripts fail on str | None syntax. Stdlib only, no deps to install.

Adding a new model

1. Fetch from OpenRouter — uv run --no-project python scripts/fetch-model.py <openrouter-id>

   • Gets pricing, context window, capabilities, endpoint stats
   • Optionally enriches with Artificial Analysis (needs AA_API_KEY in env.local)

2. Add PinchBench scores — manually from pinchbench.com

   • Add to PINCHBENCH_DATA dict in fetch-model.py
   • Format: {"best_score": 88.0, "avg_score": 81.1, "runs": 19}

3. Add Arena Elo — manually from arena.ai/leaderboard/text

   • Add to model's benchmarks.arena in model-data.json
   • Format: {"elo": 1462}

4. Add EQ-Bench data — two sources:

   • Elo + legacy traits: scrape from eqbench.com leaderboard (Playwright)
   • v3 Score + v3 traits: run EQ-Bench v3 benchmark directly (~$6/model, 10-20 min)
   • v3 Score is the primary display value (0-100, matches other columns)
   • Elo and traits show in tooltip on hover

5. Regenerate llms.txt — uv run --no-project python -c "import sys; sys.path.insert(0, 'model-benchmarks/scripts'); fm = __import__('importlib').import_module('fetch-model'); data = fm.load_model_data(); fm.generate_llms_txt(data)"

6. Update Model Personalities — the "Model Personalities" section in index.html has hardcoded editorial insight cards (9 curated profiles with personality reads and trait chips). When adding/removing models or re-running EQ-Bench, review whether the cards need updating. Trait numbers and editorial claims should match current data.

Refreshing existing models

uv run --no-project python scripts/fetch-model.py --refresh        # all models
uv run --no-project python scripts/fetch-model.py --refresh --no-aa # skip AA (rate limited at 1000/day)

merge_model() preserves manually-entered data: eq_bench, arena, pinchbench, scores, notes, speed. Only OpenRouter-sourced fields get overwritten.

Artificial Analysis merges field-by-field rather than wholesale: AA is only fetched when AA_API_KEY is present, so a keyless refresh yields an all-null block. Existing scores survive it; only a real (non-null) incoming value overwrites a stored one. This makes --refresh safe to run without the key.

Models whose OpenRouter id was renamed are remapped via OPENROUTER_ID_ALIASES while keeping their original dataset id, so PinchBench/EQ-Bench keys stay attached. Delisted models keep their last-known data and are flagged "delisted": true with a note from MODEL_NOTES.

Tests

uv run --no-project python -m unittest discover -s model-benchmarks/tests

Stdlib unittest, no dependencies. Covers the merge guarantees above.

Score Methodology

• Reasoning (0-100): Weighted avg of AA Intelligence Index (3x), GPQA (2.5x), MMLU-Pro (2x), HLE (1.5x), AIME 2025 (1x)
• Coding (0-100): Weighted avg of AA Coding Index (3x), LiveCodeBench (2x), TerminalBench Hard (2x), SciCode (1x)
• Agentic (0-100): Weighted avg of PinchBench Best (4x), IFBench (3x), PinchBench Avg (2x)
• EQ (0-100): EQ-Bench v3 score. Tooltip shows Elo ranking + trait breakdown
• Chat (Elo): Arena Elo from blind human A/B preference tests
• Cost: Blended per 1M tokens = (3 * input + output) / 4

EQ-Bench Data Model

Three distinct metrics from EQ-Bench, stored in benchmarks.eq_bench:

• v3_score (0-100) — absolute score from running v3 benchmark. This is what the column displays.
• elo (~856-1877) — relative ranking from pairwise comparisons on the public leaderboard
• v3_traits (22 dimensions, each 0-20) — detailed personality breakdown

v3 Score and Elo are separate measurements. High v3 score doesn't guarantee high Elo and vice versa (e.g., Grok 4.20: v3=68.55 but Elo=856).

Trait dimensions: analytical, boundary_setting, challenging, compliant, conversational, correctness, demonstrated_empathy, depth_of_insight, emotional_reasoning, humanlike, intellectual_grounding, message_tailoring, moralising, pragmatic_ei, reactive, safety_conscious, social_dexterity, subtext_identification, sycophantic, theory_of_mind, validating, warmth.

Some traits are "negative" (lower is better): moralising, sycophantic, compliant, reactive.

Data Sources

JS Architecture

• esc() — XSS protection on all innerHTML interpolations
• getSortValue() — null sentinels (-1 or 9999) sort missing data to bottom
• scoreHtml() — renders score with color tier (green >= 70, amber >= 45, muted < 45)
• eqHtml() — renders v3 score with Elo + traits tooltip
• scoreTier() — shared color tier function (also used for mobile EQ cards)
• Default sort: eq_score descending (EQ is the headline feature)

Skills

Use /model-research when adding or enriching models. The skill at .claude/skills/model-research/SKILL.md documents the complete workflow across all 5 data sources, including API details, Playwright scraping patterns for EQ-Bench, and common pitfalls (AA slug mismatches, rate limits, sparse data for newer models).

Key Conventions

• All model links point to https://openrouter.ai/models/${id}
• rel="noopener noreferrer" on all external links
• No build step — Tailwind CDN, inline everything
• Accessibility is not a priority (decline ARIA suggestions)
• Always regenerate llms.txt after changing model-data.json
• Models in model-data.json are sorted alphabetically by name — maintain this order when adding new models to minimize merge conflicts

Model Curation

Not comprehensive — chosen to represent the frontier across different providers, price points, and capability profiles. Check model-data.json for the current set.

Models in CURATED_MODELS in the fetch script are refreshed via --curated. All others were added manually and should be refreshed via --refresh.