Imported from niekbeckers/skills (skills/skill-creator/SKILL.md). Install upstream with npx skills add niekbeckers/skills --skill skill-creator. Copyright stays with the author.

Skill Creator

A skill for creating new skills and iteratively improving them through evaluation and benchmarking.

Overview

The skill creation process follows this loop:

1. Capture Intent — Understand what the skill should do
2. Interview & Research — Gather requirements, edge cases, success criteria
3. Write SKILL.md — Draft the skill
4. Create Test Cases — Write realistic user prompts
5. Run Evaluations — Execute with/without skill, grade results
6. Human Review — Show outputs in browser viewer for feedback
7. Iterate — Improve based on feedback and metrics
8. Optimize Description — Fine-tune triggering accuracy
9. Package — Create installable skill

Your job is to figure out where the user is in this process and help them progress. Maybe they want to create a skill from scratch. Maybe they already have a draft and want to test it. Maybe they want to optimize an existing skill's triggering. Be flexible — if the user says "just vibe with me", skip the formal evaluation.

Creating a Skill

Capture Intent

Start by understanding what the user wants. The conversation may already contain a workflow to capture (e.g., "turn this into a skill"). If so, extract:

• Tools used and their sequence
• Corrections the user made
• Input/output formats observed

Ask these questions (skip any already answered):

1. What should this skill enable Claude to do?
2. When should this skill trigger? (user phrases, contexts)
3. What's the expected output format?
4. Should we set up test cases? (Skills with objectively verifiable outputs benefit from tests. Subjective skills like writing style often don't need them.)

Interview & Research

Proactively ask about:

• Edge cases and error handling
• Input/output formats and examples
• Success criteria (what makes output "good"?)
• Dependencies (tools, APIs, file types)

Check available MCPs for useful research tools. Come prepared with context to reduce burden on the user.

Write the SKILL.md

Based on the interview, create the skill with these components:

Frontmatter (required):

---
name: skill-identifier
description: When to trigger and what it does. Include specific trigger phrases. Be slightly "pushy" — Claude tends to under-trigger skills, so make descriptions proactive.
---

Body sections:

• Clear instructions in imperative form
• Output format templates (if applicable)
• Examples with input/output pairs
• Error handling guidance
• References to bundled resources

Skill Writing Guide

Anatomy of a Skill

skill-name/
├── SKILL.md (required)
│   ├── YAML frontmatter (name, description)
│   └── Markdown instructions
└── Bundled Resources (optional)
    ├── scripts/    — Executable code for repetitive tasks
    ├── references/ — Docs loaded into context as needed
    └── assets/     — Templates, icons, fonts

Progressive Disclosure

Skills use a three-level loading system:

1. Metadata (name + description) — Always in context (~100 words)
2. SKILL.md body — Loaded when skill triggers (<500 lines ideal)
3. Bundled resources — Read as needed (unlimited size)

Keep SKILL.md under 500 lines. For large reference material, put it in references/ with clear pointers about when to read each file.

Writing Patterns

Use imperative form:

## Process
1. Read the input file
2. Extract the relevant sections
3. Format according to the template

Define output formats explicitly:

## Report Structure
ALWAYS use this template:
# [Title]
## Executive Summary
## Key Findings
## Recommendations

Include examples:

## Commit Message Format
**Example 1:**
Input: Added user authentication with JWT tokens
Output: feat(auth): implement JWT-based authentication

Style Guidance

• Explain the why, not just the what — models respond better to reasoning than rigid rules
• Avoid heavy-handed MUSTs and NEVERs — reframe as explanations
• Keep it general, not narrow to specific examples
• Write a draft, then review with fresh eyes and improve

Test Cases

After writing the skill draft, create 2-3 realistic test prompts — things a real user would actually say.

Writing Realistic Prompts

Good test prompts are:

• Concrete: Include file paths, names, specific details
• Natural: Casual language, typos, abbreviations are OK
• Varied: Different lengths, phrasings, edge cases

Bad: "Format this data" Good: "ok so my boss sent me this xlsx (in downloads, called 'Q4 sales final FINAL v2.xlsx') and wants profit margin as a percentage. Revenue is column C, costs column D i think"

evals.json Structure

Save test cases to evals/evals.json in the skill directory:

{
  "skill_name": "example-skill",
  "evals": [
    {
      "id": 1,
      "prompt": "User's task prompt",
      "expected_output": "Description of expected result",
      "files": ["evals/files/sample1.pdf"],
      "expectations": [
        "Output includes X",
        "Used script Y",
        "No errors encountered"
      ]
    }
  ]
}

Don't write assertions until after the first run — you'll draft them while runs execute.

See references/schemas.md for the full schema.

Running Evaluations

This section is one continuous sequence — don't stop partway through.

Results go in <skill-name>-workspace/ as a sibling to the skill directory. Organize by iteration (iteration-1/, iteration-2/, etc.) with each test case in its own directory (eval-1/, eval-2/, etc.).

Step 1: Spawn All Runs in Parallel

For each test case, spawn two subagents in the same turn — one with the skill, one without. Don't do with-skill first then baselines later; launch everything at once.

With-skill run:

Execute this task:
- Skill path: <path-to-skill>
- Task: <eval prompt>
- Input files: <files or "none">
- Save outputs to: <workspace>/iteration-N/eval-ID/with_skill/outputs/

Baseline run (depends on context):

• New skill: No skill at all. Same prompt, save to without_skill/outputs/
• Improving existing skill: Use the old version. Snapshot first (cp -r <skill> <workspace>/skill-snapshot/), point baseline at snapshot, save to old_skill/outputs/

Create eval_metadata.json for each test case:

{
  "eval_id": 1,
  "eval_name": "descriptive-name-here",
  "prompt": "The user's task prompt",
  "assertions": []
}

Step 2: Draft Assertions While Runs Execute

Don't wait idle — use this time to draft quantitative assertions.

Good assertions are:

• Objectively verifiable — can be checked programmatically
• Descriptively named — clear at a glance what they test
• Discriminating — pass when skill succeeds, fail when it doesn't

For subjective skills (writing style, design), rely on qualitative human review instead.

Update eval_metadata.json and evals/evals.json with assertions.

Step 3: Capture Timing Data

When each subagent completes, you receive total_tokens and duration_ms in the notification. Save immediately to timing.json:

{
  "total_tokens": 84852,
  "duration_ms": 23332,
  "total_duration_seconds": 23.3
}

This data is only available in the completion notification — capture it as runs finish.

Step 4: Grade, Aggregate, Launch Viewer

Once all runs complete:

1. Grade each run — Spawn a grader subagent (read agents/grader.md) to evaluate assertions. Save to grading.json. For programmatically checkable assertions, write and run a script instead of eyeballing.

2. Aggregate into benchmark — Run from the skill-creator directory:

   python scripts/aggregate_benchmark.py <workspace>/iteration-N --skill-name <name>

   Produces benchmark.json and benchmark.md.

3. Analyst pass — Read benchmark data and surface patterns the stats might hide. See agents/analyzer.md for what to look for.

4. Launch the viewer:

   python <skill-creator-path>/eval-viewer/generate_review.py <workspace>/iteration-N \
     --skill-name "my-skill" \
     --benchmark <workspace>/iteration-N/benchmark.json

   For iteration 2+, add --previous-workspace <workspace>/iteration-N-1>.

   Headless environments: Use --static <output.html> to write standalone HTML instead of starting a server.

5. Tell the user: "I've opened the results in your browser. The 'Outputs' tab shows each test case for qualitative review. The 'Benchmark' tab shows quantitative comparison. When done, come back and let me know."

Step 5: Read Feedback

When the user is done, read feedback.json from the workspace:

{
  "reviews": [
    {"run_id": "eval-1-with_skill", "feedback": "chart missing axis labels", "timestamp": "..."},
    {"run_id": "eval-2-with_skill", "feedback": "", "timestamp": "..."}
  ],
  "status": "complete"
}

Empty feedback means the output was fine. Focus improvements on test cases with specific complaints.

Kill the viewer server when done:

kill $VIEWER_PID 2>/dev/null

Improving the Skill

This is the heart of the loop. You've run tests, the user has reviewed results, now make the skill better.

How to Think About Improvements

1. Generalize from feedback. You're iterating on a few examples, but the skill will be used many times on different prompts. Don't overfit — if there's a stubborn issue, try different approaches rather than adding fiddly constraints.

2. Keep the prompt lean. Remove things that aren't pulling their weight. Read transcripts, not just outputs — if the skill wastes time on unproductive steps, cut those instructions.

3. Explain the why. Models are smart. They respond better to reasoning than rigid rules. If you're writing ALWAYS or NEVER in caps, reframe as an explanation of why it matters.

4. Look for repeated work. If all test runs independently wrote similar helper scripts, bundle that script in scripts/ and tell the skill to use it.

The Iteration Loop

After improving:

1. Apply changes to the skill
2. Rerun all test cases into iteration-N+1/, including baselines
3. Launch viewer with --previous-workspace pointing to previous iteration
4. Wait for user review
5. Read feedback, improve again, repeat

Keep going until:

• User says they're happy
• Feedback is all empty
• You're not making meaningful progress

Description Optimization

The description field is the primary trigger mechanism. After the skill is working well, optimize it for triggering accuracy.

Step 1: Generate Trigger Eval Queries

Create 20 queries — mix of should-trigger and should-not-trigger:

[
  {"query": "realistic user prompt with details", "should_trigger": true},
  {"query": "similar but different domain prompt", "should_trigger": false}
]

Should-trigger queries (8-10):

• Different phrasings of the same intent
• Cases where user doesn't name the skill but clearly needs it
• Uncommon use cases

Should-not-trigger queries (8-10):

• Near-misses that share keywords but need something different
• Adjacent domains
• Ambiguous phrasing where naive keyword match would wrongly trigger

Avoid obviously irrelevant negatives — they don't test anything useful.

Step 2: Review with User

Present eval queries using the HTML template:

1. Read assets/eval_review.html
2. Replace placeholders with data
3. Write to temp file and open
4. User edits queries, exports as JSON

Step 3: Run Optimization Loop

python scripts/run_loop.py \
  --eval-set <path-to-trigger-eval.json> \
  --skill-path <path-to-skill> \
  --model <model-id-powering-this-session> \
  --max-iterations 5 \
  --verbose

Use the model ID from the current session so the triggering test matches what the user actually experiences.

This splits evals into train/test, evaluates triggering, proposes improvements, iterates.

Step 4: Apply Result

Take best_description from output, update SKILL.md frontmatter, show user before/after with scores.

Environment Detection

The skill and its scripts auto-detect which environment they're running in:

Detection order:

1. agent binary in PATH → use Cursor CLI
2. OPENCODE=1 env var → use OpenCode CLI
3. CLAUDECODE=1 env var → use Claude Code CLI
4. Fall back to checking which CLI is available (claude or opencode)

What changes:

• Subagent spawning: Same Task tool works in all environments
• CLI for description optimization: agent -p vs claude -p vs opencode run
• JSON output parsing: Claude Code uses stream-json; Cursor uses tool_call events; OpenCode uses json
• Skill paths: Cursor uses ~/.cursor/skills/; OpenCode uses ~/.claude/skills/; Claude Code uses .claude/commands/

You don't need to configure anything — the scripts adapt automatically.

Environment-Specific Reference

Claude Code

# Non-interactive execution
claude -p "<query>" --output-format stream-json

# With model override
claude -p "<query>" --model claude-sonnet-4-20250514

OpenCode

# Non-interactive execution
opencode run "<query>" --format json

# With model override
opencode run "<query>" --model github-copilot/claude-sonnet-4

Cursor

# Non-interactive execution
agent -p "<query>" --output-format stream-json

# With model override
agent -p "<query>" --model gpt-5

Subagent Spawning (All Environments)

All environments use the Task tool for subagents:

Task(
    description="Run eval with skill",
    prompt="Execute task with skill at <path>: <prompt>",
    subagent_type="general"
)

Browser viewer works via open (macOS) or xdg-open (Linux).

Reference Files

Agents

Read these when spawning the relevant subagent:

• agents/grader.md — Evaluate assertions against outputs
• agents/analyzer.md — Analyze benchmark results, surface patterns
• agents/comparator.md — Blind A/B comparison (advanced)

Schemas

• references/schemas.md — JSON structures for all data files

Quick Reference

Remember: Always generate the eval viewer before revising the skill yourself. Get outputs in front of the human ASAP.