Name: benchmark
Author: bojiang

Imported from bojiang/.dotfile (skills/benchmark/SKILL.md). Install upstream with npx skills add bojiang/.dotfile --skill benchmark. Copyright stays with the author.

Benchmark

All measurement through scripts. No hand-operated steps. Every run reproducible by anyone.

Structure

benchmarks/
  001_<objective>/
    GOAL.md
    round-001-yyyy-mm-dd-HH-MM/
      ROUND.md
      1_collect_script/
        run.sh
        scratch_output/
      2_collect_output/
        run-001-yyyy-mm-dd-HH-MM/
          invoke.sh            # sh invoke.sh reproduces this exact run
          ...raw data...
        run-002-.../
          invoke.sh
          ...
      3_analysis_script/
        analyze.sh
        scratch_output/
      4_analysis_output/
        run-001-yyyy-mm-dd-HH-MM/
          invoke.sh
          ...report...

Rules

Frozen = directory renamed with .frozen suffix. Output dirs require their script dir frozen first.
Each round is self-contained. Want to change anything frozen → new round (copy scripts, regenerate outputs).
Same collect data, different analysis → new round with 2_collect_output symlinked to the source round's.
Raw data: per-record, never pre-aggregated. Capture more than the report needs, but not so much it drags the experiment.
invoke.sh uses relative paths only. No absolute paths, no environment variables.

# after Install: the listing is in your workspace's routing pool - a plan picks it for its slot curl -s -X POST https://api.opensmartroute.ai/api/v1/route -H 'Authorization: Bearer $OSR_API_KEY' -H 'Content-Type: application/json' -d '{"text": "...", "plan": true}'

{ "ocm": "1", "id": "bojiang-dotfile-benchmark", "kind": "skill", "name": "benchmark", "description": "Reproducible benchmark/experiment pipeline. Use when testing, benchmarking, or evaluating anything that must be reproducible across sessions and people. Enforces script-only execution (no hand-operated steps), frozen pipeline stages, and full raw data retention.", "publisher": "bojiang", "version": "1.0.0", "capabilities": { "domains": [ "general" ], "tags": [ "skill-md", "github" ], "languages": [ "en" ] }, "quality_prior": 0.6, "examples": [ "Reproducible benchmark/experiment pipeline. Use when testing, benchmarking, or evaluating anything that must be reproducible across sessions and people. Enforces script-only execution (no hand-operated steps), frozen pipeline stages, and full raw data retention." ], "primary": false, "metadata": { "source": { "provider": "github", "repository": "https://github.com/bojiang/.dotfile", "path": "skills/benchmark/SKILL.md", "ref": "a7354771dfbf83801f16a3854be3ca517439362f", "url": "https://github.com/bojiang/.dotfile/blob/a7354771dfbf83801f16a3854be3ca517439362f/skills/benchmark/SKILL.md", "key": "bojiang/.dotfile/skills/benchmark/SKILL.md" } }, "instructions": "# Benchmark\n\nAll measurement through scripts. No hand-operated steps. Every run reproducible by anyone.\n\n## Structure\n\n```\nbenchmarks/\n 001_<objective>/\n GOAL.md\n round-001-yyyy-mm-dd-HH-MM/\n ROUND.md\n 1_collect_script/\n run.sh\n scratch_output/\n 2_collect_output/\n run-001-yyyy-mm-dd-HH-MM/\n invoke.sh # sh invoke.sh reproduces this exact run\n ...raw data...\n run-002-.../\n invoke.sh\n ...\n 3_analysis_script/\n analyze.sh\n scratch_output/\n 4_analysis_output/\n run-001-yyyy-", "cost": { "context_tokens": 300 } }

benchmark

About

Benchmark

Structure

Rules

Use it

Manifest

Reviews