Imported from Mercity-AI/Simula (SKILL.md). Install upstream with npx skills add Mercity-AI/Simula. Copyright stays with the author.

Generate a dataset with simula

You are driving simula for a user who wants a dataset, not a config tutorial. This skill encodes the judgment so you can do almost all of it yourself. Read AGENTS.md once for the architecture and hard constraints; this file is the process and decision layer on top of it.

Prime directive

Ask the user only two things up front:

1. What dataset — a plain-English description of what each row should look like and what it's for (the downstream use). Push for the use — "extraction for a model that must emit a fixed schema" implies different choices than "examples for a demo".
2. How much — target row count.

Everything else you decide yourself from the defaults below, and you pull the user back in only at the explicit decision gates in the next section. Do not walk them through config keys, taxonomy depth, concurrency, or model knobs — those are yours unless a gate says otherwise. State the handful of consequential choices you made in one short line ("depth-2 taxonomy, omit-absent fields, fake-model smoke first") so they can veto, but don't ask permission for each.

When to bring the user in (decision gates)

Only these. If a choice isn't here, pick the default and move on.

Everything else — taxonomy depth, factors/axes, best_of_n, children_per_node, overgenerate_ratio, complexity_ratio, max_refine_attempts, concurrency, model-role assignment, whether to write a prompt module — you decide. Surface, don't ask.

Workflow

Use the CLI from the repo root. Commands: validate, taxonomy, generate, evaluate, run. examples/template.yaml is the copy-me skeleton; CONFIG.md is the field reference; don't re-derive defaults — copy the template and change only what the decisions below dictate.

1. Classify + draft. From the description, pick the archetype (table below), infer the schema shape, and choose the variation axes. Write examples/<name>.yaml from template.yaml. If the task needs atomic fields / an absent-field policy / a varying schema, also write examples/<name>_prompts.py (model after examples/job_extraction_prompts.py or examples/ecommerce_search_extraction_prompts.py).
2. Validate. python -m simula.cli validate examples/<name>.yaml — catches config/schema/ prompt-module errors with zero model calls. Always do this before anything real.
3. Smoke test, free. Set every model role to "fake" and run python -m simula.cli run examples/<name>.yaml. This is offline and deterministic — it proves the pipeline, schema, lineage, and prompt module work end to end before spending a cent. Inspect the artifacts in runs/<name>/ (especially row shape and taxonomy_mix).
4. Pilot, real (gate: spend + key). Switch roles to real models, set generation.target_size to ~20–50, and run taxonomy then generate as two steps so you (and optionally the user) can eyeball taxonomy.json and strategies.json before paying for bulk generation — see "The strategy phase" below for what to actually look for in them. Inspect dataset.final.jsonl and llm_calls.jsonl for quality: atomic fields? faithful? right axes covered? Read the accept rate.
5. Scale (gate: confirm). Raise target_size to the real number and run. Resume is on by default. Then evaluate for dedupe/coverage/diversity. Report cost from cost_summary.json.

Archetype → natural variation axes

The taxonomy is your coverage. Seed taxonomy.factors with the axes that actually matter for the domain (or let discovery run and review them). The high-leverage move is choosing the right axes — this is where domain reasoning earns its keep.

Example of the judgment expected: for job-description extraction, posting length is a major axis (a one-line listing vs a 600-word posting extract very differently) — so it belongs in the factors, not left to chance. Reason about the domain like this every time; don't ship a generic taxonomy when the domain has an obvious dominant axis.

The strategy phase — read the plan before you pay

Pay close attention to how the work is being planned during the design phase. taxonomy.json (with its per-node weights) and strategies.json (roots, weights, never_combine) together decide the dataset's entire distribution — what "typical" looks like across thousands of rows — before a single record is generated. They are drafted by a model, they are plausible-looking by construction, and every judgment error in them gets executed faithfully at full scale. Ninety seconds reading two JSON files is the cheapest QA gate in the pipeline; treat it as mandatory before any paid generation, not optional.

Cautionary example — a 2k short-story corpus that came out wrong. A free-text story dataset was generated with a taxonomy whose pov factor listed second-person, epistolary, documentary-archive, and frame/mixed-form narration as sibling branches of ordinary first/third person. Every individual piece was defensible: the branches were real narrative forms, the strategy bundles were thematically coherent, and no single prompt or config line was obviously wrong. The corpus still came out with ~57% of stories in a high-artifice form (21.9% second person alone), because several small, individually-reasonable choices compounded: the builder subdivided the interesting branches most finely (second person got 10 subtypes, workhorse close-third got 5) and the sampler of the time turned that granularity directly into probability; one strategy bundled all the experimental forms together, so every record it produced was unusual in several ways at once; and two factors (pov and structure) both contained document/frame ideas, so mixes could demand the same gimmick twice or contradict themselves. The result read like a corpus of narrative tricks, not stories.

Be nuanced about the lesson. It is not "rare forms are bad" — a story dataset should contain epistolary and second-person stories, and coverage of the space is a real goal. The failure was that nothing in the plan encoded how often and in what combinations: enumeration granularity silently became prevalence, thematic affinity ("the weird stuff goes together") silently became concentration, and per-factor independence silently became constraint-stacking. When you review the design artifacts, you are checking exactly those three translations:

• Weights vs. reality. For each factor, do the branch weights in taxonomy.json reflect how common each branch should be in realistic data, not how interesting it was to enumerate? Rare forms should sit an order of magnitude below common ones (~0.03 vs ~1.0). A factor whose weights are all ~1.0 despite obviously unequal real-world prevalence is a red flag.
• Bundles vs. concentration. Read each strategy and ask what a record sampled from it actually looks like. A strategy whose roots for some factor are only unusual branches makes them mandatory, not occasional — rare things should be diluted across strategies, never concentrated into one. Check every strategy covers every factor (bare factor name = fine).
• Combinations vs. coherence. Look for factor overlap (two factors owning the same idea) and for pairs that are individually fine but jointly incoherent or impossible at the record's scale — that's what never_combine is for. A missing rule ships silently; sanity-check the model's rules and add your own.

All of it is hand-editable: fix weights in taxonomy.json, rebalance or split bundles and add never_combine pairs in strategies.json, then rerun — both files are reused verbatim. Use --stop-after strategies (or generation.stop_after) to halt there deliberately. If the mix distribution matters a lot, sample it for free: sample_mix is pure RNG, so a few thousand draws in a scratch script show the marginals before any money moves.

Representation decisions (the high-leverage, easy-to-miss part)

Beyond schema shape and coverage axes, every dataset has subtle representation choices — how each row encodes information — that don't show up in a naive reading of the description but strongly shape what a model learns from the data. This is where datasets are quietly won or lost. For every task, actively hunt for these; don't take the user's description at face value. Each one falls into one of two buckets, and the bucket tells you whether to ask:

• Use-dependent → ASK (with a recommendation). The right answer depends on what the user is training and how they'll use the output, so you can't safely default it. Surface one plain question. Recurring forms across archetypes: how to represent absent/missing information; label granularity (coarse vs fine classes); whether to include negative / unanswerable / hard cases and at what rate; class balance; output length/verbosity; canonical vs surface forms of values.
• Universal quality → DECIDE and ENFORCE (don't ask). Almost always right for training quality, so bake them in rather than asking. Recurring forms: atomic / narrow / normalized fields; consistent formatting (dates, units, enums); faithful, non-hallucinated content; varied, deduped rows. Enforce these in the schema and the prompt module (the loose envelope alone can't express them) and have the critic reject violations.

Where you land a choice, push it into the schema, the prompt module, and the taxonomy — e.g. if a representation choice introduces a case (absent values, hard negatives), make that case an explicit taxonomy axis so it actually appears in the data at a controlled rate.

Concrete illustration — extraction (text → JSON). The same two buckets, made specific:

1. Absent-field policy (use-dependent → ASK). When a requested field's value isn't in the input text, emit the key with null/empty, or omit it? Recommend present-but-null when the downstream model must reliably produce a fixed schema — it teaches the model to acknowledge absence instead of hallucinating or silently dropping fields (then make "some fields absent" a taxonomy axis and mark those fields non-required/nullable). Use omit-when-absent for per-row/varying schemas (what job_extraction and ecommerce_search_extraction do). Ask, because it depends on their model.
2. Atomic, narrow fields (universal → ENFORCE). Good: {"salary_min": 90000, "salary_max": 120000, "salary_currency": "USD"}, {"location_city": "Berlin", "location_country": "DE"}, dates as YYYY-MM-DD. Bad: {"salary_range": "90k–120k/yr"}, {"location": "Berlin, Germany"}, free-form dates. Give the model a menu of good atomic fields plus explicit anti-examples in the prompt module (see examples/job_extraction_prompts.py) and make the critic reject compound junk.

Reason about your task's representation choices the same way every time — the extraction pair is just the worked example, not the whole list.

Defaults you apply silently

Copy from template.yaml; change only these per the task. Don't ask about any of them.

• Taxonomy depth. depth: 2 is the default and right for most datasets. Use 1 for a narrow, single-axis dataset; 3 only for genuinely broad/heterogeneous domains where 2 levels can't span the space. Deeper = more taxonomy-build calls and time, so don't reach for 3 reflexively.
• review_mode: auto_accept for smoke/pilot. Switch to write_then_edit only if the user wants to hand-edit the taxonomy before generation.
• Models. Stronger model for strategic, cheaper/faster for bulk and critic (e.g. a "pro" strategic + a "flash" bulk/critic on the same OpenRouter-compatible provider). "fake" for smoke tests. For reasoning models, set extra_body: {reasoning: {effort: low, exclude: true}} per role — there's no auto-detection.
• overgenerate_ratio ~1.2, raise toward 1.5–2 if pilot accept rate is low.
• concurrency ~4; lower it if the provider rate-limits, raise cautiously for big runs.
• complexity_ratio / max_refine_attempts — leave at template defaults unless quality demands more refinement.

Custom prompts (a Python prompt module)

YAML + schema control what shape to produce; the prompt module controls how the model is asked, and it's how you enforce the representation decisions above (atomic fields, absent-field policy, per-row varying schema, a stricter critic, a specific free-text style). Reach for one when the config can't express the task's rules. Simple datasets don't need one — don't add it reflexively.

How it works:

1. Write a .py file next to the config (convention: examples/<name>_prompts.py) and point the config at it. The path resolves relative to the YAML file:

   prompts:
     module: "<name>_prompts.py"

2. The module overrides any subset of the built-in prompt functions in simula/prompts.py, plus the SYSTEM_JSON / SYSTEM_TEXT system strings. Anything you don't define falls back to the built-in — so override only what you need.
3. Each override must keep the same parameter names as its built-in counterpart (signature- compatible). simula validate imports the module and rejects a missing file, an import error, a non-string system prompt, or an incompatible signature — always validate before generating.

The high-leverage functions to override (see simula/prompts.py for the full list + exact signatures — don't guess them):

• meta_prompt_prompt(...) — the main lever. It plans each example (context, the concrete target schema for that row, which fields are present/absent). Most task-specific rules go here.
• critique_prompt(...) (JSON) / critique_text_prompt(...) (free-text) — make the critic actively reject your anti-patterns (compound fields, hallucinated values, wrong absent-field policy).
• generate_record_prompt(...), strategy_prompt(...), the taxonomy prompts (factor_prompt, expand_prompt, level_plan_prompt, …) — override only if the defaults fight your task.

Minimal skeleton:

# examples/my_task_prompts.py — only the param NAMES must match simula/prompts.py
SYSTEM_JSON = "Return one valid JSON object. No prose, no code fences."

def critique_prompt(description, schema, meta_prompt, record):
    return f"Reject the record if any field is compound or not stated in the source. Record:\n{record}"

Copy examples/job_extraction_prompts.py (atomic fields + omit-when-absent + strict critic) or examples/ecommerce_search_extraction_prompts.py (per-row varying schema) as a working starting point rather than writing from scratch.

Guardrails (never violate)

• Smoke with "fake" before any real call. It's free and offline; it catches schema and prompt-module bugs you'd otherwise pay to discover.
• Review taxonomy.json and strategies.json before any paid generation (see "The strategy phase"). The design artifacts are the dataset's distribution; a plausible-looking bad plan executes faithfully at full scale.
• No real model calls without explicit user go-ahead (spend gate) and a key in root .env. Never put a real API key in a YAML — only api_key_env.
• Don't commit runs/, .env, or llm_calls.jsonl (prompts/responses can be sensitive).
• Check it actually worked. After a real run, read the accept rate — simula can finish below target silently if the accept rate is low. If len(final) < target, say so and either raise overgenerate_ratio or investigate rejections in dataset.raw.jsonl. Confirm taxonomy_mix is non-empty on rows (lineage) and that fields are atomic.
• Report honestly. Give real counts and real cost from cost_summary.json; if quality is off, show the offending rows from llm_calls.jsonl rather than declaring success.

Pointers (don't duplicate these here)

• Config mechanics + every field: CONFIG.md, examples/template.yaml.

• Concepts, true pipeline flow, the three model roles, config reference (with diagrams): DOCS.md.

• Architecture, hard constraints, behaviors, debugging: AGENTS.md.

• examples/ — copy the closest one to start. Pick by task and adapt rather than writing config from a blank file:

  Example	Copy it when the task is…
  basic_qa.yaml	a quick fake-model smoke test / simplest schema run
  template.yaml	you want every knob visible at its default to fill in
  cat_stories_freetext.yaml	free-text generation (schema: null), no JSON
  query_extraction_gemini.yaml	NL query → structured JSON, fixed schema
  job_extraction.yaml + job_extraction_prompts.py	text → atomic JSON, omit-when-absent, strict critic (the prompt-module reference)
  ecommerce_search_extraction.yaml + ecommerce_search_extraction_prompts.py	extraction with a per-row varying schema

  The *_prompts.py files are the worked references for the custom-prompts section above.