Imported from reeshogue22/NOSCALE (AGENTS.md). Install upstream with npx skills add reeshogue22/NOSCALE. Copyright stays with the author.

AGENTS.md - NOSCALE Project Playbook

This file defines project-specific operating rules for AI coding/research agents working in Projects/NOSCALE.

It is intentionally detailed. Treat it as normative unless the user gives an explicit override.

1) Project Identity

• Project name: NOSCALE
• Core question: Why does scaling work in language models, and what can replace brute-force scaling?
• Current working theory: CRCR (Constraint Relief and Circuit Reuse), documented in docs/theory_v1.md.
• Research style: falsifiable, experiment-first, reproducible, compute-aware.

2) High-Level Goal

Convert broad claims about scaling into:

1. Explicit hypotheses
2. Controlled experiments
3. Quantitative results
4. Updated theory with clear pass/fail criteria

A contribution is only considered progress if it produces a testable prediction or reliable empirical evidence.

3) Non-Goals

• Do not chase SOTA benchmarks in this repo.
• Do not run huge training jobs without user approval.
• Do not add large infra complexity before a toy experiment isolates the mechanism.
• Do not present narrative-only conclusions without data artifacts.

4) Current Repository Contract

Expected top-level structure:

• README.md: project overview and active direction
• AGENTS.md: this operating playbook
• docs/: theory and hypothesis docs
• experiments/: experiment code and per-experiment outputs
• notes/: chronological research notebook
• .venv/: local environment (not research artifact)

Current key files:

• docs/research_questions.md
• docs/theory_map.md
• docs/theory_v1.md
• docs/paper_queue.md
• notes/lab_notebook.md
• experiments/exp001_tiny_scaling/run_exp001.py

5) Scientific Method Loop (Required)

For any substantial experiment request, run the full loop:

1. Hypothesis
2. Operationalization
3. Experimental design
4. Implementation
5. Execution
6. Analysis
7. Conclusion
8. Logging and next-step decision

Do not skip directly from code to conclusions.

5.1 Hypothesis Requirements

A valid hypothesis must be:

• falsifiable
• specific about expected direction
• tied to at least one measurable metric

Example format:

H#: At fixed FLOPs, increasing data diversity improves transfer accuracy more than a moderate parameter increase.

5.2 Operationalization Requirements

Define:

• independent variables
• controlled variables
• dependent variables
• evaluation datasets/tasks
• stopping criteria

Write these into the experiment README before the full run.

5.3 Design Requirements

• Run at least 3 seeds unless user asks for a quick pass.
• Include one smooth metric and one thresholded metric when testing emergence claims.
• Keep one baseline model/config unchanged across sweeps.
• Use fixed evaluation sets when comparing models.

5.4 Implementation Requirements

• Add/modify code under experiments/<exp_name>/.
• Prefer one entrypoint script per experiment, e.g. run_expXXX.py.
• Ensure deterministic seeds where possible.
• Emit machine-readable outputs (.csv minimum).
• Auto-generate a report markdown file from run outputs.

5.5 Execution Requirements

• Run a smoke test before full sweep.
• Then run full configured sweep.
• Preserve logs and outputs in the experiment results directory.
• If interrupted, verify partial state before continuing.

5.6 Analysis Requirements

Minimum required reporting:

• per-run metrics
• aggregate mean/std by condition
• confidence intervals for key metrics
• hypothesis verdict: Supported, Partially supported, or Inconclusive

Avoid binary claims if evidence is mixed.

5.7 Conclusion Requirements

Conclusions must include:

• what was tested
• what was observed
• what remains uncertain
• what experiment should run next

5.8 Logging Requirements

After each full run:

• update notes/lab_notebook.md with date and summary
• list produced artifacts by path
• state final hypothesis verdict and caveats

5.9 Exploit Loop (Required for Discovery Work)

When the goal is rapid leverage discovery (not immediate mechanism proof), run:

1. Explore: one short run per intervention arm.
2. Exploit: allocate more short runs to top-k arms by utility.
3. Confirm: longer multi-seed run of winner vs scale baseline.

Rules:

• Use an explicit utility metric (quality normalized by compute).
• Keep exploit and explanation loops separate in reporting.
• Never claim causality from exploit-stage wins alone.

6) Experiment Folder Standard

Each experiment folder should contain:

• README.md (method and run instructions)
• run_<exp>.py (main runner)
• results/ (run outputs)

Recommended results/ files:

• <exp>_runs.csv
• <exp>_aggregate.csv
• <exp>_scaling_curves.png (or equivalent)
• scientific_method_report.md

If rerunning with different config, either:

• overwrite intentionally and say so in notebook, or
• create a timestamped subfolder to preserve historical outputs

7) Reproducibility and Determinism

Required practices:

• set explicit random seeds
• pin core package versions when feasible
• record runtime config in outputs or report
• avoid hidden state dependencies

When exact determinism is not feasible, report expected variance and seed count.

8) Compute and Environment Guardrails

This environment is usually CPU-first. Default to CPU-compatible workflows.

8.1 Dependency Policy

• Use uv and project-local .venv.
• Prefer CPU-only PyTorch unless user requests GPU workflows.
• Avoid installing unnecessary heavyweight packages.

8.2 Runtime Policy

• Keep toy runs fast enough for iteration.
• Start with small configs, then scale only if needed.
• If a run is unexpectedly expensive, stop and ask user before continuing.

9) Coding Standards for Research Scripts

• Python 3.11 compatible
• clear function boundaries
• explicit argument parsing
• concise, meaningful logging
• no dead code blocks
• no hidden global mutable state when avoidable

Minimum script features:

• CLI args for steps/batch/seeds/output dir
• predictable output filenames
• safe creation of output directories
• non-zero exit on unrecoverable errors

10) Documentation Standards

10.1 README Updates

Update README.md when:

• a new core theory doc is added
• a major experiment series is introduced
• project direction changes

10.2 Theory Docs

When changing theory, include:

• assumption changes
• new predictions
• explicit failure conditions

Do not silently mutate theory claims.

10.3 Paper Notes

When reading papers, capture:

• central claim
• strongest evidence
• weakest assumption
• one concrete experiment idea for NOSCALE

11) Results Interpretation Rules

• Distinguish statistical trend from practical significance.
• Treat threshold metrics as potentially deceptive for emergence claims.
• Prefer trend comparison across multiple metrics.
• Never claim causality from one toy experiment.
• Explicitly mention limitations every time.

12) Preferred Verdict Vocabulary

Use exactly one:

• Supported
• Partially supported
• Inconclusive
• Refuted

And follow with one sentence explaining why.

12.1 AGI Gate (Mandatory)

Before claiming "AGI progress" in this repo, run:

experiments/gate001_minimal_principles/gate_eval.py

with a metrics JSON containing all required fields from:

docs/agi_minimal_principles_v1.md

Decision policy:

• PASS: valid AGI-progress candidate
• ITERATE: not yet acceptable, requires targeted fixes
• REJECT: does not meet minimal principles

13) Agent Collaboration Protocol

When taking action:

1. State immediate intent briefly.
2. Verify current file/run state before major edits or reruns.
3. Apply minimal necessary changes.
4. Re-run verification/tests.
5. Summarize what changed and where.

If something was interrupted, always inspect current state first.

14) Safety and Change Hygiene

• Do not remove prior results without explicit user instruction.
• Do not rewrite notebook history to hide failed runs.
• Keep failed or inconclusive results visible; they are part of the research record.
• Avoid destructive filesystem commands unless requested.

15) Quickstart Commands

From Projects/NOSCALE:

Create env and install:

uv venv .venv
uv pip install --python .venv/bin/python numpy matplotlib pandas
uv pip install --python .venv/bin/python --index-url https://download.pytorch.org/whl/cpu torch

Run exp001:

.venv/bin/python experiments/exp001_tiny_scaling/run_exp001.py \
  --steps 220 \
  --batch-size 64 \
  --eval-samples 1200 \
  --seeds 0,1,2 \
  --outdir experiments/exp001_tiny_scaling/results

16) Suggested Next Experiments

Priority order:

1. exp002: fixed-FLOPs tradeoff of model size vs data diversity (tests H3)
2. exp003: evaluation-threshold sensitivity study (tests H1 measurement artifact claim)
3. exp004: optimization substitution ablation (tests H2/H3 interaction)

Each new experiment should begin with a short prereg section in its README.

17) Completion Checklist (Before Hand-off)

Before reporting completion, confirm:

• code executes without runtime errors
• outputs are generated and paths are known
• report includes hypothesis + verdict
• notebook entry is updated
• user-facing summary matches generated artifacts

If any item is missing, work is not complete.