fixturedb

Imported from joaofix/fixturedb (AGENTS.md). Install upstream with npx skills add joaofix/fixturedb. Copyright stays with the author.

AGENTS.md — FixtureDB Project Context

This file provides context for AI coding assistants (GitHub Copilot, Kilo AI, Claude Code, etc.) working on this repository.

What is this project?

FixtureDB is the replication package for a conference paper:

"An empirical study on test fixture usage by coding agents on open source software"

It's a master's thesis companion codebase. The pipeline detects agent-authored commits in GitHub repositories, extracts test fixtures from them, and compares agent vs human fixture characteristics.

Key concepts

• Agnet enabled repository: A Github repository that has passed our validation that checks for AI conding agent configuration files. Meaning that the repository is probably using the AI conding agent for conding tasks.
• Agent commit: a commit authored or co-authored by an AI coding agent (Claude, Copilot, Cursor, Aider, etc.), detected via Co-authored-by: trailers and author metadata
• Agent test commit: An agent commit that specifically modifies a test file.
• SEART Github Search: The tool we use to collect a complete list of Github repositories that matches our quality criteria. Stored at github-search-raw/. This is the list of repos we start with for datasets A and C.

Key datasets we build

• Dataset A (datasets/a/): agent-authored fixtures from agent-enabled repos
• Dataset B (datasets/b/): human-authored fixtures from the same repos (matched control)
• Dataset C (datasets/c/): human-authored fixtures from pre-2021 repos (cross-repo baseline)

Every dataset is built through the same CLI verbs, selected via --dataset {a,b,c} (python -m collection <verb> --dataset X) — see "Command-line interface" below. Each verb calls exactly one collector/function; no runtime branching decides which dataset a run produces:

Languages we collect fixtures for

Python, Java, JavaScript, TypeScript.

Architecture

collection/          # Main pipeline code (the "library")
  __main__.py        # `python -m collection <verb> --dataset {a,b,c}` -- the one CLI surface
  paths.py           # Central path registry: datasets/{a,b,c}/{stage}, db/{a,b,c}.db, export/{a,b,c}.zip
  config.py          # Thresholds, dates -- re-exports constants loaded from study_parameters/ and heuristics/ below
  study_parameters/  # Settings + study-design constants as YAML (non-code extensions, framework
                     # registry, per-language configs, and study_parameters.yaml: temporal
                     # boundaries/quality thresholds/sampling params) -- edit these, not config.py
  db.py              # Connection/query helpers, upsert helpers, migrations
  db_schema.py       # SQLite DDL (CREATE TABLE/INDEX statements)
  repo_metadata.py   # Domain/star-tier/repo-age computation (no DB dependency)
  tiered_agent_corpus_scanner.py  # Commit scanning, agent detection, adoption intensity (formerly agent_commit_detector.py)
  agent_signal_primitives.py  # Low-level agent config-file + commit-trailer detection (formerly agent_detector.py)
  agent_patterns.py  # Loads the agent catalog (see heuristics/ below) and derives detection dicts
  heuristics/        # Detection-heuristic catalogs as YAML/CSV -- pattern/keyword tables that
                     # drive a classification decision (agent vs. human, fixture vs. not,
                     # boilerplate repo vs. not), as opposed to study_parameters/'s plain settings
  heuristics/agent_heuristics.yaml  # paper_scope only now -- edit here to change the paper's strict-scope agent subset (this project's own catalog, kept at heuristics/ root)
  heuristics/agent-mining/agent_files.csv  # Config-file/directory patterns (pattern,tool,start_date,end_date) -- mirrors labri-progress/agent-mining's files.csv schema+content for citation
  heuristics/agent-mining/agent_authors.csv  # Commit author/trailer signatures (pattern,tool,start_date,end_date) -- mirrors labri-progress/agent-mining's authors.csv schema+content for citation
  heuristics/agent-mining/bots.csv  # CI/automation bot account patterns (pattern,tool) -- mirrors labri-progress/agent-mining's bots.csv schema+content for citation
  heuristics/fixture_definitions.yaml  # Operational definition of "fixture" per language -- edit here to update a detector pattern; also documents per-language `excluded` boundary cases the fixture detectors deliberately don't catch (reviewer audit trail)
  heuristics/exclusion_keywords.yaml  # Repo name/description keywords that signal a boilerplate/toy repo
  heuristics/feature_extraction_patterns.yaml  # Mock-framework/external-call/object-instantiation regex tables + setup/teardown pairing rules
  clone_primitives.py / ephemeral_clone.py / persistent_clone.py  # Layered cloning: raw primitive / throttled ephemeral / DB-tracked persistent
  repository_quality_control/agent_repository_counter.py  # discover-repos --dataset a
  repository_quality_control/agent_commit_counter.py      # discover-commits --dataset a
  repo_resolve.py    # discover-repos --dataset b (resolves Dataset B's repo list from Dataset A's)
  select_dataset_c_repos.py  # discover-repos --dataset c
  dedupe_dataset_c_repos.py  # repo-level dedup, between discover-repos --dataset c's two passes
  test_commit_filter.py       # filter-test-commits --dataset a
  human_test_commit_filter.py # filter-test-commits --dataset b (+ pre-2021 helper used by Dataset C tooling)
  test_commit_resume_state.py # checkpoint/resume state shared by both filters above
  dedupe_commits_by_sha.py  # cross-repo-name commit dedup, a/b, after filter-test-commits
  agent_corpus.py    # Dataset A: extract agent fixtures (extract-fixtures --dataset a)
  human_corpus.py    # Dataset B: extract human fixtures, within-repo (extract-fixtures --dataset b)
  human_corpus_repo_selection.py  # Dataset B's repo selection, split out of human_corpus.py
  dataset_c.py        # Dataset C: extract human fixtures, cross-repo baseline (extract-fixtures --dataset c)
  dedupe_fixtures_by_sha.py  # Dataset B only -- fixture-level dedup, run after every extract-fixtures --dataset b
  dataset_pipeline.py # analyze-distribution / sample / export cross-cutting stages
  dataset_validator.py # validate stage
  dataset_summary.py  # summarize stage -- writes {dataset}/summary.yaml (repo/commit/fixture counts, purity-gate rate for a/b); also run automatically at the end of `toy`
  toy.py              # `toy --dataset {a,b,c}`: small real end-to-end run under toy-dataset/
  fixture_extractor.py     # Tree-sitter AST fixture extraction
  detector.py        # Fixture pattern detection
  corpus_utils.py    # Shared repo/fixture persistence helpers
  between_group_comparison.py  # Statistical-test primitives (effect sizes etc.) used by research_questions/ below.
                     # BetweenGroupComparator in here is dead/orphaned -- don't use it
  research_questions/  # Answers the paper's RQs from db/{a,b,c}.db, writes research_questions/*.md (committed):
                     # rq1/rq2/rq3.py (structural/teardown/mocking), balance.py (control-variable check),
                     # dataset_findings.py (non-RQ descriptive findings), language_contamination.py, _shared.py
  validation_sampling.py  # Manual, on-demand Cochran-formula sampling for human review (not part of the automatic pipeline)
tests/               # pytest suite
eda/                 # Exploratory data analysis notebooks
docs/                # Full documentation
internal-docs/       # Internal notes, RUN_COMMANDS.md (authoritative per-dataset command recipes)
paper-draft/         # The paper, one markdown file per section, LaTeX tables in ```latex fences. Gitignored.
                     # Pull numbers fresh from research_questions/*.md and datasets/*/summary.md -- don't
                     # trust a previous fill-in without checking current collection state

Command-line interface

One CLI, one set of step verbs shared across all three datasets:

python -m collection discover-repos       --dataset {a,b,c}
python -m collection discover-commits     --dataset a
python -m collection filter-test-commits  --dataset {a,b}
python -m collection extract-fixtures     --dataset {a,b,c}
python -m collection analyze-distribution --dataset a --against b
python -m collection sample               --dataset {a,b,c}
python -m collection export               --dataset {a,b,c}
python -m collection validate             --dataset {a,b,c}
python -m collection summarize            --dataset {a,b,c}
python -m collection toy                  --dataset {a,b,c} [--repos N]

Not every verb applies to every dataset (discover-commits/most of filter-test-commits are Dataset-A/B-specific, since Dataset C has no per-commit history scan) — invoking one that doesn't apply exits 1 with an explicit message. Every verb resolves its default input/output directories through collection/paths.py; toy runs the identical step functions rooted under toy-dataset/ instead of datasets/+db/.

The dedupe_*.py scripts sit outside this verb interface -- invoke as python -m collection.dedupe_fixtures_by_sha --dataset b, etc. For the full ordered command chain per dataset, see internal-docs/RUN_COMMANDS.md.

Database

SQLite via collection/db.py. One output database per dataset: db/a.db, db/b.db, db/c.db. Schema includes repositories, test_files, fixtures, commit_observations, test_commits, mock_usages. Use db_session() context manager for all DB access — it handles WAL mode, retries, and connection pooling. The database is secondary: the CSV files under datasets/{a,b,c}/ are the real, reviewable output of each pipeline stage.

Key constants (config.py)

• AGENT_CORPUS_START_DATE = "2025-01-01" — adoption window start
• HUMAN_CORPUS_CUTOFF_DATE = "2020-12-31" — pre-agent cutoff for Dataset C
• MIN_STARS = 500, MIN_COMMITS = 100, MIN_TEST_FILES = 5 — quality thresholds

Testing

python3 -m pytest tests/ -v     # Full suite (~65s)
python3 -m pytest tests/test_adoption_intensity.py -v  # Single module

Tests use tmp_path fixtures for temporary git repos and SQLite databases. Never run pytest from the repo root without testpaths = ["tests"] in config — the clones/ directory contains hundreds of external repos with their own tests.

Code style

• Python 3.12+, formatted with ruff (line-length 88), linted with Ruff
• Type hints used throughout collection/
• Logging via collection/logging_utils.py (get_logger(__name__))
• Database access always through db_session() context manager
• Shared utilities in corpus_utils.py — avoid duplicating repo/fixture persistence logic

Personal instructions

• If we fix a bug, we unit test that
• If we implement a new feature, we unit test that
• We use a python virtualenv called venv to run tests, using the command "pytest tests/"
• A prompt that changed code is only considered finished if all tests are passing
• If we change collection/, we evaluate the need to update the docs/ folder as well. collection/ represents the methodology of this work as code. We want to keep documentation up to date with the methodology.
• Long-running collection commands: chain with && and a curl -d "<step> finished" ntfy.sh/joaofix_fixturedb after each step.
• Verify paper-draft/ numbers against current collected data before citing -- don't trust a previous fill-in.