Imported from docxology/template (projects/templates/template_search_project/AGENTS.md). Install upstream with npx skills add docxology/template --skill template_search_project. Copyright stays with the author.

template_search_project — Agent Guide

Purpose

Exemplar project demonstrating end-to-end use of the infrastructure/search/, infrastructure/reference/, and infrastructure/llm/ modules. Mirrors the structure of projects/templates/template_code_project/ so the pipeline runner discovers and executes both projects with the same code.

Decision memory and verifier hardening follow docs/rules/memory_and_decision_records.md: use nearby WHY: comments only for surprising local choices, keep volatile counts generated, and add negative controls for verifier-like gates.

Subfolder documentation: docs/AGENTS.md, manuscript/AGENTS.md, src/AGENTS.md, tests/AGENTS.md, scripts/AGENTS.md (each with a README.md in the same directory).

Layout

flowchart TB
    ROOT[/projects/templates/template_search_project/]
    ROOT --> SRC_DIR[/src<br/>domain logic/]
    ROOT --> TESTS_DIR[/tests<br/>real-data tests/]
    ROOT --> SCR_DIR[/scripts<br/>thin orchestrators/]
    ROOT --> MAN_DIR[/manuscript/]
    ROOT --> DATA_DIR[/data<br/>committed corpus/]
    ROOT --> DOCS_DIR[/docs/]
    ROOT --> OUT_DIR[/output<br/>regeneratable/]
    ROOT --> META[pyproject.toml<br/>README.md · AGENTS.md · .gitignore]

    SRC_DIR --> S_CFG[config.py<br/>typed YAML loader]
    SRC_DIR --> S_PIPE[pipeline.py<br/>search → enrich → BibTeX]
    SRC_DIR --> S_DEEP[deep_search.py<br/>multi-keyword fan-out]
    SRC_DIR --> S_SYN[synthesis.py<br/>prompts · LLM callable]
    SRC_DIR --> S_FIG[figures.py<br/>matplotlib plots]
    SRC_DIR --> S_MV[manuscript_variables.py<br/>substitution]
    SRC_DIR --> S_REP[report.py<br/>markdown assembly]
    SRC_DIR --> S_AN[analysis.py<br/>review-stage helpers]
    SRC_DIR --> S_DOT[dotenv.py<br/>.env loader]

    TESTS_DIR --> T_LIST[test_config · test_pipeline · test_analysis ·<br/>test_scripts · …]

    SCR_DIR --> SC0[run_deep_search.py]
    SCR_DIR --> SC1[run_search_pipeline.py]
    SCR_DIR --> SCS[s_compose_literature_review.py]
    SCR_DIR --> SC2[y_generate_search_figures.py]
    SCR_DIR --> SC3[z_generate_manuscript_variables.py]
    SCR_DIR --> SC4[zz_generate_review_report.py]
    SCR_DIR --> SC5[zzz_build_dashboard.py]
    SCR_DIR --> SCR_REV[review<br/>configurable gate]

    MAN_DIR --> M_CFG[config.yaml<br/>single source of truth]
    MAN_DIR --> M_PRE[preamble.md]
    MAN_DIR --> M_SECTIONS[00_abstract.md … 07_deep_search.md ·<br/>S01_literature_review.md · 99_references.md]
    MAN_DIR --> M_BIB[references.bib · references_deep.bib<br/>auto-generated where applicable]

    DATA_DIR --> D_CORPUS[corpus.json<br/>offline default]

    DOCS_DIR --> D_FILES[quickstart.md ·<br/>architecture.md ·<br/>output_conventions.md ·<br/>troubleshooting.md]

    classDef dir fill:#0f172a,stroke:#0f172a,color:#fff
    classDef code fill:#1e3a8a,stroke:#0f172a,color:#fff
    classDef doc fill:#0f766e,stroke:#0f172a,color:#fff
    classDef gen fill:#7c2d12,stroke:#0f172a,color:#fff
    class ROOT,SRC_DIR,TESTS_DIR,SCR_DIR,MAN_DIR,DATA_DIR,DOCS_DIR,OUT_DIR dir
    class S_CFG,S_PIPE,S_DEEP,S_SYN,S_FIG,S_MV,S_REP,S_AN,S_DOT,SC0,SC1,SCS,SC2,SC3,SC4,SC5,SCR_REV,T_LIST code
    class M_CFG,M_PRE,M_SECTIONS,D_CORPUS,D_FILES,META doc
    class M_BIB gen

Key contracts

src/publish/config.py::ProjectConfig — every knob is here. Adding a new flag means: add a field to the dataclass, add YAML parsing in from_dict, add a default in manuscript/config.yaml. Tests live in tests/publish/test_config.py.
src/pipeline/pipeline.py::run_literature_pipeline — the single-query search → enrich → BibTeX entry point into infrastructure.search.* (the multi-keyword fan-out in src/search/deep_search.py::run_deep_search performs the equivalent calls for the deep-search run mode). Returns a :class:LiteratureRunArtifacts so the script knows where every artefact landed without re-deriving paths.
src/pipeline/synthesis.py — duck-typed llm: (str) -> str callable lets tests pass deterministic local functions and runtime callers pass an Ollama-backed adapter. Prompts are module-level constants.
src/analysis/report.py::write_reading_report — single function; takes a SearchResult + citation-key map + optional synthesis records.

Run modes

Standard pipeline (`run_search_pipeline.py`)

Command	Behaviour
`python scripts/run_search_pipeline.py`	Configured live mode; only hits arXiv/Crossref or runs an LLM when those providers are explicitly enabled. The committed default uses the local fixture and no network.
`… --no-llm`	Skip the LLM stage; produce reading report without synthesis.
`… --no-cache`	Bypass cache reads (writes still happen).
`… --corpus path.json`	Required when `project_config.search.sources` includes `local`.
`… --config other.yaml`	Use an alternative config file.

Deep search (`run_deep_search.py`)

Reads the deep_search: block of config.yaml. Each keyword runs its own SearchQuery (capped at max_results_per_keyword, default 100), every paper is fully enriched (abstract + fulltext), and each paper gets a multi-section markdown reading note (LLM-generated when enabled). See src/search/deep_search.py and manuscript/07_deep_search.md.

Command	Behaviour
`python scripts/run_deep_search.py`	Honours `project_config.deep_search.enabled`; exits 2 if disabled.
`… --enable`	Force-enable regardless of config.
`… --keyword "X" --keyword "Y"`	Override the keyword list at the CLI.
`… --no-llm`	Skip per-paper LLM summaries even when config enables them.
`… --no-cache`	Bypass `SearchCache` reads.
`… --corpus path.json`	Use a local JSON corpus instead of network backends.

Testing

uv run pytest projects/templates/template_search_project/tests/ -v

All tests run offline: LocalBackend against real temp files, deterministic LLM callable, real subprocess where needed. The committed data/corpus.json is a deterministic fixture, not empirical evidence; the standard report writes a fixture-scope notice and rejects high-confidence empirical assertion language in fixture-backed synthesis.

How this project differs from `template_code_project`

template_code_project has its own algorithm (src/template_code_project/core/optimizer.py) and generates figures from numerical experiments.
template_search_project has no algorithm — its src/ is pure orchestration over infrastructure/. The "experiment" is the pipeline itself.
Both projects emit a Pandoc-ready references.bib; this project populates it from a query, while template_code_project ships a hand-curated one.
Both projects produce a manuscript PDF via the standard pipeline.

Extending

To target a different topic:

Edit manuscript/config.yaml → project_config.search.query.
Adjust project_config.search.year_min / year_max / sources as needed.
Re-run scripts/run_search_pipeline.py.

To swap LLM models:

Change llm.model (must be available locally via ollama pull <model>).
Optionally change llm.seed / llm.temperature.

To use only a curated corpus (offline reproducibility):

Generate one once via infrastructure.search.literature.write_corpus.
Set project_config.search.sources: [local] in config.yaml.
Pass --corpus path/to/corpus.json to the script.

Review phase

Configurable gate via scripts/review and review_config.yaml. The project-analysis stage runs scripts/zz_generate_review_report.py last; if output/review/summary.json is absent it invokes scripts/review subprocess before writing output/review/REVIEW_REPORT.md.

List stages:

cd projects/templates/template_search_project && uv run python scripts/review --list

Run all enabled stages (from repo root):

uv run python projects/templates/template_search_project/scripts/review \
  --project-root "$(pwd)/projects/templates/template_search_project"

Available stages

Name	Validates	Backend
`prerender_validation`	Source markdown gate	`infrastructure.validation.cli prerender` (`--repo-root` is `.` with project `cwd`)
`markdown_links`	Repo link scan	`infrastructure.validation.cli links`
`bibtex_validation`	BibTeX strict	`infrastructure.reference.citation.cli validate`
`bibliography_completeness`	`[@key]` ↔ `.bib`	`src.analysis` (subprocess)
`variables_resolved`	`{{TOKENS}}`	`src.analysis`
`output_integrity`	`output/`	`infrastructure.validation.cli integrity`
`test_suite_health`	pytest + coverage	subprocess
`infrastructure_usage`	`infrastructure.*` imports	`src.analysis`
`determinism_check`	cache / seed / temperature	`src.analysis`

Disable a stage with enabled: false in review_config.yaml.

Custom stages

Add stage_type: custom and wire the subprocess in scripts/review to src.analysis (see that file’s run_custom_stage).

Related Capabilities

This project's full-text acquisition is the narrow FulltextFetcher in infrastructure/search/literature/fulltext.py — arXiv-derived or paper.pdf_url PDF download only, with no Unpaywall/OA resolution across arbitrary DOIs — and src/ has no workflow-graph decomposition or composite reproducibility score over that text. template_literature_meta_analysis is a candidate pattern to adopt here if this project's acquisition step grows to match: its src/literature/fulltext_download.py resolves full text via Unpaywall/OA/direct-PDF with graceful degradation, feeding src/reproducibility/ — an LLM-populated workflow-graph model (models.py) scored by a no-compensation composite (scoring.py: R = sqrt(Rc * Rs) over content and structural sub-scores). Porting the workflow-graph/scoring approach here would first require a comparable multi-source full-text resolver, since the sibling project's reproducibility runner assumes richer full-text coverage than a single-source arXiv fetch provides. Not planned work — noted for future reference only.

Optional gap-filler search (Monid)

This exemplar stays offline-by-default and does not import Monid from src/. When an agent or operator already has MCP connectors or direct API keys configured, use those first. For uncovered web-search or data-endpoint gaps, the Layer-1 Monid gateway (infrastructure/search/monid/) provides discover / inspect / run with per-endpoint wallet pricing. Compare direct-provider USD/1k list prices via infrastructure/search/monid/PRICING.md; upstream skill: infrastructure/search/monid/SKILL.md. Requires MONID_API_KEY and spends prepaid balance — never wire into default CI or pipeline stages.

template_search_project

About