causaltrace

Imported from bilalarif3197/causaltrace (AGENTS.md). Install upstream with npx skills add bilalarif3197/causaltrace. Copyright stays with the author.

CausalTrace — agent notes

Regeneron HackMIT 2026 prize track. See README.md for the full picture; this file is the short version of how to work in the repo without breaking anything.

Run

# backend (http://localhost:8000)
cd backend && .venv/bin/uvicorn main:app --reload --port 8000

# frontend (http://localhost:3000)
cd frontend && npm run dev

No OPENAI_API_KEY → mock mode: the full pipeline runs against backend/fixtures/, but only the three built-in cases work. A custom narrative returns 422 by design, not by accident.

Verify (run all four before calling anything done)

cd backend && .venv/bin/python -m pytest -q          # 196 tests
cd backend && .venv/bin/python validate_fixtures.py  # 136 quotes must be verbatim
cd frontend && npx tsc --noEmit && npm run build
backend/.venv/bin/python evaluation/evaluate_assistance.py

Cases persist to backend/causaltrace.db (SQLite, gitignored). Tests use a temp DB via tests/conftest.py.

Python venv lives at backend/.venv (Python 3.13). Frontend is Next.js 16 + React 19 + Tailwind 4 + @xyflow/react, all pinned exactly.

The product invariant

AI organizes and suggests. The reviewer investigates and decides. There must be no endpoint or UI path that turns a narrative straight into a causality verdict.

Every reviewable entity holds an immutable AiSuggestion beside a separate reviewer slot. confirmed_value returns None unless a human accepted or modified it, and downstream assessment may read nothing else. A reviewer action must never write to ai. If an unreviewed suggestion can move a score, the product has regressed into an autonomous classifier with a confirmation dialog.

Re-running a suggestion stage must keep reviewed items, refresh only the AI half of untouched ones, and add new candidates. Bulk actions touch pending items only.

Other invariants — do not break these

1. The LLM never computes the Naranjo total. It answers items; naranjo.score_items() sums them in pure Python from the published weight table. Keep those halves separate.
2. Naranjo weights are pinned by tests against the published worksheet (range −4..+13). If a weight test fails, the table is wrong, not the test.
3. Two grounding gates. spans.locate_span rejects quotes absent from the source (deterministic); verifier.verify rejects real quotes that do not license their claim (separate LLM pass). A claim failing either becomes UNKNOWN. Never let an ungrounded answer carry a non-zero score.
4. Rejected claims stay visible. Demote to UNKNOWN with a reason; never silently delete.
5. Never invent data to fill a field — no inferred dates, and no inferred years (the demo narratives say "3 January" with no year, so date stays null and only display_date is set). No numeric causal probabilities anywhere.
6. Absence of evidence ≠ evidence of absence. Naranjo item 5 is NO only on positive exclusion; item 6 is UNKNOWN when no placebo was given.
7. Provider code stays in services/llm_client.py. No service module imports a vendor SDK.
8. Evidence valence needs icon + text label + colour, never colour alone.

Editing fixtures

backend/fixtures/<case_id>/<stage>.json replays stage-level model output. Every evidence_text must be a character-for-character substring of that case's narrative in backend/services/cases.py. Run validate_fixtures.py after any edit — it catches retyped quotes that would silently degrade to fuzzy matching.

Claim ids are positional (c000, c001, …) over the full extraction.json list including UNKNOWN entries, and verification.json must reference those exact ids. Inserting a claim mid-list renumbers everything after it.

Honesty constraints (these are product requirements, not style)

• Mock-mode evaluation numbers are circular: the fixtures and evaluation/cases.json references share an author. The harness prints this on every run. Do not remove that banner, and do not quote mock numbers as accuracy.
• Never describe the system as establishing causation, diagnosing, or replacing a clinician.
• Demo cases use drug-class labels ("Drug A, an oral antifungal agent"), never product names.

Harvesting real cases (evaluation/pmc_ingest.py)

Official NCBI interfaces only: E-utilities, the PMC ID converter, efetch, the BioC API. Never scrape the PMC website — it is prohibited and gets the IP blocked. Requests are throttled with backoff and cached under evaluation/.pmc_cache/ (gitignored).

Three rules that must not be relaxed:

1. Auto-parsed scores are candidates, not references. Records carry needs_review: true plus the sentence the value came from. --append refuses unreviewed records.
2. Never commit harvested article text. The OA subset includes CC BY-NC-ND. Narratives live in the gitignored cache; cases.json holds a PMCID pointer unless the licence clearly permits redistribution. evaluate.py resolves narratives from that cache.
3. Published references can be wrong. Score/band disagreements (e.g. "score 4 ... probable", where 4 is Possible) are flagged via internal_inconsistency.

truststore is required: NCBI's cross-signed chain fails certifi validation but passes against the OS trust store.

Seeing the UI (tools/shoot.py)

The frontend can only be verified visually, so drive headless Chromium rather than guessing:

backend/.venv/bin/python tools/shoot.py            # every step -> tools/shots/*.png
backend/.venv/bin/python tools/shoot.py --fresh    # rebuild the seed case (spends tokens)

It reports console errors, page errors, failed requests and horizontal overflow, and seeds a case left in a MIXED review state on purpose — an all-pending or all-accepted case hides most of the states the UI has to render. Read the PNGs directly; that is the feedback loop.

Leaves a case titled [screenshot seed] ... in the database; safe to delete.

No emojis

Use the SVG icons in components/ui.tsx (CheckIcon, PencilIcon, CrossIcon, WarnIcon, QuestionIcon, AiIcon). Dingbat characters render inconsistently and some fonts substitute colour emoji for them, which is wrong for a clinical tool.

Concurrency

POST /api/cases/{id}/suggest-batch computes the PARALLEL_STAGES concurrently. Two rules:

1. Only _compute may run in parallel, and it must never mutate the document. _merge runs single-threaded in the caller's order, so results do not depend on network timing.
2. missing and rationale are not batchable. Both read confirmed_evidence_digest, so running them alongside extraction hands them an empty case. run_suggest_batch rejects them.

Do not emulate the batch by calling /suggest/{stage} several times in parallel from the frontend: each call rewrites the whole document, so the last response would discard the rest. tests/test_batch.py pins this, including that no stage is lost and that a partial failure keeps the stages that worked.

The client negotiates JSON mode once under a lock; concurrent callers wait for the answer rather than each probing.

External data sources

Never crawl the Bookshelf or PMC websites. Bookshelf prohibits it explicitly, and efetch does not serve Bookshelf body text at all — use the OAI service. All four scripts need truststore.inject_into_ssl(); without it TLS validation fails in a way that looks like a network outage.

The label lookup must not answer Naranjo item 1

lookup_label_evidence supplies citable evidence and nothing more. A label listing a reaction is not a published case report, and absence from a label leaves item 1 UNKNOWN rather than NO. A parametrised test asserts the score stays 0 whichever way the lookup resolves.

Confirm every openFDA hit against the label's own openfda names. A loose full-text search once returned an ophthalmic product for oral TMP-SMX and reported the reaction absent from the wrong medicine's label, which is worse than finding nothing.

RUCAM

Weights come from NBK548272 and the manual's category ranges are asserted in tests/test_rucam.py. Two rules that must not be relaxed:

1. Scoring is pattern-dependent. Without ALT and ALP (and their upper limits) there is no R ratio, no pattern, and no valid RUCAM. Return not-calculable.
2. Refuse when the manual refuses. Injury before exposure, onset too long after withdrawal, or unknown onset all mean a RUCAM must not be produced. Never substitute a number.

RUCAM is liver-specific and gated on is_hepatic_event.