Imported from jeromevde/Pyfooda (AGENTS.md). Install upstream with npx skills add jeromevde/Pyfooda. Copyright stays with the author.

Pyfooda — Agent Instructions

Pyfooda builds and serves a nutritional ingredient database by matching a curated vocabulary of ~1310 food ingredients to USDA FoodData Central entries.

Repository layout

pyfooda/
  data/
    epicure_vocabulary.json   Canonical ingredient list (~1310 entries, id + display name)
    dropped_unmatched.json    Specialty names removed (no credible USDA match)
    fooddata.csv.gz           Bundled USDA FoodData Central export (decompress before use)
    fooddata.csv              Decompressed USDA data (295 K rows, generated)
    ingredients.csv           Final averaged nutrient table (generated)
    ingredients_meta.json     Per-ingredient USDA source list + per-nutrient stats (generated)
    nutrients.csv             Reference list of nutrient column names
    nutrient_coverage.json    Coverage report (generated)
  build/
    build_database.py         Full embedding-based rebuild (needs sentence-transformers)
    curate.py                 LLM-driven source curation — preferred for targeted fixes
    recompute.py              Recompute averages from current ingredients_meta.json sources
    download_usda.py          Decompresses fooddata.csv.gz → fooddata.csv
    coverage.py / coverage_report.py
    nutrient_stats.py         Averaging, outlier detection, source selection
    paths.py                  Canonical file paths
  api.py                      Public API (load ingredients, look up nutrients)
docs/
  index.html                  Static web viewer served from docs/data/ingredients.json
  data/ingredients.json       Web export (generated by export_web.py)

Key commands

# Decompress USDA data (required before any build step)
python -m pyfooda.build.download_usda

# Fix or rebuild specific ingredients with LLM curation (recommended):
python -m pyfooda.build.curate --ids fennel,lard,potato --model gpt-4o-mini
python -m pyfooda.build.curate --ids fennel --dry-run          # preview candidates only

# Rebuild the entire database with LLM curation (~$0.24, ~1 min with workers):
python -m pyfooda.build.curate --all --model gpt-4o-mini --workers 20

# After any curate run, recompute averages and write ingredients.csv:
python -m pyfooda.build.recompute

# Full embedding-based rebuild (rarely needed, requires sentence-transformers + GPU):
python -m pyfooda.build.build_database

# Regenerate web export after a recompute:
python -m pyfooda.build.export_web

Environment: set OPENAI_API_KEY before running curate.py. Compatible with OpenAI, GitHub Models (--base-url https://models.inference.ai.azure.com), and any other OpenAI-compatible endpoint.

The curate.py workflow (LLM-centric database build)

This is the preferred approach for fixing individual ingredients or rebuilding the whole database without re-running the expensive embedding step.

How it works

1. Decompresses USDA data if not already done.
2. Builds a TF-IDF index over the 10 K authoritative rows only (sr_legacy_food, foundation_food, survey_fndds_food — branded products excluded because their names are generic labels like "BEEF" or "CHICKEN" that swamp keyword retrieval).
3. For each ingredient, retrieves the top-K candidates (--top-k, default 20; use 50 for short/common names like "chicken" or "potato").
4. Strips candidates with extreme micronutrient values (>100× DRV) before presenting to the LLM.
5. Calls the LLM with a structured prompt asking it to select entries that represent the plain, whole, grocery-store form of the ingredient.
6. Writes selected source records to ingredients_meta.json.
7. Caller runs recompute.py to rebuild ingredients.csv.

Prompt design (in curate.py)

The system prompt instructs the LLM to:

• Prefer raw > plain-cooked > processed
• Reject different parts (fennel seed ≠ fennel bulb)
• Reject composite dishes, fast food, baby food, fortified supplements
• Prefer sr_legacy_food / foundation_food / survey_fndds_food data types
• Return only a JSON array of 0-based indices: [0, 2]

Scaling to ~1310 ingredients

Curation triages first: strong headword matches auto-accept, no-USDA names skip the LLM, and only ambiguous cases hit the model (batched). A full rebuild with --workers 15 --batch-size 20 should finish in minutes for well under €1.

TF-IDF limitations and workarounds

TF-IDF over short ingredient names can miss the best entries:

For a full rebuild, --top-k 50 is recommended.

Quality assessment

After a rebuild, compare against reference values:

import pandas as pd
df = pd.read_csv('pyfooda/data/ingredients.csv')
# Check problem ingredients
df[df.ingredient_id.isin(['potato','fennel','lard','beef','chicken','pie_crust'])][
    ['ingredient_id','Energy','Protein','Iron','Calcium','Vitamin C','source_count']
]

Expected ranges (per 100 g):

Red flags to investigate:

• Energy >2× expected → likely fried/processed anchor
• Iron >5 mg for a non-organ/non-seed food → seed or spice contamination
• Calcium >100 mg for a meat → dairy composite dish included
• Vitamin C > 10 mg for a fat/meat → vegetable data contamination

Data pipeline architecture

epicure_vocabulary.json
        │
        ▼
curate.py (TF-IDF candidate retrieval + LLM selection)
        │
        ▼
ingredients_meta.json   ◄─── also updated by build_database.py (embedding pipeline)
        │
        ▼
recompute.py  (averages nutrients, runs outlier stats)
        │
        ├──► ingredients.csv      (used by api.py)
        ├──► nutrient_coverage.json
        └──► export_web.py ──► docs/data/ingredients.json

Outlier detection (nutrient_stats.py)

Source rows with any micronutrient exceeding 100× DRV are excluded before averaging. DRV reference values are in _NUTRIENT_DRV in nutrient_stats.py. This catches data-entry errors (e.g. 480 µg Vitamin B-12/100 g in a pie crust branded entry) without discarding legitimately high-nutrient foods like beef liver (83 µg B-12 ≈ 35× DRV, safely below the cap).

Source selection (nutrient_stats.py)

select_source_rows picks an anchor (highest quality score = similarity + data_type bonus) then adds sources whose macros (Energy, Protein, Carbs, Fat) agree with the anchor within 15% relative tolerance. This prevents a single outlier from skewing the average.

Adding new ingredients

1. Add the entry to epicure_vocabulary.json:

   { "id": "my_ingredient", "name": "My Ingredient" }

2. Run curation for it:

   python -m pyfooda.build.curate --ids my_ingredient --model gpt-4o-mini
   python -m pyfooda.build.recompute

3. Verify the values look reasonable.
4. Run export_web.py if the docs site needs to be updated.

Common mistakes to avoid

• Do not add regex/keyword rules to filter USDA candidates. The LLM understands food semantics; hardcoded rules introduce bugs (e.g. putting "fennel" in a spice list caused all fennel-seed values to contaminate the fennel-bulb entry).
• Always run recompute.py after editing ingredients_meta.json.
• Use --dry-run before a full curate run to preview what the LLM will see.
• fooddata.csv is a build artifact — do not commit it. Only fooddata.csv.gz and fooddata.csv.sha256 are tracked.