Imported from AppThreat/vulnerability-db (AGENTS.md). Install upstream with npx skills add AppThreat/vulnerability-db. Copyright stays with the author.

AGENTS.md

Guidance for AI coding agents and automated contributors working on appthreat-vulnerability-db.

If you are using VDB rather than changing it, read SKILL.md instead. This file is about working on the code.

What this project is

vdb builds and queries an offline SQLite vulnerability database from AppThreat vuln-list, OSV, NVD, GitHub and npm. It is used as a CLI and as a Python library by downstream scanners.

Four goals, in the order they win arguments:

Accurate package locator and version matching. A wrong answer here becomes a wrong security decision downstream, so accuracy beats size and speed.

Fast offline lookup from pre-built .vdb7 files, with no network access during a scan.

Reproducible builds. App-only and app+OS databases, plus per-type shards, from a fresh build every time. There is no migration path and none is wanted.

Bounded resource use on large feeds. Memory and disk are first-class constraints, not afterthoughts.

Repository map

vdb/cli.py              CLI entry point, cache/download/search orchestration
vdb/lib/config.py       environment-driven source, path and database config
vdb/lib/db.py           SQLite connections, schema, pragmas, finalization
vdb/lib/shard_store.py  shard store discovery, fan-out, coverage, staleness
vdb/lib/shard_split.py  the splitter: full database to per-type shards
vdb/lib/db_cmd.py       the `vdb db` subcommand
vdb/lib/cve.py          CVE 5.2 conversion and storage
vdb/lib/osv.py          OSV feed ingestion and conversion
vdb/lib/aqua.py         AppThreat/Aqua vuln-list ingestion, incl. OS distros
vdb/lib/gha.py          GitHub advisory ingestion
vdb/lib/nvd.py          NVD ingestion and the shared CVE-5 conversion
vdb/lib/cpe.py          CPE parsing, normalisation, separator variants
vdb/lib/vers.py         the single version comparison implementation
vdb/lib/search.py       query APIs and result materialization
vdb/lib/search_index.py metadata indexes for text/alias/reference/symbol
test/                   pytest suite and small fixture data
contrib/                measurement harnesses and their findings documents
docs/                   reference docs: DATABASE.md (schema + queries), DESIGN.md (shard client architecture), BENCHMARKS.md
packages/mcp-server-vdb/ MCP server, depends on the library API

The workflow repository that builds public artifacts is checked out separately, usually at ../vdb.

Rules for agents

Gather context before changing behaviour. Trace the call path from the CLI or the source converter through CVESource.store(), vdb.lib.db and search before editing. Ingestion has more layers than it looks: the year floor, for example, is enforced in three separate places, and changing one of them just moves where a record is dropped.

Measure before you claim. Never state a number you did not run. If a change is justified by a count, run the count and put it in the commit message. If you skipped a measurement, say so rather than estimating.

Prefer small fixes with regression tests. Most bugs here involve data volume or a specific feed shape. Add a deterministic fixture or a synthetic harness rather than a test that needs a live download.

Do not download full feeds unless necessary. Prefer $VDB_CACHE/vuln-list.zip, local zips, and the existing test/data/*.json files. When a feed-side measurement genuinely needs the real data, download the one ecosystem zip you need, not all of them.

Bind a scratch database before constructing a source. The safe order is db.get(":memory:", ":memory:") first, then the source. A source constructed first binds whatever database is already bound, which in a scratch script can be the real one.

Treat an existing production database as read-only. Open it with mode=ro&immutable=1. Never call clear_all() on it, never rebuild in place, and do not read one database while a build of your own is writing another.

Assume artifacts are rebuilt from scratch. Do not add schema migration code or legacy-write compatibility unless asked. Search compatibility with older downloaded databases is not a requirement for storage-layout changes.

Treat database size and transient disk use as requirements. When changing SQLite logic, test row counts, distinct hashes, file sizes and temp/journal behaviour. A change that grows the published artifact is a decision for a human, not a side effect.

Do not swallow exceptions around storage. A bare except: pass in an ingestion loop turns a data-corruption bug into missing rows that nothing reports. If you must catch broadly, make the failure visible and make sure the batch cannot grow unboundedly across failures.

Avoid broad formatting churn. Preserve existing style. Reformat only lines you changed.

Never expose tokens. Do not print GITHUB_TOKEN, HF_TOKEN, registry credentials or environment dumps in tests or logs.

Keep environment variables documented. The README reference must stay in sync with vdb/lib/config.py, vdb/lib/db.py, the ingestion modules and the MCP server. Current groups: paths (VDB_HOME, VDB_CACHE, VDB_TEMP_DIR, VDB_SHARDS_DIR), shard behaviour (VDB_AUTO_FETCH, VDB_SHARD_FANOUT), download URLs (VDB_DATABASE_URL, VDB_APP_ONLY_DATABASE_URL), source controls (NVD_START_YEAR, VDB_APP_ECOSYSTEM_START_YEAR, GITHUB_TOKEN, GITHUB_GRAPHQL_URL, GITHUB_PAGE_COUNT, NPM_PAGE_COUNT, OSV_EXCLUDE_MALWARE, OSV_INCLUDE_FUZZ, VDB_OSV_STORE_BATCH_SIZE, VDB_MAX_AFFECTED_PER_BLOB), metadata and progress (VDB_INCLUDE_METADATA, VDB_METADATA_*, VDB_QUIET, VDB_PROGRESS_INTERVAL), SQLite (VDB_SQLITE_IMMUTABLE, VDB_SQLITE_CACHE_SIZE, VDB_SQLITE_JOURNAL_MODE, VDB_SQLITE_SYNCHRONOUS, VDB_DATA_PAGE_SIZE, VDB_INDEX_PAGE_SIZE), distro filters (VDB_IGNORE_OS, VDB_IGNORE_*, VDB_EXCLUDE_*, VDB_INCLUDE_*, VDB_IGNORE_LINUX_KERNEL, VDB_INCLUDE_LINUX_KERNEL), and MCP (VDB_AGE_DAYS).

Things that look like bugs and are not

Worth knowing before you "fix" one of these.

An empty vers string matches every version. Records with no version range do apply to all versions.

A non-numeric version sorts above numeric ones. This is what keeps an npm @latest dist-tag from matching every advisory with an upper bound.

Chainguard and Wolfi advisories with no published fix are stored as affecting every version rather than dropped.

cve_index uses ON CONFLICT IGNORE, so the first writer of a (cve_id, vers, purl_prefix) wins. Reordering sources changes which row survives, database-wide. It is not a performance-only change.

Per-type ecosystem shards overlap the group shards on purpose. The npm shard and the app shard both serve npm. They are separate views, not a partition.

Application ecosystem advisories are not year-filtered. That is deliberate, and NVD_START_YEAR applies only to NVD-style data and the distro feeds.

Common commands

uv sync --all-extras --dev
uv run pytest
uv run ruff check vdb test contrib
uv run vdb --help

Focused runs while working on ingestion or search:

uv run pytest test/test_source.py -k osv
uv run pytest test/test_source.py -k aqua
uv run pytest test/test_search_advanced.py
uv run pytest test/test_reachability.py
uv run pytest test/test_db.py test/test_db_size.py

When adding a database regression test, prefer :memory: databases via db.get(":memory:", ":memory:") unless file size itself is under test. Always call db.reset_connections() before switching database paths.

Local setup without large downloads

AquaSource uses a cached zip at $VDB_CACHE/vuln-list.zip when present.

export VDB_HOME=/tmp/vdb-home
export VDB_CACHE=/tmp/vdb-cache
export VDB_TEMP_DIR=/tmp/vdb-home/tmp
mkdir -p "$VDB_HOME" "$VDB_CACHE" "$VDB_TEMP_DIR"

For tests, build a small zip with only representative paths, then monkeypatch config.CACHE_DIR at it:

vuln-list-main/debian/CVE-2026-0001.json
vuln-list-main/nvd/2026/CVE-2026-0002.json

How public artifacts are built

The build-vdb7.yml workflow in the sibling AppThreat/vdb repository, using the build-and-upload-vdb7 action for the full and app images and split-and-upload-vdb7 to split, gate and publish the per-type shards. The build action sets VDB_HOME, VDB_CACHE, NVD_START_YEAR and GITHUB_PAGE_COUNT, creates vdb_cache/vuln-list.zip, then runs:

python vulnerability-db/vdb/cli.py --cache-os

Full-database jobs add --no-split, because the split-and-upload action re-splits. The workflow's include-metadata input adds --include-metadata. App-only variants set VDB_IGNORE_OS=true plus selected VDB_IGNORE_* flags.

Builds run on a fresh runner every time, so there is no build isolation to preserve and no incremental state to protect. Optimise for scan-time performance and artifact size, not for rebuild ergonomics.

VDB_EXTENDED_DATABASE_URL, VDB_APP_ONLY_EXTENDED_DATABASE_URL and USE_VDB_10Y do not exist in v7. Do not document them as live options.

Investigating a false positive or false negative

Trace in this order, and add the regression test at the lowest layer that demonstrates the bug:

1. The source converter (osv.py, aqua.py, gha.py, nvd.py), including whether the record survived the feed's year floor at all.
2. VulnerabilityDetail fields: mii, mie, mai, mae, fixed_location, package_type.
3. to_purl_vers() in utils.py.
4. CVESource.store5(), for the purl prefix and index row it writes.
5. The search path and vdb.lib.vers.vers_compare().

Between 4 and 5 sits the most common class of defect in this codebase: the store writes a locator the lookup never proposes. Both sides construct a purl_prefix independently, and when the constructions disagree the rows exist and are unreachable, which reads as "not vulnerable". contrib/reachability_audit.py exists to catch exactly this, and a clean match-set gate does not imply reachability.

PR review checklist

Are new source or search paths covered by tests?

Does the change work for a fresh build and a clean download, rather than only for a database that already exists locally?

Are large feeds streamed or processed in bounded batches?

Are SQLite temp files, journals, indexes and VACUUM behaviour considered?

Does the change alter the published artifact's size or contents? If so, is the number measured and stated?

Can a stored row still be reached by the lookup that is supposed to find it?

Are environment variable names consistent with config.py, and documented?

Are errors handled without silently hiding data corruption?

Are dependency or release workflow changes security-sensitive? See THREAT_MODEL.md.