Imported from arXiv/arxiv-search (AGENTS.md). Install upstream with npx skills add arXiv/arxiv-search. Copyright stays with the author.

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this repo is

Three Flask apps built from one shared Elasticsearch-backed core, plus indexing scripts:

All three are defined in search/factory.py and deploy independently (Dockerfile-ui, Dockerfile-api, Dockerfile-classic-api; Helm charts in deploy/).

Toolchain

Poetry, and Python 3.10 specifically. CI pins 3.10.9.

Python 3.11+ breaks the test suite. Field, Operator, SortBy, SortDirection in search/domain/base.py are (str, Enum) subclasses whose values get interpolated into Lucene query strings and ES DSL. 3.11 changed str.__format__/__str__ for such enums to return Field.Title instead of ti, so query building silently produces garbage and the classic-parser tests fail. Before debugging a mysterious lark.exceptions.UnexpectedCharacters on Field.Title:..., check python --version.

If poetry env list shows a 3.11+ interpreter, point it at 3.10 (poetry env use 3.10) rather than working around the failures.

Pipfile/Pipfile.lock are legacy — last touched in 2023, still listing nose2 and mypy==0.720. pyproject.toml/poetry.lock are what CI and the current Dockerfiles use.

Commands

poetry install                      # dev deps included
poetry run pytest                   # full suite; ~100 tests, no Elasticsearch needed (all mocked)
poetry run pytest -q search/domain/classic_api/tests/test_classic_parser.py
poetry run pytest -q search/routes/classic_api/tests/test_classic.py::TestClassicAPISearch::test_query

Run a dev server (any of the three apps — swap FLASK_APP):

FLASK_APP=app.py FLASK_DEBUG=1 ELASTICSEARCH_SERVICE_HOST=127.0.0.1 poetry run flask run

python main.py runs the UI on :8080 with template auto-reload, no env vars needed.

Local Elasticsearch + Kibana: docker-compose up (ES on :9200, Kibana on :5601). Then populate:

FLASK_APP=app.py ELASTICSEARCH_SERVICE_HOST=127.0.0.1 poetry run python create_index.py
FLASK_APP=app.py ELASTICSEARCH_SERVICE_HOST=127.0.0.1 poetry run python bulk_index.py

bulk_index.py with no args indexes tests/data/sample.json (several minutes); --paper_id/--id_list for specific papers. It fetches from the /docmeta endpoint, which is only reachable from the CUL network. reindex.py OLD NEW copies an index under the current mapping; audit.py -l ids.txt -o missing.txt finds unindexed papers.

Static checking (goals: mypy/pydocstyle zero, pylint ≥9):

poetry run mypy -p search | grep -v "test.*" | grep -v "defined here"
poetry run pydocstyle --convention=numpy --add-ignore=D401 search
poetry run pylint search

Test discovery gap

Five test files are named tests.py, which pytest's default test_*.py pattern does not collect, so poetry run pytest and CI skip them entirely:

• search/controllers/advanced/tests.py, search/controllers/simple/tests.py, search/controllers/tests.py
• search/process/tests.py
• search/services/index/tests/tests.py

They are runnable by explicit path (poetry run pytest search/controllers/advanced/tests.py) and some currently fail. Treat their coverage as unverified — if you change a controller or search/process/transform.py, run its tests.py by hand.

Architecture

Request flow is a strict four-layer stack; each layer speaks only in domain objects:

routes/ ──> controllers/ ──> services/index/ ──> Elasticsearch
   │             │                  │
   └─ serialize/ └─ domain/ ────────┘

• search/routes/ — blueprints. Parse nothing; hand request.args to a controller, get back (data, status, headers), render or serialize. ui.py also persists order/size/abstracts/date-date_type in the arxiv-search-parameters cookie via before_request/after_request.
• search/controllers/ — one package per feature (simple, advanced, api, classic_api). Each exposes a function returning the 3-tuple above, and owns request validation (WTForms in simple/forms.py, advanced/forms.py) and building a domain Query.
• search/domain/ — the shared vocabulary. SimpleQuery, AdvancedQuery, APIQuery, ClassicAPIQuery are dataclasses; search results (Document, DocumentSet) are TypedDicts. That split is deliberate — see DECISIONS.md 2019-04-22: dataclass casting was ~100× the cost of dict init and results are high-volume.
• search/services/index/ — everything Elasticsearch. SearchSession.search() dispatches on the query's type (isinstance(query, AdvancedQuery) → advanced.py, SimpleQuery → simple.py, APIQuery → api.py, ClassicAPIQuery → classic_api/). prepare.py holds SEARCH_FIELDS, the field→query-builder map that all four share; highlighting.py amends the query and post-processes hits; authors.py handles name/ORCID/author-id matching; results.py turns an ES response into a DocumentSet.
• search/serialize/ — as_json (metadata API) and as_atom (classic API, via feedgen + custom atom_extensions.py). Response shapes are pinned by JSON Schema in schema/resources/, which the serializer tests validate against.

Indexing (separate from serving): services/metadata.py fetches DocMeta from the docmeta endpoint → process/transform.py maps it to an indexable Document → SearchSession.add_document/bulk_add_documents. Each version of a paper is its own ES document (DECISIONS.md #6); simple/author search filter to current versions, advanced search does not.

Things that will surprise you

• search/services/index/__init__.py monkeypatches elasticsearch.client.Elasticsearch.search at import time to issue POST instead of GET, because GCP load balancers reject GET requests carrying a body. It runs before any other import in the module.
• ELASTICSEARCH_MAPPING defaults to the relative path mappings/DocumentMapping.json, so create_index.py/reindex.py must be run from the repo root.
• MAX_RESULTS = 10_000 (services/index/util.py) caps pagination; exceeding it raises OutsideAllowedRange.
• The API endpoints are unauthenticated. @scoped(required=scopes.READ_PUBLIC) is commented out in both routes/api/__init__.py and routes/classic_api/__init__.py.
• TESTING=yes triggers a live ES check at startup (factory.index_startup_check) that aborts if the cluster, index, or a sample query fails.
• Config is read once from os.environ at import of search/config.py. The WSGI wrappers (wsgi*.py) copy the WSGI environ into os.environ before calling the factory for exactly this reason.

Classic API compatibility

The classic API is a Python reimplementation of a Perl/Lucene 2.3.2 service, and bug-compatibility with it is a requirement, not a nicety.

• search/domain/classic_api/query_parser.py — a lark grammar over the documented search_query syntax (au:, ti:, ANDNOT, …), producing a Phrase tree.
• search/domain/classic_api/classic_search_query.py — adapt_query(), a pre-parse rewrite pass that repairs queries old Lucene accepted but the grammar rejects (bare lastUpdatedDate: aliasing, partial submittedDate ranges filled out to 12 digits, stray characters). Controllers call it before parsing; &raw=1 bypasses it.
• search/domain/classic_api/README_classic_search_query.md — the empirical record of how legacy Lucene actually behaved, as ~100 curl comparisons against export.arxiv.org alongside the local equivalent (default operator is OR, prefix-parens distribute, quotes are phrases but not literal, submittedDate matches the Atom published element, etc.). Read this before changing query parsing or Atom date fields — it is the spec that the tests encode.

Stale files — don't follow these

• README.md documents pipenv and nose2 --with-coverage. Both are gone; use poetry + pytest.
• Makefile has PROJECT := feed (copy-paste from arxiv-feed), so make format targets a nonexistent directory; its test target calls nose2. make index and make index-test still work if you swap pipenv run for poetry run.
• search/agent/ does not exist — the Kinesis indexing agent was removed in 68ee63c. bin/start_agent.py, Dockerfile-agent, the agent/localstack services in docker-compose.yml, the KINESIS_* config block, and the whole "Running the indexing agent" section of the README are all dead references.
• Dockerfile and Dockerfile-api still build on arxiv/base:0.16.x with pipenv. Dockerfile-ui and Dockerfile-classic-api are the maintained ones.
• lintstats.sh posts commit statuses to Travis CI, which is no longer used. GitHub Actions (.github/workflows/python-app.yml) runs only poetry run pytest; flake8 is commented out there.
• .pre-commit-config.yaml pins black at rev: stable and pydocstyle at master, and black is not a poetry dependency. Expect pre-commit install to need fixing before it works.
• classic-api.py and classic_api.py are byte-identical duplicates.

docs/ is Sphinx source published to https://arxiv.github.io/arxiv-search by update-docs.sh; it is unrelated to the workspace docs/prs/ work-product convention.