asp_plot

Imported from uw-cryo/asp_plot (AGENTS.md). Install upstream with npx skills add uw-cryo/asp_plot. Copyright stays with the author.

AGENTS.md

Guidance for AI coding agents (Claude Code and others) and new contributors. This file holds only what you can't infer from the code: commands, gotchas, external dependencies, and process. For the module-by-module codebase map and design rationale, read ARCHITECTURE.md on demand — and keep both files in sync with the code.

Project Overview

asp_plot is a Python package for visualizing output from the NASA Ames Stereo Pipeline (ASP): diagnostic plots and comprehensive PDF reports covering stereo DEM results, bundle adjustment, CSM camera models, stereo acquisition geometry, DEM galleries, and altimetry comparison/alignment (ICESat-2 for Earth, LOLA for Moon, MOLA for Mars). Handles terrestrial and planetary sensors. Requires Python >= 3.11; published on PyPI and conda-forge. The version lives in pyproject.toml (exposed as asp_plot.__version__).

Development Commands

# Environment (installs the package editable with dev+docs extras)
conda env create -f environment.yml
conda activate asp_plot
pre-commit install                      # REQUIRED for development

# Tests
pytest                                  # all tests; add -s to see print output
pytest tests/test_stereo.py::test_name  # one file / one test

# Lint/format (pre-commit runs these on commit; manually:)
pre-commit run --all-files              # black + flake8 + isort (profile: black)

# Rebuild after changing CLI tools or entry points
pip install -e ".[dev]"

Flake8 config is in .flake8 (extends ignore: E203, E701); pre-commit further ignores E501, E722, E207.

To build the docs locally (Sphinx + MyST; hosted on ReadTheDocs, auto-built on push to main):

# One-time: copy notebooks and figures, fetch the reports, for local preview
mkdir -p docs/examples/notebooks && cp notebooks/**/*.ipynb docs/examples/notebooks/
mkdir -p docs/_extra/examples/figures && cp notebooks/figures/* docs/_extra/examples/figures/
bash docs/fetch_example_reports.sh       # downloads the reports from their Release

sphinx-autobuild docs docs/_build/html --open-browser   # or sphinx-build for a one-off

Example Reports Are Release Assets, Not Committed Files

reports/*.pdf is gitignored. PDFs do not delta-compress, so every regeneration was a wholly new permanent blob — seven report files had become 116 immortal objects and 846 MB of history, most of a 1.8 GB clone (issue #201). They are published as GitHub Release assets instead, on a dated reports-<date> tag that is bumped only when the reports change — deliberately not the per-version releases, since release.yml creates one per pyproject.toml bump and the reports do not change per version.

To publish a regenerated set:

bash reports/regenerate_reports.sh          # gitignored; needs ~/Desktop/asp-plot-examples
gh release create reports-$(date +%F) \
    reports/*.pdf reports/*_figure_selections.yml reports/regenerate_reports.sh \
    --target main --latest=false \
    --title "Example reports, $(date +%F)" --notes "What changed and why."
# then bump REPORTS_RELEASE in docs/fetch_example_reports.sh and commit that one line

Four things matter here:

• Publish the release before pushing the commit that points at it. The docs build uses curl -f, so a tag that does not exist yet fails the build.
• --latest=false, or the reports release takes the "Latest" badge from the current version release.
• Attach the *_figure_selections.yml sidecars too. They are gitignored (they hardcode absolute local paths) but they are what makes regeneration figure-stable and offline (issue #121), so keeping them with the PDFs they produced makes a release self-describing.
• Attach regenerate_reports.sh as well, for the same reason and one more. It is gitignored too, which means it is in no clone and in no backup — a mirror of this repository contains zero objects for it, because git never tracked it. The release is therefore its only durable copy, and a release that carries the reports, the selections that shaped them, and the commands that produced them can be reproduced by someone who has the example datasets and nothing else.

Always regenerate with --reuse-selections so pages stay figure-for-figure comparable and no SlideRule request is made. Reports are capped at 200 effective dpi at their placed width (asp_report --figure-max-dpi, 0 disables), which is what keeps the assets small.

Gotchas

• ASP tools must be on PATH for the workflows that wrap them: stereo/parallel_stereo, bundle_adjust, point2dem, pc_align, geodiff, dg_mosaic (called via run_subprocess_command()). Append the ASP bin directory to PATH rather than prepending: the ASP release bundles its own python, which would shadow your environment's interpreter and break imports.
• Internet is required for basemaps (contextily/Esri tiles), ICESat-2 requests (SlideRule), and ESA WorldCover sampling (public AWS S3 COGs). Tests must not depend on the network — basemap fetches are stubbed (#151).
• Mars altimetry needs the *_pts_csv.csv (with PLANET_RAD), never the *_topo_csv.csv: MOLA TOPOGRAPHY is referenced to the oblate areoid while ASP DEMs use the spherical IAU datum — a latitude-dependent offset up to ~10 km that pc_align cannot remove. The loader rejects the topo file with an explanatory error.
• ASP's mapproject writes no log file; its command is reconstructed from output GeoTIFF metadata (mapproject.py), not parsed from logs like the other tools.
• Attribution vs sensor naming is deliberate: copyright/attribution names the rights-holder (detect_satellite_attribution() → "Vantor" or "Airbus DS"), while readers in the asp_plot/sensors/ package are named for the satellite family (WorldView, Pleiades). Don't reconcile them into one name (#137).
• Package vs CLI naming is a deliberate split (v2.0.0, #165): the package is asp_plot (import asp_plot, pip install asp-plot), the report command is asp_report (asp_plot/cli/asp_report.py). There is no asp_plot console script and no alias — don't "fix" either name to match the other. The other CLIs (stereo_geom, csm_camera_plot, bundle_adjust_cameras, request_planetary_altimetry, gallery, dem_benchmark) are unprefixed.
• There is no "unadjusted" CSM state on disk after a bundle_adjust run — it writes only *.adjusted_state.json for the optimized cameras, while the originals exist solely as vendor XML. To get the before-camera for csm_camera_summary_plot(), re-run with an identity transform: write a 4x4 identity to identity.txt and pass --initial-transform identity.txt --apply-initial-transform-only with the same images/XMLs into a separate -o prefix. That skips iterations, matching, and reports (seconds, not minutes), and the resulting .adjust files are exactly zero translation + identity quaternion, so you can verify the cameras really are untouched. Demonstrated in notebooks/WorldView/worldview_spacenet_ucsd_stereo.ipynb.
• CSM camera differences deliberately diverge from ASP's orbit_plot.py (#53): csm_analysis.read_angles_common_frame() puts both cameras in one satellite body frame estimated from the original ephemeris, instead of estimating a frame per camera from its own positions the way csm_io.read_angles() (the ASP mirror) does. A solver perturbs positions and resamples the ephemeris finer, and over the resulting ~140 m central-difference baseline a 2 m perturbation tilts a per-camera frame by ~0.8° — which is what made the example figure look full of outliers. Angle diffs are also wrapped to [-180, 180). Don't "re-sync" either back to read_angles(); csm_io.py stays a verbatim ASP mirror and is not where this belongs.
• Airbus DIMAP quaternions are scalar-first (Q0 = scalar); they are reordered to the scalar-last q1..q4 layout the roll/pitch/yaw code expects in PleiadesMetadata.getAtt_df(). Don't "fix" the reorder.
• The altimetry outlier cut is a 30-NMAD gate plus 3σ; both values are deliberate. A whole ICESat-2 pass can be cloud (over UCSD one pass was a marine-layer return 150–210 m above ground, 21 % of the sample), which defeats a plain mean ± 3σ cut by inflating σ; the gate removes it. Don't "simplify" the cut to median ± 3 NMAD: on the heavy-tailed residuals a DSM has against ICESat-2 in a city that removes a fifth of the real points and lowers every reported NMAD by a third (_outlier_mask in altimetry_source.py; the case is described in the benchmark notebook's UCSD section). If a benchmark or report shows an RMSE far above its NMAD, look for a cloud pass in the residual map before touching the DEM.
• asp_plot.sensors.rpc.read_rpc() refuses any raster that carries a CRS, on purpose: an RPC describes the raw image grid, and ASP's mapproject copies RPC metadata onto its outputs. However, GDAL sometimes writes a lat/lon geotransform from a NITF's corner coordinates when converting it (one of five CORE3D scenes came out that way), and the reader then calls a raw scene "already map-projected". Strip it with gdal_edit.py -unsetgt -a_srs "" scene.tif.
• SpaceNet CORE3D NITF scenes are JPEG2000, and the ASP 3.8.0-alpha build ships GDAL's JP2OpenJPEG driver as a plugin it never loads: gdalinfo and rasterio both fail with No JPEG2000 capable driver is available. GDAL_DRIVER_PATH=<ASP root>/lib for one gdal_translate to GeoTIFF is enough; the older 3.7.0 build had the driver built in, which is why earlier UCSD runs never hit this.
• ASP multiview triangulation of mapprojected images needs ISISROOT (observed with ASP 3.8.0-alpha, non-ISIS -t pleiades session): the joint triangulation aborts with an uncatchable Isis::IException ($ISISROOT/IsisPreferences was not found) that surfaces as a generic "Failed to run"/killed job. Workaround: export ISISROOT=<ASP install root> (the release bundles IsisPreferences there). Pair runs and raw-image multiview runs are unaffected; full write-up in a PR #155 comment.

External Data Sources

• ICESat-2 ATL06-SR (Earth): requested through the SlideRule API; results cached as parquet next to the report.
• LOLA (Moon) / MOLA (Mars): async queries to the ODE GDS REST API via the request_planetary_altimetry CLI — the user gets a download link by email, then passes the CSV to asp_report --altimetry-csv. Coordinates are east-positive 0–360 longitude, planetocentric latitude.

Testing

Tests are in tests/ with sample data in tests/test_data/ (synthetic rasters, XML camera files, BA CSVs, ICESat-2 parquet, pc_align outputs, jitter data). Most modules have a matching tests/test_<module>.py; test_imports.py verifies everything imports. Some fixture derivatives (e.g. match-point CSVs next to .match files) are gitignored and regenerate during test runs — untracked files appearing under tests/test_data/ after pytest are expected, don't commit them. Example notebooks in notebooks/ are organized by sensor (WorldView, Pleiades, ASTER, LRO_NAC, Mars_MGS, Mars_MRO) — see ARCHITECTURE.md for what each demonstrates.

Versioning and Release Process

Follow semantic versioning. To release:

1. Update version in pyproject.toml
2. Update CHANGELOG.md with version and date
3. Merge to main

The rest is automated: release.yml detects the version bump, creates a GitHub Release + tag, and publishes to PyPI via OIDC trusted publishing; conda-forge's autotick bot then opens a feedstock PR. PyPI trusted publishing and the conda-forge feedstock are already configured (reference recipe in conda-forge-recipe/meta.yaml).

The autotick bot only bumps version and sha256 — it never syncs dependencies or entry points. So whenever you add/remove a runtime dependency in pyproject.toml or add/rename a [project.scripts] entry point, the feedstock's recipe/meta.yaml must be edited by hand in the same release (requirements: run: and build: entry_points: + the matching test: commands:). Otherwise the conda build succeeds and then fails its own test phase, conda-build moves the package to broken/, and nothing is uploaded — PyPI advances while conda-forge silently stalls on the last good version. This is not hypothetical: adding pyyaml in v1.16.0 (#121) went unmirrored and stalled conda-forge at 1.15.1 for five releases (1.16.0 → 1.19.0), with a red ✗ on the feedstock's default branch the whole time. After releasing, check https://anaconda.org/conda-forge/asp-plot actually advanced rather than assuming the bot handled it.

Common File Patterns

ASP output files follow specific naming patterns (find them with the glob_file() utility):

• DEMs: *-DEM.tif or *_dem.tif
• Disparity: *-F.tif
• Match files: *.match (binary) or *__*.txt (plain text, from ASP >= 3.7.0 --matches-as-txt; binary preferred when both exist)
• Bundle adjust residuals: *-initial_residuals_pointmap.csv, *-final_residuals_pointmap.csv
• Log files: *log-bundle_adjust*.txt, *log-stereo*.txt, *log-point2dem*.txt

Multi-view (>2 scene) runs keep only the joint products (*-PC.tif, *-DEM.tif, *-IntersectionErr.tif) at the stereo-directory top level; the per-pair intermediates live one level down in <prefix>-pairN/ (N-L_sub.tif, N-R_sub.tif, N-D_sub.tif, the .match file, N-align-{L,R}.txt, and an N-stereo.default config copy naming that pair's images). Discover them with find_pair_directories() rather than globbing — code that assumes the flat pair layout silently degrades to "missing files" placeholders (#160).