zyra

Imported from NOAA-GSL/zyra (AGENTS.md). Install upstream with npx skills add NOAA-GSL/zyra. Copyright stays with the author.

Agent Workflow and Guardrails

Purpose: This repository is agent-friendly. Follow these rules to keep changes safe, consistent, and easy to review.

Core Rules

• Lint first: Before making any code changes, run:
  • poetry run ruff format . && poetry run ruff check .
  • Fix all issues so ruff check returns clean. Only then proceed.
• Small, surgical edits: Change only what’s needed. Keep file/identifier names stable unless the task explicitly requires refactors.
• Tests: Prefer running focused tests for the area you changed:
  • poetry run pytest -q -k <pattern> (or the full suite if requested).
• No secrets: Never commit credentials or tokens. Prefer environment variables and documented configuration paths.
• Paths: Avoid absolute paths. Use configurable env vars (e.g., DATA_DIR) and importlib.resources for packaged assets.
• Style: Follow the project’s coding style and naming conventions (see README and repository guidelines).

Pre-commit Hooks

• Pre-commit is enabled. Config: https://github.com/NOAA-GSL/zyra/blob/main/.pre-commit-config.yaml
• One-time setup:
  • poetry install --with dev
  • poetry run pre-commit install
• Run hooks manually on all files: poetry run pre-commit run -a
• Hooks enforce Ruff formatting/linting (authoritative list in the config above).
• DCO: All commits must include a Signed-off-by line (see CONTRIBUTING). Add -s to git commit or enable global sign-off: git config --global format.signoff true.
• Hosted/ephemeral sessions (Claude Code on the web and similar) clone the repo fresh into a container where the one-time setup above has not run, so .git/hooks/ is empty and none of these hooks fire. .claude/hooks/session-start.sh installs the DCO commit-msg hook on session start so unsigned commits fail locally instead of at the PR. It installs only that hook — the rest shell out to poetry run, which needs a provisioned Poetry environment the container may not have.

Agent Checklist (before returning code)

• Ran poetry run ruff format . && poetry run ruff check . and confirmed clean
• Ran targeted tests (or full suite if requested) and confirmed passing
• Limited scope to requested changes; updated or added documentation if needed
• Called out any assumptions, follow-ups, or external system requirements (e.g., FFmpeg)

Notes for CI and PRs

• CI should run poetry run ruff format --check . && poetry run ruff check . to ensure compliance. DCO sign-off is enforced by the GitHub DCO app on pull requests.
• Group related changes into single PRs with a clear description and run instructions.

Repository Guidelines

Project Structure & Module Organization

• Code lives under src/zyra/:
  • connectors/: source/destination integrations and CLI registrars (connectors.ingest, connectors.egress, with shared backends/). Prefer these over legacy acquisition/ for new work.
  • processing/: data processing (e.g., GRIB/NetCDF/GeoTIFF) exposed via CLI.
  • visualization/: visualization commands (static, animated, interactive); CLI registration lives alongside commands.
  • transform/: lightweight helpers (metadata scans, enrich/merge, dataset JSON updates).
  • api/: FastAPI service exposing selected CLI functionality (see zyra.api_cli).
  • wizard/: optional assistant UX; ships JSON resources under src/zyra/wizard/*.json.
  • utils/: shared helpers/utilities (credentials, dates, files, images, I/O).
  • assets/: packaged resources (e.g., basemaps/overlays under assets/images/). Access via importlib.resources (zyra.assets).
  • cli.py: root CLI entry (zyra); api_cli.py: API service CLI (zyra-cli).

Assistant/Wizard Resources

• Wizard resources (JSON) are packaged under src/zyra/wizard/*.json and may be used to drive interactive flows.
• If a CLI capabilities file is present in the repo, keep it in sync with CLI changes; otherwise, prefer runtime discovery and module docstrings for accuracy.

Build, Test, and Development

• Install: poetry install --with dev (use extras as needed: -E connectors -E processing -E visualization or --all-extras).
• Shell: poetry shell.
• Run: poetry run python path/to/script.py or poetry run python -m module.
• Lint/format: poetry run ruff format . && poetry run ruff check ..
• Tests: poetry run pytest -q (filter with -k pattern).
• System deps: certain flows require FFmpeg (ffmpeg, ffprobe).

Coding Style & Naming

• Python 3.10+; UTF-8; Unix newlines; 4-space indent.
• Names: files/functions in snake_case; classes in PascalCase; constants in UPPER_CASE.
• Imports grouped: stdlib, third-party, local.
• Formatting and linting via Ruff (replaces Black/Isort/Flake8 in this repo).

Testing Guidelines

• Tests live under tests/; files named test_*.py, functions test_*.
• Prefer small fixtures and sample assets for data-heavy flows.

Commit & PR Guidelines

• Commits: imperative, concise (e.g., "Add GRIB parser"); group related changes.
• PRs: clear scope and rationale; link issues; include run instructions and screenshots for visual changes.
• Checks: pass Ruff format/lint; run relevant sample scripts if modified.

Security & Configuration

• Do not commit secrets. Prefer IAM roles or env vars (e.g., used by S3 connectors).
• Avoid hard-coded absolute paths; prefer env-configurable paths (e.g., DATA_DIR).
• Use importlib.resources for packaged assets under zyra.assets.
• Pin dependencies when adding new ones; document any system deps (e.g., FFmpeg, PROJ/GEOS).

Dependency Management (Poetry)

• Sources of truth: pyproject.toml and poetry.lock.
• Export for pip workflows if needed:
  • poetry export -f requirements.txt --output requirements.txt --without-hashes
  • Include dev: poetry export -f requirements.txt --with dev -o requirements-dev.txt

Documentation Sources

• Synced wiki in repo (for offline/background reading): /docs/source/wiki/ — this is a cron-synced mirror of the GitHub Wiki. Prefer this path when you need background information by browsing the code repository.
• Wiki (live, high-level docs, patterns, overview): https://github.com/NOAA-GSL/zyra/wiki
• API Reference (module/CLI docs): https://noaa-gsl.github.io/zyra/
• Key wiki pages to ground answers:
  • Workflow Stages: https://github.com/NOAA-GSL/zyra/wiki/Workflow-Stages
  • Stage Examples: https://github.com/NOAA-GSL/zyra/wiki/Stage-Examples
  • Install & Extras: https://github.com/NOAA-GSL/zyra/wiki/Install-Extras
• Pipeline Patterns: https://github.com/NOAA-GSL/zyra/wiki/Pipeline-Patterns

Notes

• The /docs/source/wiki/ content is generated via a scheduled sync from the GitHub Wiki. Do not hand-edit files in that folder unless the sync process specifically instructs otherwise. If you see a discrepancy between the repo copy and the live wiki, defer to the live wiki as the source of truth and file an issue to correct the mirror.

Documentation contribution guidelines (where to put updates)

• Keep the top-level README concise. For detailed usage and options, prefer module-level READMEs and the wiki.
• Module READMEs live next to code and should contain focused examples and flags:
  • Connectors (ingest/egress/discovery):
    • src/zyra/connectors/ingest/README.md
    • src/zyra/connectors/egress/README.md
    • src/zyra/connectors/discovery/README.md
  • Processing: src/zyra/processing/README.md
  • Visualization: src/zyra/visualization/README.md
  • Transform: src/zyra/transform/README.md
  • API service: src/zyra/api/README.md
  • MCP tools: src/zyra/api/mcp_tools/README.md
• Sphinx includes these module READMEs under "Module READMEs" in docs/source/index.rst. If you add a new README, include it there.
• Keep function/class docstrings current; autodoc consumes them for the API reference pages.
• Use the GitHub Wiki for narrative guides and high-level design. The mirrored copy under docs/source/wiki/ is not edited directly.

When In Doubt: Decision Flow

• CLI usage or flags
  • Check generated docs: https://noaa-gsl.github.io/zyra/api/zyra.cli.html
  • Run local help where possible: poetry run zyra --help (or subcommand --help).
  • Cross-check examples in Stage-Examples.md.
• Background/context
  • Read the synced wiki in /docs/source/wiki/.
  • If stale or missing, read the live wiki: https://github.com/NOAA-GSL/zyra/wiki
• Implementation details
  • Search code under src/zyra/ (connectors, processing, visualization, transform, api, utils).
• Roadmap/status
  • Issues (enhancements/bugs): https://github.com/NOAA-GSL/zyra/issues
  • Discussions (design/ideas): https://github.com/NOAA-GSL/zyra/discussions
• Security & policy
  • Zyra-API-Security-Quickstart.md, Privacy-and-Data-Usage-Best-Practices-for-Zyra.md, Logging-in-Zyra.md
• Deploy/run
  • Zyra-Containers-Overview-and-Usage.md; API routers/endpoints page

Quick Test Loop (suggested)

# Format + lint (required)
poetry run ruff format . && poetry run ruff check .

# Focused tests
poetry run pytest -q -k <pattern>

# Sanity: help text prints and exits
poetry run zyra --help

Stage Naming & Aliases (for CLI and docs)

• Preferred: import → process → simulate → decide → visualize → narrate → verify → export
• Aliases: acquire/ingest → process/transform → visualize/render → export/disseminate (legacy: decimate)
• Implemented today: acquire (http|s3|ftp|vimeo), process (decode-grib2|extract-variable|convert-format), visualize (heatmap|contour|animate|compose-video|interactive), export (local|s3|ftp|post|vimeo)
• Streaming: many commands accept - (stdin/stdout) to support piping.