greenland

Imported from powera/greenland (AGENTS.md). Install upstream with npx skills add powera/greenland. Copyright stays with the author.

This project primarily uses Python, with a PYTHONROOT of src/ . Key style/test notes.

• Use type hinting in all new Python code - and add missing type hints to any code other than src/benchmarks .
• Don't re-use variable names (like "key") in the same function for different purposes (to avoid type / initialization errors).
• Use GREENLAND_TEST_MODE=1 as an env variable before any Python calls that are "tests" or not specifically confirmed to change data. Use GREENLAND_DISABLE_LLM=1 for any calls which can modify local data but should not use remote LLM-powered APIs. Only run without one of these two settings after explicit user approval.
• Always use absolute imports (e.g., "from agents.common_args import") rather than relative updir imports (e.g., "from ..common_args import").
• To run scripts, always use PYTHONPATH and never use cd commands: PYTHONPATH=src python src/agents/dramblys.py --help
• If you are running without GREENLAND_DISABLE_LLM, you should always persist to the local database; if it will be useful also write the stdout/stderr to temp files.
• Before "risky" local database changes, make a copy of the SQLITE database. Include the date and a 10-20char explanation of the task; ie. linguistics.sqlite.bak-20260101-add-sentences
• Tests are in src/tests. Any changes to src/clients or src/storage/crud require tests. Changes to src/barsukas should not have tests unless requested by the user.
• Before creating a Git commit, always run black and mypy on modified Python files to ensure code quality and type correctness.
• data/release is generally generated by Barsukas or script outputs; it should only be directly edited after explicit instruction (and you should warn the user that modifying the database first is the normal pattern). It is sometimes useful to read these files to get a sense of what content is in the database - it is faster and less dense than doing sqlite queries.

== Purpose and code structure ==

The main purpose of the project is to create a multilingual linguistic database, and to generate files for the Trakaido language-learning app.

Use DataSourceConfig (defined in src/storage/backend/config.py) to pass configuration (db_path, model_name, debug, backend_type, etc.) to agents and other components. Do not pass db_path or similar parameters directly.

src/barsukas is the main web UX used by humans to interact with the database.

src/agents contains scripts to do bulk operations against the database, generally making LLM calls. Each agent is named with a Lithuanian animal name.

Do not run commands that make live Wikidata/Wikipedia API calls (e.g. resolving Q-ids, fetching concept seeds) without first confirming with the developer.

src/storage contains the SQLAlchemy schema for the main database; the default location is data/wordfreq/linguistics.sqlite .

src/storage/translation_helpers.py contains all language code manipulation functions and constants (LLM_FIELD_TO_LANG_CODE, LANG_CODE_TO_LLM_FIELD, convert_llm_response_to_lang_codes, etc.). Do not create local language mappings - import from translation_helpers.py.

Regional variants live in src/langtools/dialect_overrides.py, which is the source of truth for what a dialect code means. There are two kinds:

• Storage dialects (zh-tw, es-419, pt-br) are ordinary languages in every other respect - their own LemmaTranslation rows, LLM prompt config, release column, and audio. Adding one means four matching registrations: the registry entry with translation_target=True, LANGUAGE_FIELDS + LLM_FIELD_TO_LANG_CODE + LANGUAGE_HIERARCHY in translation_helpers.py, an AVAILABLE_TRANSLATION_LANGUAGES entry in wordfreq/translation/constants.py, and prompts/audio/{word,sentence}/.txt. A test in src/tests/storage/test_translation_helpers_languages.py enforces the set. There is no "fallback" from a storage dialect to the main version.
• Presentation dialects (es-mx, fr-ca, en-gb) store no text of their own. They carry a prompt note and a TTS locale, and read the variety named by get_translation_language() - es-mx is es-419's words in a Mexican accent. Prefer one of these to a new storage dialect whenever the two varieties would hold near-identical lemma-level vocabulary: there is no point storing the same words twice to say the same thing.

src/clients/ contains all code to access LLMs. The system was built around the expectation that different small local models would run for different tasks; currently it is expected that a remote ChatGPT/Claude/Gemini is used.

src/audiotools contains the production audio pipeline: TTS generation (OpenAI, OuteTTS) for flashcard audio, and the S3 upload path. Run its scripts the usual way, e.g. PYTHONPATH=src python src/audiotools/gen_lithuanian_word_audio.py --help The acoustic-analysis research tooling (qualityreview/, stirna.py, the calibration sample generators) stays in the audio/ submodule; that submodule imports clients.audio from this repo, not the reverse.

== Extended notes ==

Set GREENLAND_DISABLE_LLM=1 to hard-block every live LLM call:

GREENLAND_DISABLE_LLM=1 PYTHONPATH=src python src/agents/dramblys.py ...

Every backend then raises clients.lib.LLMCallsDisabledError instead of sending its request. The check sits at each backend's outbound-request method, not on a wrapper: backends define their own generate_chat/warm_model and callers hold client objects directly, so a guard on any higher layer can be routed around. Use it whenever an agent should run only its mechanical/rule-based paths (e.g. langtools.en form generation) - the run fails loudly rather than quietly falling back to a paid model. It overrides GREENLAND_ALLOW_LIVE_LLM, which only opts a test out of the separate pytest guard and must never re-enable a disabled backend.

GREENLAND_TEST_MODE=1 is the stricter, second kill switch:

GREENLAND_TEST_MODE=1 PYTHONPATH=src python src/agents/vieversys.py ...

It blocks every LLM call exactly as GREENLAND_DISABLE_LLM does - keyless backends such as a local Ollama included, since the point is that no model runs, not merely that no key is spent - and additionally blocks any read of the keys/ directory. A run under test mode therefore cannot load a credential of any kind: LLM, TTS, S3, or the postgres password. The two switches are distinct and DISABLE_LLM is the weaker one: an agent may legitimately need an S3 or database credential while running with LLM calls off.

Credential reads are blocked under pytest as well, so the suite cannot load a real secret by accident; GREENLAND_ALLOW_LIVE_KEYS=1 opts a deliberate test out, and (as with the LLM guard) an explicit GREENLAND_TEST_MODE=1 overrides that opt-out. Tests must pass a double rather than a real uploader, which is why the audiotools.s3_ops helpers take the uploader as an argument instead of reaching for a singleton.

When you do a live LLM test run of a new agent or pipeline phase, persist the results to the local database. The call has already been paid for, and the point of the run is to see the data land in the real schema - a script that prints its output and exits verifies only that the LLM replied, leaving the storage path (column types, coercion helpers, NULL handling, the UI render) untested and the spend wasted. Prefer the ordinary persistence path the real callers use (e.g. sentences.translation.store_translation_results) over bespoke session writes, so the test run exercises the same code production does. Ask before making the call, not before saving what it returns.

For agent CLI scripts that should be runnable directly, add this at the top (before any local imports), adjusting the number of .parent calls to reach src/: import sys from pathlib import Path if str(Path(file).parent.parent) not in sys.path: sys.path.insert(0, str(Path(file).parent.parent))

New one-off migrations live in the root-level migrations/ directory and use a YYYYMMDD_ filename prefix so their order is visible. Despite not being in src/ , they should set the python path to do imports like code in src does. Use DataSourceConfig for database selection, and be safe to rerun.

When writing HTML templates, try to avoid inline CSS/JS; use separate files. Also, always use ordinary form submits for POST data - do not do an AJAX-based submission. Avoid using disappearing UX elements most of the time.

== Testing ==

There are three test targets, run via ./run_tests.sh :

./run_tests.sh smoke 14 tests, ~1s - run frequently ./run_tests.sh affected tests covering your changed files; depends ./run_tests.sh all 2872 tests, ~45s - run only when ready to finish task

A "base" target of ~500 tests / ~10s is still wanted, but does not exist yet; base and the old portable both error out naming the replacement.

Extra arguments pass through to pytest: ./run_tests.sh all -k combined_rank -x

smoke asserts only that the tree imports and the barsukas app starts. It is deliberately shallow: the recurring failure in this repo is a module being moved while something that named it as a string was not updated, and imports catch that immediately. Mark new smoke tests with @pytest.mark.smoke and keep the total under ~20 - it is worthless if it stops being fast. Tests needing a database, an LLM client, or fixtures belong in the full suite instead.

"affected" uses src/tests/affected_map.py to turn the directories you changed into the test directories that exercise them. This is what should be run normally after changes. "all" runs everything - it should only be done when ready to have the user push to Github. It should run after black/mypy and other pre-commit work is done. It needs to be updated if there is a new top-level directory or major refactor/dependency changes.

After cloning, enable pre-commit hooks to automatically check black formatting: git config core.hooksPath .githooks