Imported from bj-data-eng/data-engine (AGENTS.md). Install upstream with npx skills add bj-data-eng/data-engine. Copyright stays with the author.

Agent Guide

This repository is a Python 3.14 package for the Data Engine workflow runtime, GUI, and daemon. Use this guide when making automated changes in the repo.

Project Shape

• Runtime code lives under src/data_engine/.
• Tests live under tests/.
• Installer scripts live under INSTALL/.
• Smoke-data generation lives under scripts/ and src/data_engine/devtools/.
• Keep untracked persisted reference notes under workspaces/_local/ when something should remain available locally without becoming tracked repo content.
• Generated local data/workspaces are intentionally ignored: data/, data2/, and workspaces/.
• Build artifacts are ignored: build/, dist/, *.egg-info/, and generated docs under src/data_engine/docs/html/.

Environment

• Use the repo virtualenv on Windows: .\.venv\Scripts\python.exe.
• The package requires Python >=3.14.
• VS Code provisioning should point to the interpreter that launched data-engine. On Windows, normalize pythonw.exe to the sibling python.exe.

Testing

• Full suite:

.\.venv\Scripts\python.exe -m pytest -q

• Qt-focused suite:

.\.venv\Scripts\python.exe -m pytest tests\gui\qt -q

• Daemon-focused suite:

.\.venv\Scripts\python.exe -m pytest tests\daemon -q

• Lint and docstring checks:

.\.venv\Scripts\ruff.exe check .
.\.venv\Scripts\pydoclint.exe src\data_engine

• Packaging check:

.\.venv\Scripts\python.exe -m build
.\.venv\Scripts\python.exe -m twine check dist\*

• When instrumentation has been discussed and explicitly chosen for a debugging session, flip the dev instrumentation env var default in code to 1 for the duration of that work and flip it back when done.
• For that toggle-only instrumentation-default change, do not run tests just to verify the flip; we already know that change is safe on its own.

Parallel Worktrees

• Use semi-permanent worker lanes for high-throughput parallel work: ../data-engine-worktrees/lane-1 through ../data-engine-worktrees/lane-6.
• Roles: the coordinator owns slicing, architecture, integration, final tests, and commits. Worker agents own bounded implementation or audit tasks inside one assigned lane.
• Treat main after the latest accepted commit as the source of truth. Before a new parallel batch, reset every lane worktree to that commit, confirm it is clean, and after each accepted integration commit run git reset --hard <accepted-commit> plus git clean -fd inside every lane.
• Prefer fresh worker agents per task while reusing the same lane. Keep prompts small: objective, lane path, branch name, owned files/modules, tests to run, and expected report format. Do not include broad repo history unless directly needed.
• Give each worker a disjoint write scope. Use stable lane themes when helpful to reduce context reload: runtime/state, scheduler/daemon host, UI boundary, platform compatibility, authoring surface/tests, and audit-only.
• Keep the semi-permanent lane branches generic: codex/lane-1 through codex/lane-6.
• Worker task branches should be created from the lane branch and be task-scoped, using names like codex/lane-1-runtime-audit. After integration, reset the lane branch to the accepted coordinator commit before the next task.
• During refactors, finish the migration through callers, read models, command gating, tests, and UI surfaces instead of stopping at one local fix or leaving mixed old/new paths in place. Do not add internal compatibility shims just to preserve old project-internal call shapes; update internal callers to the new boundary instead. Preserve stability for the author-facing surface: flows, flow context, and data_engine.helpers.
• For speed, worker lanes may use the main repo venv at ../../data-engine/.venv for python -m pytest, ruff, and pydoclint when the command runs from the worker worktree root.
• Do not rely on the main repo console scripts, such as data-engine.exe, to test worker-lane code; console entry points may resolve to the installed checkout. Use python -m ... from the worker root, or create a lane-local .venv when testing packaging, console scripts, editable installs, or dependency changes.
• Worker outputs should be patch-first, not broad narratives: changed files, short summary, tests run with results, intentional deviations, and blockers only. Audit-only outputs should be file:line, issue, and recommended action. The coordinator integrates from diffs, not prose: inspect git diff --stat, review relevant file diffs, selectively apply accepted changes to main, run focused tests, then the full suite when runtime behavior changed.

Windows And Unix Compatibility

• This project is actively tested on Windows. Do not add Unix-only subprocess calls such as ps, shell-specific path assumptions, or POSIX-only daemon semantics without a Windows path.
• Keep Unix/macOS behavior intact when fixing Windows issues. Prefer platform-specific branches with tests for both paths.
• Use data_engine.platform.processes for local process listing, PID liveness, process-tree termination, Windows launcher-process collapsing, and Windows subprocess creation flags.
• Use data_engine.platform.interpreters for python.exe / pythonw.exe selection and host-concrete interpreter paths.
• Use data_engine.platform.paths for generic path display, stable absolute paths, path identity text, TOML-safe path text, and platform-aware sort keys. workspace_models re-exports some of these for compatibility, but new non-workspace code should import from platform.paths.
• Windows venv launchers can produce parent/child pythonw.exe process pairs. Daemon diagnostics should collapse launcher shims and report the real Data Engine process.
• Do not use DETACHED_PROCESS for Windows daemon launch unless retesting live daemon lifetime and terminal-window behavior. The stable path uses Windows creation flags that avoid console windows while keeping the daemon reachable.
• Windows daemon PID liveness must not rely on Unix ps output. Use the shared platform-aware process inspection helpers.

Workspace And Runtime State

• Shared workspace state lives inside a workspace under .workspace_state/.
• Machine-local runtime state lives under the local app data runtime artifacts directory.
• When moving runtime/control behavior to a new architecture, remove redundant fallback paths as you go. Do not leave old session/history/snapshot heuristics in place beside the new event-stream or service-owned truth unless a temporary bridge is explicitly documented and scheduled for removal.
• The GUI must not create real runtime SQLite bindings for the synthetic unconfigured workspace placeholder.
• Prefer explicit runtime_cache_ledger and runtime_control_ledger names in new internal code. Keep the public RuntimeLedger and runtime_ledger aliases only where needed for API/test compatibility.
• Client-session tracking is important for ephemeral daemon lifetime. If the UI flickers between "has control" and disconnected, inspect the selected workspace control DB and daemon log before changing UI code.
• Workspace provisioning is target-workspace specific. The Settings view has its own workspace selector and should make the provisioning target explicit.

Generated Smoke Data

• Use the smoke-data generator when live-testing flows:

.\.venv\Scripts\python.exe scripts\generate_smoke_data.py --root . --workspace-id example_workspace --workspace-id docs2

• Do not commit generated files from data/, data2/, or workspaces/.
• If large workbook generation is interrupted, clean up partial files before using the workspace for behavior tests.

Packaging

• Current distribution name is py-data-engine.
• Prefer docstring-first API documentation for author-facing code. Put parameter details, return behavior, validation rules, and copyable examples beside the class/function implementation so VS Code hover help and Sphinx autodoc stay aligned. Keep guide pages for concepts, workflows, and cross-API recipes instead of duplicating API reference text.
• Write documentation by describing what the API is, what it does, and when to use it. For AI-authored docs, lead with the concrete contract, behavior, and intended use. Avoid contrast-first phrasing such as "this is not..." or broad negative lists; the set of things an API is not is infinite.
• Before publishing, verify:

git status --short
.\.venv\Scripts\python.exe -m build
.\.venv\Scripts\python.exe -m twine check dist\*

• Keep version values centralized in src/data_engine/platform/identity.py; package metadata, daemon constants, and Sphinx config should read from APP_VERSION.

Git Hygiene

• Check git status --short before and after edits.
• Do not stage or commit ignored local workspaces, smoke data, build outputs, machine-local settings, or other runtime residue such as local databases, logs, debug artifacts, temp exports, or repro files.
• Do not commit machine-local absolute paths from your development environment. This includes repo-root paths, user-profile paths, local app-data paths, worktree paths, and interpreter paths embedded in docs, samples, UI mock data, or scaffolding code. Prefer relative paths, obvious placeholders, or runtime-resolved paths instead.
• Do not commit secrets or host identity data. Keep API keys, auth headers, cookies, signed URLs, machine IDs, hostnames, usernames, email addresses, and copied production-like identifiers out of the repo unless they are clearly fake and intentional test fixtures.
• When adding or refreshing examples, sample payloads, UI shell data, or screenshots, sanity-check the diff for local path, sensitive-data, and runtime-residue leakage before commit.
• Do not revert user edits unless explicitly asked.