Imported from JoaoTravalini/industrial-fleet-data-platform (AGENTS.md). Install upstream with npx skills add JoaoTravalini/industrial-fleet-data-platform. Copyright stays with the author.

Engineering Instructions

These instructions apply to the entire focused industrial-fleet-data-platform repository.

Repository Scope

• This repository contains the Data Engineering portion of the larger Industrial Fleet Intelligence Platform.
• The authoritative complete implementation remains the full monorepo at https://github.com/JoaoTravalini/industrial-fleet-intelligence.
• Keep this repository self-contained. Runtime code must not import from, shell out to, or assume the full monorepo exists locally.
• Do not add API, frontend, PostgreSQL, ML training, ML inference, anomaly detection, drift monitoring, Ollama, SHAP, MLflow, or model-serving code here.

Code Quality

• Source code, comments, documentation, variables, and user-facing text must be in professional English.
• Use Python 3.12.
• Use type hints for new Python code.
• Prefer explicit configuration and deterministic behavior over hidden environment assumptions.
• Fail clearly instead of silently swallowing validation, parsing, or infrastructure errors.
• Keep changes narrowly scoped to the data-platform contract.

Data Engineering Rules

• Never represent synthetic telemetry as real industrial data.
• Preserve the telemetry event contract exactly unless a deliberate contract migration is documented and tested.
• Keep generation deterministic through explicit seeds, start timestamps, and interval configuration.
• Preserve raw Kafka records in Bronze. Bronze must not deduplicate business events.
• Treat (topic, partition, offset) as Kafka source coordinates.
• Treat event_id as business event identity.
• Treat payload_sha256 as raw payload identity/integrity evidence.
• Silver is the validated canonical business-event layer. It owns schema validation, typing, deduplication, duplicate audit, and quarantine behavior.
• Gold is descriptive analytics only. Do not add failure predictions, anomaly scores, health scores, risk scores, SHAP fields, or model outputs to Gold.
• Keep Parquet as the implemented storage format. Delta Lake is a possible future extension, not part of the current implementation.

Runtime Data

• Do not commit generated Bronze, Silver, Gold, checkpoint, Kafka volume, Spark warehouse, cache, or log data.
• Keep only small intentional fixtures under data/sample/.
• Keep static reports only when they clarify simulator, Kafka, Bronze, Silver, or Gold behavior for reviewers.
• Do not use docker compose down -v as normal cleanup; it deletes named Kafka state.

Docker Safety

• Bind externally exposed local service ports to 127.0.0.1.
• Use pinned image tags, not latest.
• Keep Kafka and Spark local-development infrastructure only.
• Docker-backed validators may require running services; normal unit tests must not require live Kafka or Spark.

Validation

• Run focused tests after code changes:

.\.venv\Scripts\python.exe -m pytest

• Run formatting and lint checks:

.\.venv\Scripts\python.exe -m ruff check --no-cache .
.\.venv\Scripts\python.exe -m ruff format --check --no-cache .

• When Docker is available, docker compose config is the cheapest Compose validation.