Imported from Prerit108/EDA_Visulasier (files(2)/AGENTS.md). Install upstream with npx skills add Prerit108/EDA_Visulasier --skill files(2). Copyright stays with the author.

AGENTS.md — DataPulse AI

This file gives AI coding agents (e.g. Claude Code, Cursor, Copilot Workspace) the context needed to work correctly in this repo. Read this before generating or editing any code.

Project Summary

DataPulse AI is a full-stack Gen AI web app that performs interactive EDA and guided data cleaning on uploaded CSV/Parquet/JSON files. The core design principle: all statistics and data transformations run locally via Polars (deterministic, 0 tokens); LangChain + Claude is used only to narrate and suggest based on already-computed profile data — never to compute a statistic, never to see raw data rows.

Read prd.md, ARCHITECTURE.md, and DATA_MODEL.md before implementing new features — they define scope, system design, and schemas respectively. Don't deviate from the Profile/Action schemas in DATA_MODEL.md without updating that file first.

Tech Stack (do not substitute without discussion)

• Backend: Python, FastAPI, uvicorn — deployed on Render
• Data engine: Polars as the primary DataFrame library (chosen for memory efficiency and native Parquet support on Render's constrained tiers — see ARCHITECTURE.md §3.3). pandas/numpy are used only as a scoped conversion boundary for specific scikit-learn-based methods (e.g. KNN imputation) — never as the default engine.
• LLM orchestration: LangChain, using ChatAnthropic from langchain-anthropic for all model calls.
• Frontend: React (Vite build) — deployed on Vercel
• Charting: a JS charting library (Plotly.js or Recharts) — charts are rendered client-side from JSON aggregates; the backend never renders images.
• Streaming: Server-Sent Events (SSE), not WebSockets.
• Containerization: Dockerfile for the backend service (Render).

Repo Conventions

• Backend lives in backend/, frontend in frontend/. They are deployed independently (Render + Vercel) and communicate only via HTTPS/SSE over the documented API contract — never assume they share a filesystem or process.
• New cleaning strategies go in backend/engine/cleaning_registry.py, following the registry pattern in DATA_MODEL.md §8. Do not hardcode transform logic inline in route handlers.
• Every Profile object emitted by the profiling engine must conform to the schema in DATA_MODEL.md §4. If a field doesn't apply to a given column type, omit it consistently (e.g. categorical columns don't get outliers), matching the pattern already used for numeric vs. categorical columns in the schema.
• The summary/suggestions fields from the narration chain are the only output the LLM is allowed to produce. It must never be asked to compute or alter a statistic — that always happens in the Data Engine first.

Secrets & Environment Variables

• ANTHROPIC_API_KEY must be read from environment variables only — set as a Render environment variable in production, loaded via python-dotenv locally.
• Never hardcode API keys in source files.
• Never place API keys in any frontend file — the frontend only ever calls the Render backend, never the Anthropic API directly.
• VITE_API_BASE_URL (frontend → backend URL) is the only environment-dependent value the frontend needs; set it via Vercel project environment variables, not hardcoded.
• .env must be listed in .gitignore in both backend/ and frontend/. Maintain .env.example files with variable names but no real values.

Token & Cost Discipline

• One narration call per state change (initial upload, or after a single cleaning action) — never a call per column, per chart, or in a loop.
• Always set an explicit max_tokens cap on narration calls (see prd.md §6 and implementation_plan.md Phase 4 for the target range).
• Never send raw data rows to the LLM — only the compact profile/diff JSON defined in DATA_MODEL.md §6. If a new feature seems to require sending more data to the LLM, flag it before implementing — the answer is usually "compute a smaller aggregate locally instead."
• If adding a new feature that would require additional LLM calls, state the expected token cost impact before implementing it.

Testing Expectations

• Every cleaning strategy function should be unit-testable in isolation against a known before/after Polars DataFrame, with no network or LLM dependency.
• The profiling engine should be tested against deliberately messy sample datasets (missing values, outliers, imbalance, duplicates) to confirm correct detection.
• Validate narration responses against the expected Pydantic schema before returning them to the frontend; never let a malformed LLM response crash a request — fall back to a templated summary instead (see ARCHITECTURE.md §3.5).

What NOT to do

• Do not default to pandas for the main data engine — use Polars, and only convert to pandas/numpy for the specific scikit-learn calls that require it, scoped to the minimum columns necessary.
• Do not render charts as server-side images (matplotlib/seaborn) — return chart-ready JSON and let the frontend render with a JS charting library.
• Do not use WebSockets — SSE is the chosen streaming transport for this project.
• Do not introduce a persistent database or user auth in MVP scope — session state is in-memory with TTL cleanup for v1 (see ARCHITECTURE.md §5); persistence is a documented future feature (prd.md §11B/D), not part of the current build.
• Do not use localStorage/sessionStorage if any part of this project is rendered as a Claude.ai Artifact — keep state in memory/JS variables in that context.
• Do not change the Profile schema, Action schema, or narration contract without first updating DATA_MODEL.md to match — that file is the source of truth, not the code.
• Do not let the LLM see or handle raw dataset rows under any circumstance, including for future features like natural-language querying — the LLM may generate a filter expression, but that expression must always be executed locally against the data, never interpreted by the LLM itself.

When Implementing a New Feature from the Future Roadmap

1. Check prd.md §11 and implementation_plan.md Phase 9 first — it may already be scoped there.
2. Confirm whether it needs a DATA_MODEL.md schema extension (§9 lists anticipated ones) — update that file as part of the same change, not after.
3. Preserve the local-compute-first principle: new statistics or transforms are computed in Polars (or a scoped scikit-learn conversion); the LLM only narrates or performs genuinely language-dependent tasks (e.g. translating a natural-language query into a filter expression) — never raw numeric computation.