Imported from FlanaganSe/typing-ml (AGENTS.md). Install upstream with npx skills add FlanaganSe/typing-ml. Copyright stays with the author.

AGENTS.md

Read this file before editing any code in this repo. It encodes the rules that, if broken, will cost a later milestone more time to recover than this milestone will save you. Sources: .claude/plans/typing-form-trainer-mvp.md (Global Rules), docs/research.md §3 / §11.6 / §13.

CLAUDE.md is a symlink to this file.

The one-minute summary

• The pipeline is pure. Scoring never sees a DOM node, a MediaStream, a navigator, or a MediaPipe object.
• The mirror flip lives in exactly one place (the homography, landing in M2). Zero scaleX(-1) in CSS. Ever.
• Pin every version at install time via Context7 or npm. Do not copy versions from prose.
• KeyboardEvent.code is the identifier. KeyboardEvent.key is display copy only.
• performance.now() timestamps only. Branded type PerfTimeMs lands in M0b. Never mix with Date.now().
• "Unknown" / ambiguous is a valid verdict. Do not force a binary answer on low-confidence evidence.

Pinned-package policy

1. Before adding or upgrading a dep, resolve the current version via Context7 (mcp__context7__resolve-library-id → mcp__context7__query-docs) or via npm view <pkg> version. Training-data version numbers are stale.
2. Record the exact version in package.json (no ^/~ prefixes for the critical deps below).
3. Pin @mediapipe/tasks-vision to an exact patch version when it is added in M1a. The installed version must match what the app is tested against on reference hardware.
4. Pin the hand_landmarker.task model by SHA-256 in this file when it is downloaded in M1a. CI verifies the hash of the served file.

Pinned MediaPipe assets (pinned 2026-04-16, M1a)

package:  @mediapipe/tasks-vision@0.10.34     (exact; no ^/~)
wasm dir: apps/web/public/mediapipe/wasm/     (self-hosted; fetched from jsdelivr @0.10.34 at install time)
wasm set: vision_wasm_module_internal.{js,wasm}
          (ES-module variant only — see `useModule: true` in mediapipe-provider.ts
          and the `self.import` polyfill in apps/web/src/worker/vision.worker.ts;
          the ES-module variant is SIMD-only, documented in tasks-vision's
          FilesetResolver.forVisionTasks docstring)
model:    apps/web/public/models/hand_landmarker.task
source:   https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task
sha256:   fbc2a30080c3c557093b5ddfc334698132eb341044ccee322ccf8bcf3607cde1
size:     7,819,105 bytes

The .task file and the WASM bundle are not committed — they are fetched at pnpm install time by scripts/fetch-vision-assets.mjs (source of truth: apps/web/public/models/manifest.json). CI step check-model-hash is fail-closed: a hash drift fails the build.

CI assertion strategy for MediaPipe landmarker output (Q4 — committed, M1a partial)

Q4 of research §17 asks whether CPU and GPU delegates diverge enough that CI cannot assert exact landmark equality. M1a shipped the measurement tooling (DetectOnly page); the first number is in docs/validation-2026-04-16.md.

M1a measurement (2026-04-16, partial — not same-clip):

• PCK@0.05 CPU↔GPU = 3.9 % (threshold mean 11.9 px, 2562 comparisons)
• Methodology: 5 s CPU capture → delegate flip → 5 s GPU capture on a held pose. Natural hand motion between the two windows dominates the divergence signal, so this is a lower bound on agreement, not a clean delegate-divergence number. Research §17 Q4's proper protocol requires a same-clip replay (WebM decode through both delegates frame-for-frame); that tooling lands in M1b (packages/replay-harness/src/decode-webm.ts).

Committed CI band (M1a, intentionally wide):

• Landmark XY: ±15 px tolerance
• Confidence: ±0.10
• Detection rate: ≥ 70 % of frames

Rationale: Δ > 2 %, so we are in the band branch. The band is sized wide enough to absorb both real CPU↔GPU divergence and the M1a methodology weakness, so a clean Chromium replay of the M1b fixtures always passes. A real regression (landmarks off by tens of pixels, confidence collapsed, hands reported when absent) still fails.

Post-M1b re-measurement: M1b re-runs Q4 via same-clip WebM decode and may propose tightening this band. Any tightening updates this block in place and logs the change in docs/decision-log.md.

Dragons (from research §13 — the CI-enforced rules)

These are the mistakes that have killed similar products. Each maps to a specific enforcement point.

1. Pure-pipeline boundary. packages/pipeline/** must not import document, window, navigator, @mediapipe/*, or any DOM / MediaStream / MediaPipe symbol. Enforced in two places:
   • ESLint no-restricted-imports + no-restricted-globals under files: ['packages/pipeline/**'] in eslint.config.js.
   • CI rg grep against packages/pipeline/src. If either fires, the fix is to move code into packages/vision or apps/web, not to disable the rule.
2. One horizontal flip. The camera-to-keyboard mirror correction lives in the homography (M2). There is zero transform: scaleX(-1) in CSS, on <video>, or anywhere else in the rendering layer. A unit test in M2 asserts this; grep guards it until then.
3. @mediapipe/hands is banned. Deprecated package. Use @mediapipe/tasks-vision (pinned). Enforced project-wide via ESLint no-restricted-imports.
4. MediaPipe z is never press-depth truth. It is noisy and wrist-relative. Press detection uses 2D fingertip position and y-descent only.
5. No custom model training in MVP. Homography is the keyboard detector. A "just a small YOLO" PR is out of scope; log in docs/decision-log.md and escalate to the plan's §15 decision framework before training anything.
6. KeyboardEvent.code everywhere, .key never. code is the physical key identifier (KeyF, Space, Digit6). key is for UI display of the typed character.
7. Timestamps are performance.now()-origin PerfTimeMs. Never mix with Date.now(). The branded type lands in M0b and will make this a type error.
8. Frame adapter at the app/worker boundary takes VideoFrame / ImageBitmap / canvas-like sources, not MediaStream. The pure pipeline takes HandFrame. Any conversion lives in packages/vision or apps/web.
9. preventDefault() on key events is off-limits. The app observes keys; it does not consume them.

FingerId — canonical list (do not invent variants)

LThumb LIndex LMiddle LRing LPinky
RThumb RIndex RMiddle RRing RPinky
None

Modifiers map to 'Ignore' in the fingering profile — they are intentionally out of scope for MVP form feedback.

Ambiguous keys (do not emit wrong_finger for standard index-finger variation): KeyB, KeyY, Digit6.

Package boundaries (one-line rules)

• packages/core-types — branded types, enums, Zod schemas. Zero runtime deps.
• packages/pipeline — pure. Inputs: HandFrame + key events + Calibration + FingeringProfile. Outputs: FingerAttribution + quality events. No DOM, no navigator, no MediaPipe, no raw pixels.
• packages/vision — MediaPipe adapter + frame-source conversion + OneEuroFilter + hand identity. May touch browser frame primitives. Does not leak MediaPipe objects downstream.
• packages/keyboard-model — US ANSI QWERTY geometry, canonical fingering, homography math.
• packages/replay-harness — load key events + HandFrame traces, feed pipeline, emit metrics.
• packages/fixtures — synthetic JSON fixtures and real captures (committed in M1b).
• apps/web — React shell, getUserMedia, rVFC frame loop, calibration UI, overlays, webcam-off mode.

Test and verification commands

• pnpm install — --frozen-lockfile in CI. Lockfile is source of truth.
• pnpm typecheck — TS strict across all workspaces.
• pnpm lint — ESLint flat config. Dragons 1 + 3 enforced here.
• pnpm test — Vitest across Node-only code (scorer, types, fixtures). No browser required.
• pnpm test:vision — Chromium-based vision replay. No-op in M0a; lands in M1b.
• pnpm dev — Vite dev server over HTTPS via vite-plugin-mkcert. Requires mkcert -install once on the host.

CI also runs a rg boundary grep against packages/pipeline/src as defense-in-depth for dragon rule 1.

Timestamp hygiene (M0b preview)

All timestamps in this system — keydowns, frame presentation times, scoring windows — share the performance.now() origin. event.timeStamp on KeyboardEvent and presentationTime from requestVideoFrameCallback both live on this clock already; do not convert, do not subtract an epoch.

When the branded PerfTimeMs type lands (M0b), mixing with number or Date.now() becomes a type error. Do not cast around it.

How to deviate

Small deviation (e.g., "ESLint 9 flat config replaces .eslintrc.cjs"): add a dated entry to docs/decision-log.md with context, decision, evidence, affected milestone. Then proceed.

Large deviation (change package names, re-architect the monorepo, drop HTTPS, skip a dragon rule): stop. That is a plan change, not a deviation. Surface it to the human before coding.