Imported from sfoulad/midad-by-foulad (.skills/SKILL.md). Install upstream with npx skills add sfoulad/midad-by-foulad --skill .skills. Copyright stays with the author.

CrossPoint Reader Development Guide

Project: Open-source e-reader firmware for Xteink X4 (ESP32-C3) Mission: Provide a lightweight, high-performance reading experience focused on EPUB rendering on constrained hardware.

AI Agent Identity and Cognitive Rules

• Role: Senior Embedded Systems Engineer (ESP-IDF/Arduino-ESP32 specialized).
• Primary Constraint: 380KB RAM is the hard ceiling. Stability is non-negotiable.
• Evidence-Based Reasoning: Before proposing a change, you MUST cite the specific file path and line numbers that justify the modification.
• Anti-Hallucination: Do not assume the existence of libraries or ESP-IDF functions. If you are unsure of an API's availability for the ESP32-C3 RISC-V target, check the freeink-sdk source or the FreeInk SDK docs (https://freeink.org/llms.txt for an LLM-readable index) first.
• No Unfounded Claims: Do not claim performance gains or memory savings without explaining the technical mechanism (e.g., DRAM vs IRAM usage).
• Resource Justification: You must justify any new heap allocation (new, malloc, std::vector) or explain why a stack/static alternative was rejected.
• Verification: After suggesting a fix, instruct the user on how to verify it (e.g., monitoring heap via Serial or checking a specific cache file).

Development Environment Awareness

CRITICAL: Detect the host platform at session start to choose appropriate tools and commands.

Platform Detection

# Detect platform (run once per session)
uname -s
# Returns: MINGW64_NT-* (Windows Git Bash), Linux, Darwin (macOS)

Detection Required: Run uname -s at session start to determine platform

Platform-Specific Behaviors

• Windows (Git Bash): Unix commands, C:\ paths in Windows but / in bash, limited glob (use find+xargs)
• Linux/WSL: Full bash, Unix paths, native glob support

Cross-Platform Code Formatting:

./bin/clang-format-fix -g

Never invoke or probe clang-format directly. The repository wrapper is the only sanctioned entry point.

Platform and Hardware Constraints

Hardware Specs

• MCUs: ESP32-C3 (single-core RISC-V @ 160MHz) and ESP32-S3 (sticky, dual-core Xtensa LX7)
• RAM: ~380KB usable on ESP32-C3 (VERY LIMITED - primary project constraint)
  • NO PSRAM on C3.
  • Single Buffer Mode: Only ONE 48KB framebuffer (not double-buffered)
• Flash: 16MB (Instruction storage and static data)
• Display: 800x480 E-Ink (Slow refresh, monochrome, 1-2s full update)
  • Framebuffer: 48,000 bytes (800 × 480 ÷ 8)
• Storage: SD Card (Used for books and aggressive caching)

The Resource Protocol

1. Stack Safety: Limit local function variables to < 256 bytes. The ESP32-C3 default stack is small; use std::unique_ptr or static pools for larger buffers.
2. Heap Fragmentation: Avoid repeated new/delete in loops. Allocate buffers once during onEnter() and reuse them.
3. Flash Persistence: Large constant data (UI strings, lookup tables) MUST be marked static const to stay in Flash (Instruction Bus), freeing DRAM.
4. String Policy: Prohibit std::string and Arduino String in hot paths. Use std::string_view for read-only access and snprintf with fixed char[] buffers for construction.
5. UI Strings: All user-facing text must use the tr() macro (e.g., tr(STR_LOADING)) for i18n support. Never hardcode UI strings directly. For the avoidance of doubt, logging messages (LOG_DBG/LOG_ERR) can be hardcoded, but user-facing text must use tr().
6. constexpr First: Compile-time constants and lookup tables must be constexpr, not just static const. This moves computation to compile time, enables dead-branch elimination, and guarantees flash placement. Use static constexpr for class-level constants.
7. std::vector Pre-allocation: Always call .reserve(N) before any push_back() loop. Each growth event allocates a new block (2×), copies all elements, then frees the old one — three heap operations that fragment DRAM. When the final size is unknown, estimate conservatively.
8. SD Persistence Throttling: Settings, state, credentials, and other PersistableStore JSON files live on SD under /.crosspoint/ through HalStorage; SPIFFS is not mounted. Guard redundant writes and debounce progress saves to avoid serialization, SD I/O, and storageMutex cost.
9. new is not nothrow on ESP32: With -fno-exceptions, bare new that fails calls abort() — it does NOT return nullptr. Always use new (std::nothrow) and null-check the result, or use makeUniqueNoThrow<T>() from lib/Memory/Memory.h. Never write bare new for any fallible allocation.

Project Architecture

Build System: PlatformIO

PlatformIO is BOTH a VS Code extension AND a CLI tool:

1. VS Code Extension (Recommended):

   • Extension ID: platformio.platformio-ide (see .vscode/extensions.json)

   • Provides: Toolbar buttons, IntelliSense, integrated build/upload/monitor

   • Configuration: .vscode/c_cpp_properties.json, .vscode/tasks.json

   • Usage: Click Build (✓), Upload (→), or Monitor (🔌) buttons

2. CLI Tool (pio command):

   • Installation: Python package (typically pip install platformio)

   • Windows Location: C:\Users\<user>\AppData\Local\Programs\Python\Python3xx\Scripts\pio.exe

   • Verify: which pio (Git Bash) or where.exe pio (cmd)

   • Usage: pio run, pio run -t upload, etc.

Configuration Files:

• platformio.ini: Main build configuration (committed to git)
• platformio.local.ini: Local overrides (gitignored, create if needed)
• partitions.csv: ESP32 flash partition layout

Build Environment

• Standard: C++20 (-std=c++2a). No Exceptions, No RTTI.
• Logging: ALWAYS use LOG_INF, LOG_DBG, or LOG_ERR from Logging.h. Raw Serial output is deprecated.
• Environments (in platformio.ini):
  • default: Development (LOG_LEVEL=2, serial enabled)
  • gh_release: Production (LOG_LEVEL=0)
  • gh_release_rc: Release candidate (LOG_LEVEL=1)
  • slim: Minimal build (no serial logging)

Critical Build Flags

These flags in platformio.ini fundamentally affect firmware behavior:

-DEINK_DISPLAY_SINGLE_BUFFER_MODE=1  // Single framebuffer (saves 48KB RAM!)
-DARDUINO_USB_MODE=1                 // Enable USB CDC
-DARDUINO_USB_CDC_ON_BOOT=1          // Serial available immediately at boot
-DXML_CONTEXT_BYTES=1024             // XML parser memory limit (EPUB parsing)
-DUSE_UTF8_LONG_NAMES=1              // SD card long filename support
-DMINIZ_NO_ZLIB_COMPATIBLE_NAMES=1   // Avoid zlib name conflicts
-DXML_GE=0                           // Disable XML general entities (security)
-DDESTRUCTOR_CLOSES_FILE=1           // FsFile destructor auto-closes (SdFat)

DESTRUCTOR_CLOSES_FILE implications:

• SdFat's FsBaseFile destructor calls close() automatically when the object goes out of scope

• Do NOT add explicit file.close() calls for local FsFile variables — the destructor handles it

• Explicit close() is still required in these cases:

  1. Close before delete: Must close before Storage.remove() on the same path

  2. Close before reopen: Must close before reopening the same FsFile variable (e.g., write then reopen for read, or rewrite the same path)

  3. Member variables: FsFile members persist beyond any single function scope, so close at the intended release point (e.g., in onExit())

SINGLE_BUFFER_MODE implications:

• Only ONE framebuffer exists (not double-buffered)
• Grayscale rendering requires temporary buffer allocation (renderer.storeBwBuffer())
• Must call renderer.restoreBwBuffer() to free temporary buffers
• See lib/GfxRenderer/GfxRenderer.cpp:439-440 for malloc usage

Directory Structure

• lib/: Internal libraries (Epub engine, GfxRenderer, UITheme, I18n)
  • lib/hal/: Hardware Abstraction Layer (HalDisplay, HalGPIO, HalStorage)
  • lib/I18n/: Internationalization (translations in translations/*.yaml, generated string tables)
• src/activities/: UI logic using the Activity Lifecycle (onEnter, loop, onExit)
• freeink-sdk/: Low-level SDK (EInkDisplay, InputManager, BatteryMonitor, SDCardManager)
• .crosspoint/: SD-based binary cache for EPUB metadata and pre-rendered layout sections

Hardware Abstraction Layer (HAL)

CRITICAL: Always use HAL classes, NOT SDK classes directly.

Location: lib/hal/

Why HAL?

• Provides consistent error logging per module
• Abstracts SDK implementation details
• Centralizes resource management

Example - HalStorage:

#include <HalStorage.h>

// Use Storage singleton (defined via macro)
HalFile file;
if (Storage.openFileForRead("MODULE", "/path/to/file.bin", file)) {
  // Read from file
  // No file.close() needed — DESTRUCTOR_CLOSES_FILE=1 handles it at scope exit
}

Usage: Use HalFile (the mutex-wrapping handle), NOT raw SdFat FsFile or Arduino File. Do NOT add file.close() for local variables (see DESTRUCTOR_CLOSES_FILE above).

SdFat is not thread-safe; all SD access MUST go through HalStorage:

• SdFat's SdSpiCard tracks SPI bus state with an unsynchronized m_spiActive bool. Two tasks calling SdFat concurrently can confuse that state machine and end with one task calling SPIClass::endTransaction() against a paramLock the other task is holding. That trips FreeRTOS's xTaskPriorityDisinherit assert (tasks.c:5156, pxTCB == pxCurrentTCBs[0]) and panics the system. See SdFat issue #518.
• HalStorage serializes everything via storageMutex. Downstream code uses HalFile (declared in <HalStorage.h>); every method call (read, write, seek, close) takes the mutex. HalFile's destructor also takes the mutex before letting the underlying SdFat FsFile close.
• Never call into SdFat / SdSpiCard / FsBaseFile / SDCardManager / raw FsFile directly — that bypasses the mutex.

Coding Standards

Naming Conventions

• Classes: PascalCase (e.g., EpubReaderActivity)
• Methods/Variables: camelCase (e.g., renderPage())
• Constants: UPPER_SNAKE_CASE (e.g., MAX_BUFFER_SIZE)
• Private Members: memberVariable (no prefix)
• File Names: Match Class names (e.g., EpubReaderActivity.cpp)

Header Guards

• Use #pragma once for all header files.

Comment Style

• Keep comments short and write them for the merged state, as if the code had always worked this way.
• Remove before/after narration, investigation measurements, and rationale that belongs in the commit message.
• Keep only non-obvious mechanism, field/parameter meaning, or the reason a special case exists.

Memory Safety and RAII

• Smart Pointers: Prefer std::unique_ptr.
• RAII: Use destructors for cleanup. Call vTaskDelete() explicitly for deterministic task release. Do NOT call file.close() on local FsFile variables — DESTRUCTOR_CLOSES_FILE=1 handles it at scope exit (see Critical Build Flags).

ESP32-C3 Platform Pitfalls

std::string_view and Null Termination

string_view is not null-terminated. Passing .data() to any C-style API (drawText, snprintf, strcmp, SdFat file paths) is undefined behaviour when the view is a substring or a view of a non-null-terminated buffer.

Rule: string_view is safe only when passing to C++ APIs that accept string_view. For any C API boundary, convert explicitly:

// WRONG - undefined behaviour if view is a substring:
renderer.drawText(font, x, y, myView.data(), true);

// CORRECT - guaranteed null-terminated:
renderer.drawText(font, x, y, std::string(myView).c_str(), true);

// CORRECT - for short strings, use a stack buffer:
char buf[64];
snprintf(buf, sizeof(buf), "%.*s", (int)myView.size(), myView.data());

IRAM_ATTR and Flash Cache Safety

All code runs from flash via the instruction cache. During internal-flash operations such as OTA writes or NVS updates, the cache is briefly suspended. Any code that can execute during this window — ISRs in particular — must reside in IRAM or it will crash silently.

// ISR handler: must be in IRAM
void IRAM_ATTR gpioISR() { ... }

// Data accessed from IRAM_ATTR code: must be in DRAM, never a flash const
static DRAM_ATTR uint32_t isrEventFlags = 0;

Rules:

• All ISR handlers: IRAM_ATTR
• Data read by IRAM_ATTR code: DRAM_ATTR (a flash-resident static const will fault)
• Normal task code does not need IRAM_ATTR

ISR vs Task Shared State

xSemaphoreTake() (mutex) cannot be called from ISR context — it will crash. Use the correct primitive for each communication direction:

RISC-V Alignment

ESP32-C3 faults on unaligned multi-byte loads. Never cast a uint8_t* buffer to a wider pointer type and dereference it directly. Use memcpy for any unaligned read:

// WRONG — faults if buf is not 4-byte aligned:
uint32_t val = *reinterpret_cast<const uint32_t*>(buf);

// CORRECT:
uint32_t val;
memcpy(&val, buf, sizeof(val));

This applies to all cache deserialization code and any raw buffer-to-struct casting. __attribute__((packed)) structs have the same hazard when accessed via member reference.

Template and std::function Bloat

Each template instantiation generates a separate binary copy. std::function<void()> adds ~2–4 KB per unique signature and heap-allocates its closure. Avoid both in library code and any path called from the render loop:

// Avoid — heap-allocating, large binary footprint:
std::function<void()> callback;

// Prefer — zero overhead:
void (*callback)() = nullptr;

// For member function + context (common activity callback pattern):
struct Callback { void* ctx; void (*fn)(void*); };

When a template is necessary, limit instantiations: use explicit template instantiation in a .cpp file to prevent the compiler from generating duplicates across translation units.

Error Handling Philosophy

Source: src/main.cpp:132-143, lib/GfxRenderer/GfxRenderer.cpp:10

Pattern Hierarchy:

1. LOG_ERR + return false (90%): LOG_ERR("MOD", "Failed: %s", reason); return false;
2. LOG_ERR + fallback: LOG_ERR("MOD", "Unavailable"); useDefault();
3. assert(false): Only for fatal "impossible" states (framebuffer missing)
4. ESP.restart(): Only for recovery (OTA complete)

Rules: NO exceptions, NO abort(), ALWAYS log before error return

Heap Buffer Allocation

Prefer makeUniqueNoThrow over malloc. Both are nothrow (return nullptr on OOM rather than calling abort()), but malloc requires a manual free on every return path — a common source of leaks. makeUniqueNoThrow<uint8_t[]>(size) from lib/Memory/Memory.h frees automatically when it goes out of scope.

Preferred pattern:

#include <Memory.h>

auto buffer = makeUniqueNoThrow<uint8_t[]>(bufferSize);
if (!buffer) {
  LOG_ERR("MODULE", "OOM: %d bytes", bufferSize);
  return false;
}

processData(buffer.get(), bufferSize);
// freed automatically — no manual free needed, no leak on early return

malloc or new (std::nothrow) are still acceptable when the buffer must be passed to a C API that takes ownership and frees it itself (e.g., certain SDK callbacks). In that case follow the manual pattern:

auto* buffer = static_cast<uint8_t*>(malloc(bufferSize));  // or new (std::nothrow) uint8_t[bufferSize]
if (!buffer) {
  LOG_ERR("MODULE", "OOM: %d bytes", bufferSize);
  return false;
}
sdkApiThatTakesOwnership(buffer, bufferSize);  // SDK calls free() / delete[]

Rules:

• Prefer makeUniqueNoThrow — automatic cleanup eliminates leak risk on error paths
• ALWAYS check for nullptr after any allocation and LOG_ERR before returning false
• Raw allocation only when a C API takes ownership; document why in a comment

Examples in codebase:

• Memory utilities: Memory.h (makeUniqueNoThrow)
• Cover image buffers: HomeActivity.cpp:166
• Bitmap rendering: GfxRenderer.cpp:439-440

Heap Allocation with new: Always Use makeUniqueNoThrow

CRITICAL: With -fno-exceptions, bare new on OOM calls abort() — it does NOT return nullptr. Always use makeUniqueNoThrow from lib/Memory/Memory.h, which wraps new (std::nothrow) and returns a std::unique_ptr that is null on OOM and automatically frees on scope exit.

Preferred pattern:

#include <Memory.h>

auto obj = makeUniqueNoThrow<MyClass>(args);
if (!obj) { LOG_ERR("MOD", "OOM: MyClass"); return false; }

auto buf = makeUniqueNoThrow<uint8_t[]>(size);
if (!buf) { LOG_ERR("MOD", "OOM: %d bytes", size); return false; }

// Pass to C APIs via .get(); unique_ptr frees automatically on return
someApi(buf.get(), size);

new (std::nothrow) directly is acceptable when the object must be passed to a C API that takes ownership and calls delete itself:

auto* obj = new (std::nothrow) MyClass(args);
if (!obj) { LOG_ERR("MOD", "OOM: MyClass"); return false; }
sdkApiThatTakesOwnership(obj);  // SDK calls delete

Rules:

• Prefer makeUniqueNoThrow — automatic cleanup eliminates leak risk on error paths
• NEVER use bare new — always makeUniqueNoThrow or new (std::nothrow)
• ALWAYS LOG_ERR before returning false on OOM
• Use .get() to pass the raw pointer to C-style APIs; ownership stays with the unique_ptr
• new (std::nothrow) directly only when a C API takes ownership; document why in a comment

Examples in codebase:

• Memory utilities: Memory.h (makeUniqueNoThrow)

UI and Orientation Guidelines

Orientation-Aware Logic

• No Hardcoding: Never assume 800 or 480. Use renderer.getScreenWidth() and renderer.getScreenHeight().
• Viewable Area: Use renderer.getOrientedViewableTRBL() to stay within physical bezel margins.

Logical Button Mapping

Source: src/MappedInputManager.cpp:20-55

Constraint: Physical button positions are fixed on hardware, but their logical functions change based on user settings and screen orientation.

Button Categories:

1. Physical Fixed (Up/Down side buttons):

   • Button::Up → Always HalGPIO::BTN_UP

   • Button::Down → Always HalGPIO::BTN_DOWN

2. User Remappable (Front buttons):

   • Button::Back → Maps to SETTINGS.frontButtonBack (hardware index)

   • Button::Confirm → Maps to SETTINGS.frontButtonConfirm

   • Button::Left → Maps to SETTINGS.frontButtonLeft

   • Button::Right → Maps to SETTINGS.frontButtonRight

3. Reader-Specific (Page navigation with optional swap):

   • Button::PageBack → Uses side button (swappable via SETTINGS.sideButtonLayout)

   • Button::PageForward → Uses side button (swappable)

Implementation:

• Activities use logical buttons (e.g., Button::Confirm)
• MappedInputManager translates to physical hardware buttons
• User can remap front buttons in settings
• Orientation changes handled separately by renderer coordinate transforms

Rule: Always use MappedInputManager::Button::* enums, never raw HalGPIO::BTN_* indices (except in ButtonRemapActivity).

UITheme (The GUI Macro)

• Rule: All UI rendering must go through the GUI macro (UITheme).
• Do not hardcode fonts, colors, or positioning. This ensures orientation-aware layout consistency.

Common Patterns

Singleton Access

Available Singletons:

#define SETTINGS CrossPointSettings::getInstance()  // User settings
#define APP_STATE CrossPointState::getInstance()    // Runtime state
#define GUI UITheme::getInstance()                   // Current theme
#define Storage HalStorage::getInstance()            // SD card I/O
#define I18N I18n::getInstance()                     // Internationalization

Activity Lifecycle and Memory Management

Source: src/main.cpp:132-143

CRITICAL: Activities are heap-allocated and deleted on exit.

// main.cpp navigation pattern
void exitActivity() {
  if (currentActivity) {
    currentActivity->onExit();
    delete currentActivity;  // Activity deleted here!
    currentActivity = nullptr;
  }
}

void enterNewActivity(Activity* activity) {
  currentActivity = activity;  // Heap-allocated activity
  currentActivity->onEnter();
}

Memory Implications:

• Activity navigation = delete old activity + new create next activity
• Any memory allocated in onEnter() MUST be freed in onExit()
• FreeRTOS tasks MUST be deleted in onExit() before activity destruction
• Member FsFile handles MUST be closed in onExit() (local FsFile variables auto-close via destructor)

Activity Pattern:

void onEnter()  { Activity::onEnter(); /* alloc: buffer, tasks */ render(); }
void loop()     { mappedInput.update(); /* handle input */ }
void onExit()   { /* free: vTaskDelete, free buffer, close member FsFiles */ Activity::onExit(); }

Critical: Free resources in reverse order. Delete tasks BEFORE activity destruction.

FreeRTOS Task Guidelines

Source: src/activities/util/KeyboardEntryActivity.cpp:45-50

Pattern: See Activity Lifecycle above. xTaskCreate(&taskTrampoline, "Name", stackSize, this, 1, &handle)

Stack Sizing (in BYTES, not words):

• 2048: Simple rendering (most activities)
• 4096: Network, EPUB parsing
• Monitor: uxTaskGetStackHighWaterMark() if crashes

Rules: Always vTaskDelete() in onExit() before destruction. Use mutex if shared state.

Global Font Loading

Source: src/main.cpp:40-115

All fonts are loaded as global static objects at firmware startup:

• Noto Serif: 12, 14, 16, 18pt (4 styles each: regular, bold, italic, bold-italic)
• Noto Sans: 12, 14, 16, 18pt (4 styles each)
• Ubuntu UI fonts: 10, 12pt (2 styles)

Total: ~80+ global EpdFont and EpdFontFamily objects

Compilation Flag:

#ifndef OMIT_FONTS
  // Most fonts loaded here
#endif

Implications:

• Fonts stored in Flash (marked as static const in lib/EpdFont/builtinFonts/)
• Font rendering data cached in DRAM when first used
• OMIT_FONTS can reduce binary size for minimal builds
• Font IDs defined in src/fontIds.h

Usage:

#include "fontIds.h"

renderer.insertFont(FONT_UI_MEDIUM, ui12FontFamily);
renderer.drawText(FONT_UI_MEDIUM, x, y, "Hello", true);

Testing and Debugging

Build Commands

Via CLI:

# Build firmware (default environment)
pio run

# Build and upload to device
pio run -t upload

# Build specific environment
pio run -e gh_release

# Clean build artifacts
pio run -t clean

Via VS Code:

• Use PlatformIO toolbar: Build (✓), Upload (→), Clean (🗑️)
• Or Command Palette: PlatformIO: Build, PlatformIO: Upload, etc.

Monitoring and Debugging

# Enhanced monitor with color/logging (recommended)
python3 scripts/debugging_monitor.py

# Standard PlatformIO monitor
pio device monitor

Via VS Code: Click Monitor (🔌) button in PlatformIO toolbar

Code Quality

# Static analysis (cppcheck)
pio check

# Format only Git-modified C/C++ files, on every host
./bin/clang-format-fix -g

Do not run raw clang-format or probe it with command -v; use the wrapper even for diagnostics.

Debugging Crashes

Common Crash Causes:

1. Out of Memory (Most common):

   LOG_DBG("MEM", "Free heap: %d bytes", ESP.getFreeHeap());

   • Monitor heap usage throughout activity lifecycle

   • Check if large allocations (>10KB) occur before crash

   • Verify buffers are freed in onExit()

2. Stack Overflow:

   LOG_DBG("TASK", "Stack high water: %d", uxTaskGetStackHighWaterMark(taskHandle));

   • Occurs during deep recursion or large local variables

   • Increase task stack size in xTaskCreate() (2048 → 4096)

   • Move large buffers to heap with malloc

3. Use-After-Free:

   • Activity deleted but task still running

   • Always vTaskDelete() in onExit() BEFORE activity destruction

   • Set pointers to nullptr after free()

4. Corrupt Cache Files:

   • Delete .crosspoint/ directory on SD card

   • Forces clean re-parse of all EPUBs

   • Check file format versions in docs/file-formats.md

5. Watchdog Timeout:

   • Loop/task blocked for >5 seconds

   • Add vTaskDelay(1) in tight loops

   • Check for blocking I/O operations

Verification Steps:

1. Check serial output for stack traces
2. Monitor heap with ESP.getFreeHeap() before/after operations
3. Verify task deletion with task list (vTaskList())
4. Test with LOG_LEVEL=2 (debug logging enabled)

Midad Thin-Fork Architecture — Mandatory

Midad is a thin fork of CrossPoint. Protecting future upstream mergeability is a mandatory architecture requirement, not optional cleanup — every session must treat it that way.

• Read docs/upstream-sync-architecture.md before any architecture-changing work.
• New Midad-specific functionality belongs in Midad-owned files/modules whenever possible. Do not place substantial Midad/Foulad implementation inside CrossPoint-owned files.
• CrossPoint-owned files may contain only the smallest stable integration hook necessary. Before modifying one, explicitly check whether the behavior can live entirely behind a Midad-owned module instead.
• Every PR must identify any upstream-owned files it modifies and justify each one.
• Never manually reconstruct, copy, or cherry-pick a CrossPoint release or feature. CrossPoint updates go through .github/workflows/update-from-crosspoint.yml using a real git merge — never a hand port.
  • One narrow, formally documented exception: scripts/reviewed-upstream-backports.tsv, enforced by thin-fork-guard.sh's verify_reviewed_backport(). Applies ONLY when an already-accepted, official CrossPoint commit fixes something on a branch that has real-merged an adopted baseline, and a full real merge would also pull in a separate, deliberately-deferred, unrelated upstream feature the branch does not want yet (e.g. CrossPoint PR #3061's fix, landing after the unrelated, deferred bbca4886 x4pro/papermono commit — see docs/upstream-sync-architecture.md's "Reviewed upstream backport" section for the full mechanism). Every use requires an exact, content-verified manifest record (named commit, exact base/head blobs) — never a commit range, never a path-level allowlist, never routine synchronization. It is explicitly temporary: the moment a real convergence merge absorbs the same content, the record goes stale and the guard reports it for removal. This exception exists because the guard itself would otherwise have no way to distinguish "genuinely accepted upstream content, deliberately isolated from an unrelated deferred feature" from an ordinary hand port — it does not relax the general rule above for anything else.
• CrossPoint sync PRs must always be merged with "Create a merge commit" — never squash or rebase. Squashing or rebasing a sync PR discards the real merge commit, which breaks the ancestry thin-fork-guard.sh relies on to recognize future sync PRs and to reason about upstream divergence.
• CI enforces this: .github/workflows/thin-fork-guard.yml fails a PR that introduces new fixed-upstream merge-conflict surface, or that diverges an upstream-owned file that was previously byte-identical to upstream. Treat a guard failure as an architecture problem to fix, not a check to route around.
• The same principle applies to crosspoint-simulator (the host-build HAL replacement used by pio run -e simulator), confirmed directly by its maintainer reviewing our upstream PR (crosspoint-reader/crosspoint-simulator#30, closed 2026-08-09) and formalized in that repo's own FORKING.md: generic simulator/platform-emulation fixes (gaps in the ESP-IDF/Arduino emulation layer — esp_http_client.h, Arduino.h, WiFi.h, freertos/, etc. — or bugs in simulator behavior like rendering/storage/threading) go upstream; Midad-specific HAL API changes (anything that only compiles because of a signature Midad's firmware added that CrossPoint's develop doesn't have) stay in a small Midad fork of crosspoint-simulator, never as a hand-patch to the gitignored .pio/libdeps/simulator/ checkout. See the "Local simulator patches" list under Testing and Debugging below for the current concrete instances of each category. The Midad simulator fork itself does not exist yet — see that list's item (a) for status.

Contributing Back to CrossPoint

Midad benefits heavily from CrossPoint, so when we discover a fix or improvement that is genuinely generic and useful to CrossPoint, we should contribute it upstream rather than keeping unnecessary fork-only code. Before carrying a generic fix permanently in Midad, check whether it belongs upstream instead.

Upstream contribution candidates — generic, not Midad-specific:

• Generic bug fixes
• Memory/performance improvements
• ESP-IDF/simulator compatibility fixes (crosspoint-reader/crosspoint-simulator#30 is the model: HTTP_METHOD_DELETE went upstream as a real ESP-IDF emulation gap)
• Generic Arabic/RTL improvements
• Device/HAL improvements that apply to CrossPoint-supported hardware generally, not just Midad's
• Correctness fixes unrelated to Midad services or branding

Stays in Midad, never pushed upstream just to shrink the fork — Midad Cloud, BLE services, Foulad identity/device tracking, branding, apps, or fork-specific HAL APIs (e.g. forceCleanBaseOnHalf — see the "Local simulator patches" list below for why that one was already tried and correctly rejected).

Workflow — never skip straight to publishing:

1. Identify the upstream contribution candidate.
2. Explain why it belongs in CrossPoint rather than Midad.
3. Prepare the proposed patch/commit and draft PR/issue text.
4. Show it to the user for approval.
5. Only after the user explicitly says post/submit/open the PR, publish it to the CrossPoint (or crosspoint-simulator, etc.) repository.

Never open an upstream PR, issue, discussion, or comment automatically. Steps 1–4 are proactive and don't need to wait for the user to ask; step 5 always does — this is a "publish externally" action and stays gated the same way any other outbound action does under this doc's action-authorization rules, no matter how clearly generic the fix looks.

The objective: give useful generic improvements back to CrossPoint while keeping Midad-specific architecture isolated and the fork as small as practical.

Git Workflow and Repository Awareness

Repository Detection Protocol

CRITICAL: ALWAYS verify repository context before git operations. This could be:

• A fork with origin pointing to personal repo, upstream to main repo
• A direct clone with origin pointing to main repo
• Multiple collaborator remotes

Verification Commands (run at session start):

# Check current branch
git branch --show-current

# Check all remotes
git remote -v

# Check working tree status
git status --short

Example Output (forked repository):

origin      https://github.com/<your-username>/crosspoint-reader.git (fetch/push)
upstream    https://github.com/crosspoint-reader/crosspoint-reader.git (fetch/push)

Git Operation Rules

1. Integration branches and PR comparisons target develop, not master or the remote's symbolic HEAD.
2. Never push to any remote or open/close a PR without explicit user approval. Complete local work and any requested local commit, then stop.
3. If the user explicitly approves a push, inspect remotes again and use fork for the feature branch unless the user specifies otherwise.
4. Never add Claude, Codex, or assistant self-attribution as a commit co-author or generated-by trailer.
5. When a change supersedes or adapts another person's PR, verify the original human author from Git/GitHub and add that person as Co-Authored-By; skip bot authors.

Branch Naming Convention

For feature/fix branches:

feature/<short-description>       # New features
fix/<issue-number>-<description>  # Bug fixes
refactor/<component-name>         # Code refactoring
docs/<topic>                      # Documentation updates

Examples:

• feature/sd-download-progress
• fix/123-orientation-crash
• refactor/hal-storage

Commit Message Format

Pattern:

<type>: <short summary (50 chars max)>

<optional detailed description>

Types: feat, fix, refactor, docs, test, chore, perf

Example:

feat: add real-time SD download progress bar

Implements progress tracking for book downloads using
UITheme progress bar component with heap-safe updates.

Tested in all 4 orientations with 5MB+ files.

When to Commit

DO commit when:

• User explicitly requests: "commit these changes"
• Feature is complete and tested on device
• Bug fix is verified working
• Refactoring preserves all functionality
• All tests pass (pio run succeeds)

DO NOT commit when:

• Changes are untested on actual hardware
• Build fails or has warnings
• Experimenting or debugging in progress
• User hasn't explicitly requested commit
• Files excluded by .gitignore would be included — always run git status and cross-check against .gitignore before staging (e.g., *.generated.h, .pio/, compile_commands.json, platformio.local.ini)

Rule: If uncertain, ASK before committing.

Generated Files and Build Artifacts

Files Generated by Build Scripts

NEVER manually edit these files - they are regenerated automatically:

1. HTML Headers (generated by scripts/build_html.py):

   • src/network/html/*.generated.h

   • Source: HTML templates in data/html/ directory

   • Triggered: During PlatformIO pre: build step

   • To modify: Edit source HTML in data/html/, not generated headers

2. I18n Headers (generated by scripts/gen_i18n.py):

   • lib/I18n/I18nKeys.h, lib/I18n/I18nStrings.h, lib/I18n/I18nStrings.cpp

   • Source: YAML translation files in lib/I18n/translations/ (one per language)

   • To modify: Edit source YAML files, then run python scripts/gen_i18n.py lib/I18n/translations lib/I18n/

   • Commit: Source YAML files only. All three generated files (I18nKeys.h, I18nStrings.h, I18nStrings.cpp) are in .gitignore and regenerated at build time.

3. Build Artifacts (in .gitignore):

   • .pio/ - PlatformIO build output

   • build/ - Compiled binaries

   • *.generated.h - Any auto-generated headers

   • compile_commands.json - LSP/IDE metadata

Modifying Generated Content Workflow

To change HTML pages:

1. Edit source: data/html/<pagename>.html
2. Build: pio run (auto-triggers scripts/build_html.py)
3. Generated headers update: src/network/html/<pagename>Html.generated.h
4. Commit ONLY source HTML, NOT generated .generated.h files

To add/modify translations (i18n):

1. Edit or add YAML file: lib/I18n/translations/<language>.yaml

   • Each file must contain: _language_name, _language_code, _order, and STR_* keys

   • English (english.yaml) is the reference; missing keys in other languages fall back to English

2. Run generator: python scripts/gen_i18n.py lib/I18n/translations lib/I18n/

3. Generated files update: I18nKeys.h, I18nStrings.h, I18nStrings.cpp

4. Commit source YAML files only. All three generated files are in .gitignore and regenerated at build time.

To use translated strings in code:

#include <I18n.h>
// Use tr() macro with StrId enum (defined in generated I18nKeys.h)
renderer.drawText(FONT_UI, x, y, tr(STR_LOADING), true);

To add custom fonts:

1. Place source fonts in lib/EpdFont/fontsrc/ (gitignored)
2. Run conversion script (see lib/EpdFont/README)
3. Update global font objects in src/main.cpp:40-115
4. Add font ID constant to src/fontIds.h

Local Development Configuration

platformio.local.ini (Personal Overrides)

Purpose: Personal development settings that should NEVER be committed.

Use Cases:

• Serial port configuration (varies by machine)
• Debug flags for specific testing
• Local build optimizations
• Developer-specific paths

Example platformio.local.ini:

# platformio.local.ini (gitignored)
[env:default]
upload_port = COM7              # Windows: COMx, Linux: /dev/ttyUSBx
monitor_port = COM7

build_flags =
  ${base.build_flags}
  -DMY_DEBUG_FLAG=1             # Personal debug flags
  -DTEST_FEATURE_ENABLED=1

Configuration Hierarchy:

1. platformio.ini - Committed, shared project settings
2. platformio.local.ini - Gitignored, personal overrides
3. Local file extends/overrides base config

Rules:

• NEVER commit platformio.local.ini
• NEVER put personal info (serial ports, credentials) in main platformio.ini
• Use ${base.build_flags} to extend (not replace) base flags

Testing and Verification Workflow

Testing Checklist

AI agent scope (what you CAN verify):

1. ✅ Build: Build once after the last code edit with the relevant pio run target. Do not clean by default, repeat a target that already passed, or rebuild after formatting/comment-only/documentation-only changes.

2. ✅ Quality: pio check when relevant + ./bin/clang-format-fix -g

3. ✅ Format: Commit messages (feat:/fix:), no .gitignore-excluded files staged (e.g., *.generated.h, .pio/, platformio.local.ini)

4. ✅ CI: Fix GitHub Actions failures before review

5. ✅ Code review: Ensure orientation-aware logic is correct in all 4 modes by inspecting switch/case coverage

6. ✅ Simulator smoke test (for any change touching the reader, home, browser, or OPDS flows): pio run -e simulator, then run .pio/build/simulator/program in the background from the repo root and watch its stdout log. ./fs_/ is the virtual SD card (gitignored) — drop test EPUBs there. With macOS Accessibility granted to the terminal host, drive it via osascript/System Events keystrokes (focus process "program" first): ←→ front buttons, ↑↓ page back/forward, Return select, Esc back. Verify through the log lines (activity transitions, section build/deserialize, cover diag) and the artifacts written under fs_/.crosspoint/ (e.g. section .bin version bytes). OPDS flows reach real servers via host curl — always use the Foulad eBooks TEST account for simulator/testing logins: username 11, password 11 (never a real account). Local simulator patches. crosspoint-simulator is fetched from git HEAD and lags this firmware, so .pio/libdeps/simulator/ sometimes needs local edits. All of them live in a gitignored directory: pio clean or a libdeps re-fetch silently reverts every one. If the simulator misbehaves or fails to build, check this list before debugging anything else. This list moves over time as upstream catches up — the entries below were last re-verified directly against live crosspoint-reader/crosspoint-reader develop and crosspoint-simulator main on 2026-08-13; re-check the live repos rather than trusting this list blindly if it's been a while, the same way this pass caught two entries that had already been fixed upstream since they were first documented. See the upstream-vs-fork rule in "Midad Thin-Fork Architecture" above for the general policy this list is an instance of.

   Active Midad-specific patches (currently necessary, not upstream candidates):

   a. Build blocker — displayBuffer arity. The firmware's HalDisplay::displayBuffer takes three arguments (mode, turnOffScreen, forceCleanBaseOnHalf); the simulator's still takes two, so GfxRenderer.cpp fails to compile and the simulator cannot be built at all until patched. Today: add the third parameter to both simulator/src/HalDisplay.h and HalDisplay.cpp and ignore it — the simulator has no e-ink half-refresh base to keep clean, so discarding it is the correct behaviour.

   This was submitted upstream (crosspoint-reader/crosspoint-simulator#30) and explicitly rejected by the maintainer on 2026-08-09: real CrossPoint develop's HalDisplay::displayBuffer is still two-arg (re-confirmed live on 2026-08-13, HEAD cb995809), forceCleanBaseOnHalf exists nowhere in upstream firmware or the simulator, so it's currently Midad-specific HAL surface, not an upstream gap — not a quality rejection of the patch itself, and not a permanent guarantee either: CrossPoint's own fork policy allows a fork-specific API to later become upstream, at which point this local patch (and the fork item below) should disappear. The maintainer's FORKING.md is explicit that this class of change belongs in a fork of the simulator repo, not a hand-patched .pio/libdeps/ checkout and not an upstream PR. Policy: this belongs in a small Midad fork of crosspoint-simulator (not yet created — tracked as a follow-up after the thin-fork guardrails work), with Midad's platformio.ini [env:simulator] pointing lib_deps at that fork instead of upstream. Until that fork exists, this remains a necessary local .pio/libdeps/ patch as a stopgap — not the target state.

   b. Not a bug — the QR code is always blank. The simulator ships stub QR functions (simulator/src/qrcode.cpp) whose qrcode_getModule() returns 0 unconditionally, so QrUtils::drawQrCode faithfully draws nothing. The encoder and draw path are fine on device. Do not chase this; verify QR rendering on hardware, or by encoding the payload against the real ricmoo/QRCode in a standalone host program.

   Resolved upstream — no local patch needed. All three of these were previously documented here as local patches or fork-only HAL surface; re-checking live crosspoint-reader/crosspoint-reader develop and crosspoint-simulator main on 2026-08-13 found all three already fixed/present upstream. If you're carrying a local patch for any of these, drop it — the git-HEAD libdeps fetch already has the fix.

   • HTTP_METHOD_DELETE. Was missing from the simulator's esp_http_client.h enum (HttpDownloader's DELETE path, added for device sign-out, failed to compile against it) — submitted as one half of PR #30 above and accepted as-is, landing on crosspoint-simulator's main branch as a genuine platform-emulation gap (real ESP-IDF defines it; the stub belongs there regardless of whether Midad's firmware calls it yet). Note: the maintainer later added HTTP_METHOD_PATCH ahead of DELETE in that same enum to match real ESP-IDF's ordering (PUT, PATCH, DELETE), so HTTP_METHOD_DELETE's numeric value shifted — never hardcode this enum's underlying integer value anywhere; use the name.

   • O_WRONLY vs O_RDWR in HalStorage::openFileForWrite. Was a fidelity gap: the simulator opened write handles O_WRONLY while the device's SDCardManager opens O_RDWR, so any page rendered while a section build was still in progress deserialized as EMPTY (Section::loadPageDuringBuild() reads pages back through the still-open build handle). crosspoint-simulator main's HalStorage.cpp already opens O_RDWR | O_CREAT | O_TRUNC, with an inline comment giving the identical rationale — this was a genuine simulator-behavior bug, not Midad HAL surface, so it was always a legitimate upstream fix and is now merged there.

   • Async-refresh HAL surface. HalDisplay::displayBufferAsync/ waitRefreshComplete/supportsAsyncRefresh now exist in both real CrossPoint firmware's develop and crosspoint-simulator's main — this API is no longer Midad-specific and needs no local simulator patch. Consume the upstream implementation rather than maintaining a local stub. One detail worth knowing: upstream's HalDisplay::supportsAsyncRefresh() currently returns false in the simulator (its displayBufferAsync() just calls the existing blocking refreshDisplay() — SDL presentation has no genuine overlap to advertise). That's correct upstream behavior, not a gap — don't recreate a local stub that returns true here unless a specific test actually needs the simulator to claim overlap support, and if so, justify that separately rather than assuming the old rationale for a since-removed local patch still applies.

   Not patchable — the web server is a separate, out-of-sync file, not a stub. src/network/CrossPointWebServer.cpp/WebDAVHandler.cpp are NOT compiled from the firmware's own src/network/ for simulator builds; crosspoint-simulator's own CLAUDE.md notes it "still owns those host-side compatibility paths" and vendors a full separate copy at simulator/src/CrossPointWebServer.cpp (a top-level src/ file there, not under network/). Any route/handler added to the firmware's CrossPointWebServer.cpp — e.g. the /dictionaries page and /api/dictionaries/* endpoints — silently 404 in the simulator even though /api/status and other long-standing routes work fine, because the simulator's binary is running its own older copy of the file that never learned about the new route. Confirmed via grep -c "<new route's symbol>" .pio/libdeps/simulator/simulator/src/CrossPointWebServer.cpp returning 0. Unlike the HAL-stub gaps above, this isn't a one-line patch — the simulator's copy would need the new handlers ported into it too, which is real effort disproportionate to most web-UI changes. Don't chase a 404 here as a firmware bug; verify new CrossPointWebServer routes on-device instead.

   Seeing the screen without a GUI. SDL windows are not always exposed to the macOS Accessibility API, and when they aren't, osascript keystrokes silently go nowhere and the simulator cannot be driven at all (System Events reports "no windows"). Two workarounds, both more reliable than keystrokes: dump each frame by adding a getenv("SIM_FB_DUMP") PBM write at the top of HalDisplay::refreshDisplay (the framebuffer is 1bpp with 1=white, so invert for PBM's 1=black) and convert with sips -s format png; and reach a specific activity by temporarily replaceActivity-ing into it from main.cpp behind a -D flag rather than navigating there. Note that an activity forced into a state this way still runs its own loop(), which can transition it out from under you — a half-populated struct will send it down an error path within a poll interval or two, so populate enough of the state to make it sit still.

Human tester scope (flag these for the user): 7. 🔲 Device: Test on hardware 8. 🔲 Orientations: Verify all 4 modes (Portrait/Inverted/Landscape CW/CCW) 9. 🔲 Heap: ESP.getFreeHeap() > 50KB, no leaks — the simulator has a flat 1MB heap and CANNOT reproduce fragmentation/OOM behavior; memory verdicts are hardware-only 10. 🔲 Cache: If EPUB modified, delete .crosspoint/ and verify re-parse 11. 🔲 E-ink refresh: ghosting/fading/waveform behavior — the simulator's SDL window renders instantly and can't show these

CI/CD Pipeline Awareness

GitHub Actions run automatically on pull requests:

Rules:

• Fix CI failures BEFORE requesting review
• CI runs on: Push to PR, PR updates
• Format check fails → Run ./bin/clang-format-fix -g
• Build check fails → Fix compile errors

Serial Monitoring and Live Debugging

Serial Monitor Options

1. Enhanced: python3 scripts/debugging_monitor.py (color-coded, recommended)
2. Standard: pio device monitor (basic, no colors)
3. VS Code: Monitor (🔌) button (IDE-integrated)

Live Debugging Patterns

Heap: LOG_DBG("MEM", "Free: %d", ESP.getFreeHeap()); (every 5s in loop) Stack: uxTaskGetStackHighWaterMark(nullptr) (< 512 bytes → increase stack) Flush: logSerial.flush(); (force output before crash)

Port Detection: Windows: mode | Linux: ls /dev/ttyUSB* /dev/ttyACM* or dmesg | grep tty

Cache Management and Invalidation

Cache Structure on SD Card

Location: .crosspoint/ directory on SD card root

Structure: .crosspoint/epub_<hash>/{book.bin, progress.bin, cover.bmp, sections/*.bin}

Hash: std::hash<std::string>{}(filepath) → Moving/renaming file = new hash = lost progress

Cache Invalidation Rules

Cache is automatically invalidated when:

1. File format version changes (see docs/file-formats.md)

   • book.bin version number incremented

   • section.bin version number incremented

2. Render settings change:

   • Font family or size (SETTINGS.fontFamily, SETTINGS.fontSize)

   • Line spacing (SETTINGS.lineSpacing)

   • Paragraph spacing (SETTINGS.extraParagraphSpacing)

   • Screen margins (SETTINGS.screenMargin)

3. Viewport dimensions change:

   • Screen orientation change

   • Display resolution change

4. Book file modified:

   • Moved, renamed, or content changed (new hash)

Manual Cache Clear (safe operations):

# Delete ALL caches (forces full regeneration)
rm -rf /path/to/sd/.crosspoint/

# Delete specific book cache
rm -rf /path/to/sd/.crosspoint/epub_<hash>/

# Keep progress, delete only rendered sections
rm -rf /path/to/sd/.crosspoint/epub_<hash>/sections/

When to Clear Cache:

• EPUB parsing errors after code changes to lib/Epub/
• Corrupt rendering (missing text, wrong layout)
• Testing cache generation logic
• After modifying:
  • lib/Epub/Epub/Section.cpp
  • lib/Epub/Epub/BookMetadataCache.cpp
  • Render settings in CrossPointSettings

Cache File Format Versioning

Source: lib/Epub/Epub/Section.cpp, lib/Epub/Epub/BookMetadataCache.cpp

Current Versions (as of docs/file-formats.md):

• book.bin: Version 7 (metadata structure)
• section.bin: Version 25 (layout structure)

Version Increment Rules:

1. ALWAYS increment version BEFORE changing binary structure
2. Version mismatch → Cache auto-invalidated and regenerated
3. Document format changes in docs/file-formats.md

Example (incrementing section format version):

// lib/Epub/Epub/Section.cpp
static constexpr uint8_t SECTION_FILE_VERSION = 26;  // Was 25, now 26

// Add new field to structure
struct PageLine {
  // ... existing fields ...
  uint16_t newField;  // New field added
};

Philosophy: We are building a dedicated e-reader, not a Swiss Army knife. If a feature adds RAM pressure without significantly improving the reading experience, it is Out of Scope.