Imported from manitaggarwal/readwise (
AGENTS.md). Install upstream withnpx skills add manitaggarwal/readwise. Copyright stays with the author.
AGENTS.md
Guidance for AI coding agents working in this repository.
Project overview
Self-hosted Readwise clone: upload Markdown clippings, extract highlights via OpenAI, store in SQLite, browse/search in a web UI, and resurface highlights by email on a cron schedule.
Stack: Node.js 18+ / Express 5, SQLite (sqlite3), vanilla HTML/CSS/JS frontend, OpenAI SDK, Nodemailer, node-cron. No frontend bundler.
Repository layout
| Path | Role |
|---|---|
server.js |
Express app, API routes, cron scheduling, email resurfacing; exports app for tests |
database.js |
SQLite schema init (highlights, settings); singleton db connection |
ingest.js |
extractHighlights() via OpenAI; CLI: node ingest.js <file.md> |
public/index.html |
Entire web UI (inline CSS + JS) |
server.test.js |
Jest + Supertest API tests |
spec.md |
Feature spec and inner-loop test expectations |
goal.md |
Autonomous optimization loop rules (extraction F1 eval) |
eval/, harness/ |
Read-only eval datasets and scoring scripts |
.agents/skills/ |
Optional agent skills (ponytail, lfd-design) |
Commands
# Native dev (requires .env with OPENAI_API_KEY and SMTP_*)
npm install
npm start # node server.js, port 3000
npm test # jest (supertest against exported app)
# Docker (DB persisted to ./data, uploads to ./uploads)
cp .env.example .env # fill in keys first
docker compose up -d --build
docker compose logs -f
# Extraction eval (optimization loop only; needs .env)
bash harness/score.sh # dev set F1
bash harness/score.sh --holdout # holdout F1
bash harness/probe.sh # order-sensitivity checkLocal DB defaults to database.sqlite in project root. Docker sets DB_PATH=/app/data/database.sqlite.
Architecture
public/index.html ──fetch──► server.js (Express)
├── database.js (SQLite)
├── ingest.js (OpenAI on upload)
└── nodemailer + node-cron (resurface)
- Single-process monolith. All HTTP routes live in
server.js. Import./databasefor DB access; import{ extractHighlights }from./ingestfor LLM parsing. - Static frontend.
express.static('public')servesindex.html. Frontend usesconst API_URL = ''(same-origin relative paths). - Cron lifecycle.
setupCron()readssettingsrowid=1, stops any prior job, schedules fromtime+frequency(dailyorweeklySunday). Called on server start and afterPOST /settings. - Upload flow. Multer writes to
uploads/, file is read,extractHighlights()runs, rows inserted, temp file deleted. - Resurface.
runResurface()picks 3 random highlights wherelast_reviewedis null or older than 1 day, emails them, updateslast_reviewedandreview_count. Target email from settings row, elseSMTP_USER.
API endpoints
| Method | Path | Notes |
|---|---|---|
POST |
/upload |
Multipart file; returns { success, count, highlights } |
POST |
/highlights |
JSON body; text required; fields: text, attribution, source, author |
GET |
/highlights |
Paginated: ?page=1&limit=10&search=. Response: { data, total, page, limit, totalPages } |
DELETE |
/highlights/:id |
Returns { success: true } or 404 |
GET |
/settings |
{ email, time, frequency } |
POST |
/settings |
Updates row and reschedules cron; { success: true } |
POST |
/resurface |
Manual email trigger |
Highlight objects use both attribution (who said the quote) and author (work author). The UI sends both on manual entry.
Database
Schema in database.js:
highlights:id,text,attribution,source,author,created_at,last_reviewed,review_countsettings: singleton rowid=1withemail,time(HH:MM),frequency(daily|weekly)
Uses callback-style sqlite3 API (db.run, db.get, db.all, db.prepare). No ORM.
Frontend (public/index.html)
All UI changes go in this single file unless introducing a build step (not currently used).
Patterns in use:
- Card-based layout with
.card,.highlightlist items - Debounced search (
handleSearch, 300ms) callingGET /highlights?page=&search= - Pagination rendered into
#paginationControls - Theme system: CSS custom properties on
:root;data-theme="light"|"dark"on<html>for manual override;@media (prefers-color-scheme: dark)when no override (Auto). Preference stored aslocalStorage.theme(system|light|dark). Inline<head>script applies saved theme before paint to avoid flash. Toggle buttons callsetTheme();initTheme()syncs active state on load.
When adding styles, extend the existing CSS variables (--bg, --text, --card-bg, etc.) in both light and dark blocks rather than hardcoding colors.
Extraction (ingest.js)
- Model:
gpt-5.4-miniwithresponse_format: { type: "json_object" } - System prompt requires verbatim quote text and fields
text,attribution,source,author - Returns
parsed.highlightsarray; empty array if key missing - Shared by
/uploadand CLI stdout
Optimization loop
If working under goal.md (extraction quality optimization):
- Read-only:
eval/,harness/,docs/,goal.md - Editable:
.js,.ts,.html,.css,.pyelsewhere - Log hypotheses to
LOG.mdbefore each change; commit each cycle - Do not read
eval/holdout/expected.json - Harness enforces capacity caps (no large JSON lookup tables) and eval overlap detection — violations return
VOID: constraint violation
For normal feature work, ignore the optimization loop unless explicitly asked.
Environment variables
From .env.example:
OPENAI_API_KEY— required for upload/extractionSMTP_HOST,SMTP_PORT,SMTP_USER,SMTP_PASS,SMTP_FROM— required for resurfacingPORT— default 3000DB_PATH— set by Docker Compose; optional locally
Conventions
- CommonJS throughout (
require/module.exports).server.jsonly listens whenrequire.main === module. - Error responses:
{ error: string }with appropriate HTTP status. - No TypeScript, no React. Keep the frontend vanilla unless the task explicitly introduces tooling.
- Tests:
supertesthits the exported Expressappwithout starting a listener. NoteGET /highlightsreturns a paginated object, not a bare array — update tests accordingly when touching that endpoint. - Gitignored runtime paths:
node_modules/,uploads/,.env,database.sqlite,data/,harness/actual_*.json
Agent skills
Project-local skills in .agents/skills/:
ponytail— minimal/YAGNI implementation modeponytail-review,ponytail-audit,ponytail-debt— complexity and debt analysislfd-design— design optimization harnesses andgoal.mdfor/goalruns
Read the relevant SKILL.md when the user invokes those modes.