x-bookmarks-rag

Imported from sudoanmol/x-bookmarks-rag (AGENTS.md). Install upstream with npx skills add sudoanmol/x-bookmarks-rag. Copyright stays with the author.

x-bookmarks-rag

Natural-language search over the owner's X (Twitter) bookmarks. It captures bookmarks with a real browser, extracts the full text behind articles and links, embeds everything locally, and searches with hybrid retrieval.

No X API. Everything runs on this machine, except two remote services: Groq on its free tier, and Modal (owner's free credits) for GPU captions and transcripts.

This file is the handoff. Read all of it before you change anything. Many rules below cost hours to learn. The reasons are given so you can tell when a rule stops applying.

1. Rules you must not break

These come from the owner. They override your defaults.

The owner writes and expects ASD-STE100 Simplified Technical English: short sentences, active voice, one meaning per word.

The session file

~/.config/x-bookmarks/state.json (mode 0600) holds a live X login. It sits outside the repo on purpose.

• Never copy it into the repo.
• Never print its contents.
• Only X Article pages may use it. Every external page gets a browser context with no storage state, so a third-party site never sees the login.

2. Commands

uv run xbm login      # opens a browser; the owner signs in by hand
uv run xbm sync       # every stage below, in order, on what is new
uv run xbm status     # coverage per stage, and readiness of each service
uv run xbm capture    # capture new bookmarks since the watermark
uv run xbm inspect    # report what the captured data contains
uv run xbm normalize  # rebuild the DB from raw pages on disk
uv run xbm extract    # fetch full text behind articles and links
uv run xbm caption    # vision model on a Modal GPU (--limit, --retry)
uv run xbm transcribe # Whisper on Modal GPUs (--limit, --retry)
uv run xbm translate  # Groq gpt-oss-120b over foreign chunks
uv run xbm index      # chunk and embed  (--rebuild, --no-* exclusions)
uv run xbm search "..."  # hybrid search  (-n, --author, --source)

uv run pytest -q      # 137 tests, all offline, ~1s

Ollama must be running, with embeddinggemma pulled. That is the only model the project needs.

3. Architecture

Data flows one way. Each stage only reads what the stage before it wrote.

X GraphQL  ->  capture  ->  raw pages (JSON on disk)
                              |
                           parse + normalize
                              |
                           bookmarks / authors / media / links
                              |
                    extract (Playwright + Defuddle)  ->  documents
                    caption (Modal + vLLM)           ->  captions
                    transcribe (Modal + Whisper)     ->  documents (kind video)
                              |
                           chunk  ->  embed  ->  chunks + chunk_vec + chunk_fts
                              |
                           search (RRF over vector + BM25)

Two connection functions. Do not mix them.

• db.connect() — plain SQLite. Use for everything normal.
• index.connect() — loads the sqlite-vec extension. Any query touching chunk_vec fails with no such module: vec0 without it.

4. Data model

All tables live in db.py. data/bookmarks.db, WAL mode, busy_timeout=30000.

Two design points worth keeping:

• Soft delete. Un-bookmarking sets removed_at, so captions survive.
• sort_index, not post time. It orders by bookmark time, which is what incremental sync needs.

5. Current state (all verified, not estimated)

Layers 1 and 2, image captions, and video transcripts are done and working. Search returns real passages from extracted pages, OCR text from images, and timestamped transcript text, not just post text.

The unusable documents are mostly "thin extraction" (under the word floor), plus 10 binary targets (PDFs). links.url is stored under normalize_url(), the same key as documents.url, so the two join directly.

6. Decisions already made. Do not re-open these.

Embedding model: embeddinggemma. Benchmarked against 4 alternatives on 55 generated queries over the real corpus.

At 55 queries the 95% band is about ±0.13, so no model separates. The decision was made on speed: 3,437 chunks embed in about 2 minutes, where a 4B model needs 30 to 60. The benchmark measured dense retrieval alone; the real system also runs BM25, which narrows the gap further. The rejected models were deleted to free 17.4 GB.

Depth-1 crawling: none. The owner chose this after seeing the numbers: 5,342 outbound URLs, about 2.2 hours, and roughly 7x index growth, dominated by GitHub repository navigation and documentation sidebars. Nothing is lost — the outbound links are still inside documents.body, so this can be turned on later without re-fetching.

No LLM answer layer. The owner will use this mostly through MCP, where the agent is already the model. Pre-summarizing would compress the evidence, and the agent would then summarize the summary. Retrieval returns passages; the caller reasons.

One vector space for all content types, rather than separate indexes.

Groq for enrichment, not embeddings. Groq serves no embedding models. translate.py uses openai/gpt-oss-120b with reasoning_effort: low. The free tier allows 8,000 tokens per minute, prompt and reply together, so text goes in paragraph-aligned pieces of 1,500 characters and a 429 waits on retry-after.

Translation is per chunk, cached by text hash. xbm translate builds the same chunks xbm index would, and asks about a chunk when its X tag is not English or its letters are mostly non-Latin. X's tag is wrong on most short posts ("ro", "de", "pt" on English text), so the model replies ENGLISH for those and the cache stores NULL. index.build puts the English in chunks.text and the original in chunks.source_text. Of 35 candidates, 13 were really foreign: 4 Chinese and 3 Japanese posts or quotes, their screenshots, a Chinese gist and repo page, a Japanese video, a Spanish post.

Captions run on Modal, not Groq. The Groq free tier (8,000 tokens per minute, serial) needed days for 600 photos. Modal runs the same model, Qwen/Qwen3.8-27B-FP8, under vLLM on one H100: 583 photos in one run, about 15 minutes of GPU and roughly $1. The app is ephemeral (app.run()), so it stops when xbm caption exits. The owner pays for every GPU second: after any Modal run, confirm modal container list --json prints [].

Transcripts run on Modal too, and are documents rows. faster-whisper large-v3-turbo on up to 8 L4s did all 51.2 hours in under 6 minutes. This replaced the planned Groq + local mlx-whisper split: an M3 Pro would need about 2 hours. A transcript is a documents row with kind = 'video', keyed by https://x.com/<handle>/status/<id>/video/<n>, with one <p> per minute led by [m:ss]. So chunking, search, and read_document (paging and query) needed no new code. READABLE_SQL exempts transcripts from the 60-word page floor. A clip with no speech is stored with word_count = 0: done, never served.

7. Hard-won lessons

Each of these cost real time. Do not rediscover them.

Playwright and X

1. The GraphQL query ID changes on every X frontend deploy. Match only the trailing operation name (Bookmarks), never the full path.
2. Never make blocking calls inside a Playwright event handler in the sync API. Collect Response objects in the handler; read bodies in the main flow.
3. X throttles one session across concurrent browsers. The first parallel run returned 61 of 153 articles empty. Every one succeeded on a serial retry. Articles now use ARTICLE_WORKERS = 1; links keep WORKERS = 4, because they spread over hundreds of hosts.
4. Never scroll an X Article page. The view mounts and unmounts surrounding elements as you move, so scrolling makes extraction worse — measured at 542 words after two scrolls and 449 after four, against 478 with no scroll. The GraphQL content_state is captured in parallel as the authoritative copy and proves the render was complete.
5. bypass_csp=True is required. GitHub otherwise blocks script injection.
6. Playwright's sync API is bound to its creating thread. Each worker thread must build its own sync_playwright(), browser, and DB connection.
7. Rewrite arXiv /pdf/ to /abs/. A PDF URL makes the browser start a download instead of rendering. looks_like_a_file() guards the rest.
8. JavaScript apps can be empty at domcontentloaded. Retry once on networkidle before calling an extraction thin.

Everything else

9. load_dotenv() must take an explicit path. find_dotenv() walks up from the calling frame, which breaks for scripts run from stdin.
10. Defuddle returns HTML, not markdown, even with markdown: true. That is fine and deliberate: documents.body keeps HTML because the <a href> and <img src> inside are what later layers need. Markup comes off at chunk time in chunk.to_text().
11. Schema changes: drop and rebuild, never migrate. CREATE TABLE IF NOT EXISTS will not add a column to an existing table, and the owner forbids migrations. raw_pages makes rebuilding safe.
12. SQLite: LIMIT goes after UNION ALL. Wrap each side in a subquery.
13. embeddinggemma uses asymmetric prefixes. Documents get title: none | text: , queries get task: search result | query: . Using the wrong one is a real handicap. See embed.py.
14. Verify test expectations against real output before trusting them. Two early test failures were wrong expectations, not wrong code.
15. Cards name their target by t.co, so resolve the short link. Guessing which link a card belongs to attached cards to the wrong URL.
16. Store the smallest MP4, not the best. Transcription wants speech, not pixels. A 44.5-minute clip is 3.3 GB at top bitrate and 81 MB at 256 kbps. Across 40 bookmarks this was 10.73 GB versus 0.27 GB, a 40x saving.
17. Do not FK captions to media. replace_media deletes and reinserts on every normalize. ON DELETE CASCADE would wipe every caption on xbm sync. documents has no FK for the same reason.
18. vLLM FP8 needs a CUDA devel image. DeepGEMM compiles kernels at startup and asserts on a missing CUDA toolkit. debian_slim has none. Use nvidia/cuda:<ver>-devel, matched to the vLLM wheel (0.30 → CUDA 13).
19. Qwen3.8 is a hybrid Mamba model. vLLM's default max_num_seqs=1024 exceeds its Mamba cache blocks on an H100 (796). Set it to the batch size.
20. A raise in Modal @enter crash-loops on the GPU while the client waits. caption.Model.load keeps the error and fails the first call instead, so the run ends and the app stops.
21. Finish the Modal .map() generator. Leaving it unfinished closes it inside app.run() and raises aclose(): asynchronous generator is already running. In zip, put the generator first.
22. Incremental sync stops one page after the watermark hit. An earlier version only stopped when a scroll returned no pages, so it walked all 69.
23. faster-whisper 1.2.1 needs PyAV below 19. PyAV 19 removed the metadata_errors argument it passes to av.open.
24. faster-whisper's decode_audio drops invalid data without a word. A damaged download decodes to nothing and looks silent: a 2-hour podcast came back empty once. The container checks the byte count against Content-Length, and treats audio with no detected language as an error.
25. A linked page has many owners. 28 URLs are bookmarked more than once. Link documents used to carry the first bookmark's tweet_id, so the others could not find or read their own page. Ownership now comes from links through the bookmark_documents view, and a shared page is chunked once per bookmark.
26. A capture with no media or links is the truth. normalize replaces them unconditionally, so a stale photo and its caption leave the index.

7b. The CLI

• xbm sync runs capture, extract, caption, transcribe, translate, index. A failed stage does not stop the rest; the run exits non-zero at the end. Each stage is a _stage() function in cli.py that its own command and sync both call.
• Exclusions persist in ~/.config/x-bookmarks/config.toml (exclude = [...], translate = true), read by config.settings(). --no-* flags add to it for one run. An excluded source is skipped at its stage and dropped from the index (vectors too), but stays in the database.
• GPU stages never retry failures on their own; only --retry does. The usual failure is media deleted at X, and each try starts a GPU.
• sync captions only at 20 new photos (caption.SYNC_MIN_PHOTOS). vLLM startup costs the same 5 to 8 H100 minutes for 1 photo as for 600. xbm caption ignores the threshold. Transcripts need no threshold: L4 startup is about a minute.

8. MCP server

This is done. mcp_server.py uses MCP 2.0 and stdio transport. Codex has the server registered as x-bookmarks.

Why three tools, not one

Search returns one passage per bookmark, so a long thread cannot crowd out a sharp short post. That hides scale:

The worst case is one bookmark whose linked Claude Code CHANGELOG is 78,729 words in 155 chunks. A caller sees 1 of 155. It needs a way to go deeper, and a way to know that deeper exists.

The tools

1. search_bookmarks(query, limit=10, author=None, source=None, after=None, before=None) Wraps search.search(). Return per hit: tweet_id, url, author, created_at, score, best_source, best_chunk (the passage that matched), best_ref (the page URL it came from), chunk_count, word_count, media, lang. chunk_count and word_count are what make tool 3 discoverable. Also best_source_text (the original of a translated passage) and, on the response, last_sync. after is inclusive, before exclusive, on the post date.

   Filters apply before ranking (search.allowed_chunks). They used to apply to the top 80 candidates, so --source video -n 3 returned 2 hits. sqlite-vec's chunk_id IN (...) is a true pre-filter, except with one id: SQLite rewrites a one-item IN to =, and vec0 returns no rows for that inside a KNN query. _vector_ranks short-circuits the single-candidate case.

2. get_bookmark(tweet_id) Full post text, quoted post, author, date, every link, every media item. Each media item includes caption when one exists. Cheap and bounded. Do not include document bodies here.

3. read_document(tweet_id_or_url, query=None, offset=0) The extracted page text. This one is dangerous if done naively: returning the CHANGELOG whole is roughly 100,000 tokens and would destroy the caller's context. So:

   • with query: run the same hybrid search restricted to that document and return the best passages;
   • without: page through with offset and return has_more.

Use stdio transport. Add a xbm-mcp entry point in [project.scripts].

Registration

Codex uses this global registration:

codex mcp add x-bookmarks -- uv run --project ~/Developer/x-bookmarks-rag xbm-mcp

The server was tested through an MCP stdio client. All three tools returned structured results against the live corpus.

9. Backlog after MCP

In the owner's chosen order. Each layer must leave a working product.

1. Image captions and OCR. Done. uv run xbm caption after each sync captions only new photos, then uv run xbm index.
2. Video transcripts. Done. uv run xbm transcribe after each sync, then uv run xbm index. Uses media.small_url, never url.
3. Foreign language translation. Done. uv run xbm translate after each sync, then uv run xbm index.
4. Web UI. Over the same search.search() function.

10. How to verify your work

• uv run pytest -q — 137 tests, all offline, about one second. Keep it that way. Tests use synthetic GraphQL fixtures in tests/fixtures.py and stub embed.embed_documents / embed.embed_query with a deterministic vector.
• Run a real query and read the passages: uv run xbm search "what was the article about kv caching". A good result shows the matching passage and a from <url> line, not the post text.
• Never report a step as done without running it. The owner checks.