context-subtitle-translator

Imported from wongJG/context-subtitle-translator (AGENTS.md). Install upstream with npx skills add wongJG/context-subtitle-translator. Copyright stays with the author.

AGENTS.md — Context Subtitle Translator

Design, architecture, and hard-won lessons for this browser extension. Read this before making changes to the interception/translation pipeline or the Playwright tests.

Objective

An open-source, context-aware LLM subtitle translator for general HTML5 video players (no local files, no SVOD in v1). It intercepts subtitle data, translates it via an LLM API using a strict neighbor-line window (sliding context of adjacent lines), and renders the result natively by injecting a dual-line (original + translation) cue track into the video player. Targets Chrome + Firefox, MV3.

Tech stack

• WXT 0.19.29, TypeScript, MV3. Entrypoints live at the repo root entrypoints/ (NOT src/entrypoints). Valid imports: wxt/sandbox (defineContentScript, defineBackground), wxt/browser (browser).
• Build: npx wxt build → .output/chrome-mv3; npx wxt build -b firefox → .output/firefox-mv3 (gecko id context-subtitle-translator@local set in wxt.config.ts).
• Never use @/ path aliases for imports — they are NOT configured and break the build. Use relative imports.
• package.json scripts: dev, build, build:firefox, test.
• Tests: Playwright (@playwright/test), config playwright.config.ts, specs in test/.

Architecture / data flow

video page
  ├─ MAIN world hook  (entrypoints/subtitle-main.content.ts, runAt:'document_start')
  │     patches window.fetch + XMLHttpRequest
  │     → on subtitle response: parseSubtitleText() → write {url,text} to window.localStorage.__cstPending
  │
  ├─ ISOLATED content script (entrypoints/subtitle.content.ts = "orchestrator")
  │     polls window.localStorage every 500ms → drain __cstPending
  │     → fallback TrackInterceptor (video.textTracks, hidden) + optional DomInterceptor
  │     → for each detected cue group: buildWindow() → background TRANSLATE → inject into native track
  │     → also ingests background `CST_CAPTURE` messages
  │
  └─ background (entrypoints/background.ts) message router
        TRANSLATE → providers.translateChat()/translateDeepl() → returns translations
        webRequest.onCompleted (subtitle-ish URLs) → re-fetch body → sendMessage CST_CAPTURE → tab

Rendering: src/injector.ts. Translations are injected into a dedicated native text track (video.addTextTrack('subtitles', 'cst-translated'), mode showing) so the player renders them natively. Cue text is original\ntranslation (dual-line). All other tracks on the video are forced to mode='hidden' (via enforce(), called on setCues, on addtrack, and every 500ms by the orchestrator) to avoid duplicate native subtitles. Tests read the cst-translated track's cue text directly.

Single active subtitle. Players with many subtitle tracks (arte ships EN/FR/DE-forced/ DE-HOH/ES/IT VTTs in one master manifest) fetch the VTT of the current selection, and hls.js preloads more — so the background may capture several distinct files over time. ingestFromUrl() (in subtitle.content.ts) keeps only the subtitle the player actually renders: files that are not timeline-compatible with the active track (or that arrive before any active track) are held as candidates (candidates map) and promoted to active via render-confirm — tryConfirmCandidates() (run every 500ms) reads the text the player is currently rendering (DOM overlay like .ardplayer-untertitel, or a native track's activeCues) and promotes the candidate whose cue at currentTime matches that rendered text, requiring a long (≥15 normalized chars) cue that is unique among candidates so shared boilerplate (* Gong *, tagesschau's identical opening lines) can never confirm a wrong file — the per-episode line …untertitelt (DD.MM.YYYY) is what confirms the right one. A cuesShareTimeline() check (≥60% cue start-time overlap within 0.3s) still handles genuine same-content track switches directly (arte EN/FR/forced share cue boundaries). This keeps tagesschau teaser/related ARD players from clobbering the real subtitle (each teaser fetches its own untertitel-*.xml, is never rendered, so it stays a candidate and is ignored). On sites with no render signal (no subtitle DOM, no native text tracks), the first captured file is accepted immediately, preserving the original behavior. getVideo() prefers the playing <video> over the largest one, so on pages with multiple videos the active player wins.

Active-tab only. The extension only works in the active tab: the orchestrator gates all work (drain, render-confirm, translate, inject, interval) on document.hidden, the MAIN-world fetch/XHR hook skips captures when hidden, and the background WebRequestFallback drops CST_CAPTURE for non-active tabs (browser.tabs.get(tabId).active). On visibilitychange → hidden the orchestrator simply pauses — it keeps the cached masterCues/translations/ pending/candidates and the populated cst-translated track, so the subtitle survives the tab switch and is instantly there when the user returns. On return it re-attaches the track/DOM interceptors, renderInject()s the cached cues and resumes translation. resetState() is only used for real SPA navigation (onNavigation), NOT for tab switches. The TrackInterceptor (observeTextTracks) skips the injected cst-translated track entirely, so re-attaching it on return can never re-read the injected cues as source cues. Because every capture path (MAIN-hook write, CST_CAPTURE message, __cstPending drain) is gated on document.hidden/tab.active, a background capture can never inject a wrong subtitle into this tab even while it is hidden.

Interceptors (priority order)

1. NetworkInterceptor (PRIMARY) — MAIN-world fetch/XHR hook. Best coverage; catches the exact path tagesschau uses (player fetches a subtitle file via XHR/fetch). Captures full response body.
2. WebRequestFallback (BACKGROUND, REQUIRED for worker players) — chrome.webRequest .onCompleted in entrypoints/background.ts for subtitle-ish URLs, then re-fetches the body and forwards it to the tab via CST_CAPTURE. This is the ONLY path that catches subtitle requests made from inside a dedicated Web Worker: players like arte.tv (hls.js with enableWorker) fetch the subtitle playlist + VTT from a blob: worker, where self.fetch/XHR run in the worker's own global — invisible to the MAIN-world hook (content scripts never run in dedicated workers). Dedup by URL (30s window) avoids re-fetching the same subtitle; parseSubtitleText() gates on real cue content so stray .json endpoints are skipped; both MAIN-hook and background captures can fire for the same file (e.g. tagesschau) — ingest() dedups by cue id so nothing double-translates.
3. TrackInterceptor (FALLBACK) — isolated script scans <video>.textTracks, sets mode='hidden', listens cuechange. Covers players that only expose text tracks. NOTE: it only listens on tracks present at attach time (no addtrack listener), and worker-mode hls.js does not expose a populated native track, so it does NOT rescue arte.tv. It skips the injected cst-translated track, so re-attaching it after a tab switch or navigation can never re-read the extension's own injected cues as source cues.
4. DomInterceptor (OPTIONAL) — enabled via domSelector setting; observeDom for hardcoded caption DOM nodes.

Subtitle format support (in src/parsers.ts): VTT, SRT, JSON timed-text, and EBU-TT-D/TTML (parseTTML). parseSubtitles detects XML (<) and routes to parseTTML first, else VTT/SRT block parser. TTML detection is required — ARD/tagesschau serves untertitel-*.xml (EBU-TT-D with <tt:p begin="…" end="…">), which the -->-only parser cannot read. parseSubtitleText(text) (used by the MAIN hook AND the background fallback) runs parseSubtitles then a JSON timed-text fallback.

URL/text heuristics for "is this a subtitle?" (src/subtitle-url.ts, shared by MAIN hook + background): \.vtt|.srt|.xml|.ttml|.json extensions, or URL containing subtitle|caption|timedtext|untertitel, or body containing --> / HH:MM:SS timecodes. Thumbnail-sprite VTTs must be rejected: hls.js players ship a separate *_SPR.vtt / sprite.vtt WebVTT whose cue "text" is an image crop URL like foo.jpg#xywh=0,0,240,135 — rejection is content-based, NOT URL-based (a URL keyword blacklist would false-positive on legitimate subtitle URLs). parseSubtitleText() (in src/parsers.ts) treats any cue body that is a lone image reference (*.jpg|png|webp|gif#… or x-sprite:) as a sprite cue, and returns [] for a whole file whose cues are all sprite cues (dropping stray sprite cues from mixed files), so sprite VTTs never reach the injected track. arte.tv serves one (123937-000-A_SPR.vtt, ~875 thumbnail cues) that otherwise pollutes the subtitle output. (123937-000-A_SPR.vtt, ~875 thumbnail cues) that otherwise pollutes the subtitle output.

Cross-world (MAIN ↔ ISOLATED) communication — IMPORTANT

The MAIN-world hook cannot call the isolated-world content script directly via the normal content-script messaging (chrome.runtime.sendMessage / chrome.storage busy / window .postMessage) — all of these failed in testing because the page's own chrome shim or isolated-world boundaries block them.

Working solution: the MAIN hook writes to window.localStorage (shared between MAIN and ISOLATED worlds on the same origin). It appends each capture {url, text} to a queue (__cstPending holds a JSON array, capped at 50), because a single slot would lose captures when several subtitle files are fetched within one 500ms poll window (tagesschau fetches the main + teaser untertitel-*.xml in quick succession). The orchestrator polls window.localStorage.__cstPending every 500ms, drains the whole queue in order and ingests. Do NOT "fix" this by switching to postMessage/runtime.sendMessage — it will silently break interception.

Context-aware translation

• src/context.ts buildWindow(cues, centerStart, centerEnd, n) (default n=2 = neighborWindow). Produces a prompt with neighbors labeled CONTEXT (do not translate) and the center lines labeled TRANSLATE the following N line(s):. This is what keeps speaker/pronoun/disambiguation correct. Tests assert both neighbor and center texts appear in the LLM prompt.
• src/providers.ts translateChat uses req.apiBase (default OpenRouter/OpenAI), builds system + user messages, parses choices[0].message.content, splits lines 1:1 with center cues. translateDeepl for DeepL.

Settings (browser.storage.local key settings)

In src/types.ts Settings: enabled, provider, apiKey, model, apiBase, deeplFree, srcLang, tgtLang, neighborWindow, bufferSeconds, overlayPosition, fontScale, domSelector.

• apiBase added for testing — lets the extension point at a local mock LLM.
• domSelector optional DOM-interceptor trigger.
• Translation is progressive: scheduleTranslate() only batches untranslated cues whose start <= video.currentTime + bufferSeconds (default 60). As playback advances, upcoming cues enter the horizon and are translated, so far-future subtitles are NOT shot all at once. Neighbor context (buildWindow) still spans the full cue list regardless of the buffer.

Playwright tests (test/e2e.spec.ts)

Three tests, all currently passing:

1. translates subtitles via TextTrack interceptor — page with <video> + <track>; asserts original text + [EN] translation in the injected cst-translated native track, mock LLM called, both cue texts in prompt.
2. translates subtitles via network interceptor — canvas <video> + fetch('/subs.vtt'); asserts original captured, [EN] translation in the injected track, full-file prompt context, and that a normal fetch('/data.json') is NOT intercepted.

Playwright gotchas

• Extensions require full Chromium (the headless shell does not support extensions). fullChromiumPath() scans ~/.cache/ms-playwright/chromium-*/chrome-linux/chrome. Launch with chromium.launchPersistentContext(dir, { headless: false, executablePath: fullChromiumPath(), args: ['--headless=new',--load-extension=${EXT_PATH}, --disable-extensions-except=${EXT_PATH}, '--no-sandbox', '--disable-setuid-sandbox', '--disable-gpu'] }).
• Mock LLM server (localhost) returns [EN] <line> per center line; parses the prompt via regex /TRANSLATE the following \d+ line\(s\):\n([\s\S]*?)\n\nCONTEXT/. It must handle the CORS preflight OPTIONS request (the extension's fetch from a page origin to http://127.0.0.1 triggers one) with access-control-allow-origin: * etc.
• Settings are applied via the popup page: chrome.storage.local.set({ settings: {...} }) with apiBase: 'http://127.0.0.1:PORT/v1'.
• Test 3 self-skips if tagesschau is unreachable (graceful), so it won't hard-fail CI offline.

Do / Don't

• DO keep the window.localStorage MAIN→isolated bridge.
• DO preserve relative imports and the MV3/WXT patterns; match existing module style.
• DON'T introduce @/ aliases, comments in code (per repo policy), or assume new libraries are available — verify against package.json / existing imports first.
• DON'T run arbitrary git commits unless explicitly asked.