Imported from elizaOS/eliza (plugins/plugin-native-talkmode/AGENTS.md). Install upstream with npx skills add elizaOS/eliza --skill plugin-native-talkmode. Copyright stays with the author.

@elizaos/capacitor-talkmode

Capacitor plugin for voice conversations: STT → chat orchestration → TTS, across browser, iOS, Android, and Electrobun (desktop).

Purpose / role

Provides a unified TalkMode Capacitor plugin that Eliza agents can call to run full voice conversation sessions. On native (iOS/Android/Electrobun), it uses platform STT and ElevenLabs streaming TTS with PCM/MP3 playback; on web it falls back to the Web Speech API for both STT and TTS. This is a Capacitor plugin, not an elizaOS Plugin object — it is imported directly into UI/app code via @elizaos/capacitor-talkmode, not registered through the elizaOS plugin registry.

Plugin surface

This is a Capacitor plugin exposing a single TalkMode object. It does not register elizaOS actions, providers, services, or evaluators. The surface is:

Session modes (TalkModeSessionMode): idle, compose, push-to-talk, hands-free, passive.

State machine (TalkModeState): idle → listening → processing → speaking → error.

Layout

plugins/plugin-native-talkmode/
  src/
    index.ts           Capacitor registerPlugin call; exports TalkMode singleton + all types
    definitions.ts     All TypeScript interfaces and types (TalkModePlugin, TTSConfig, etc.)
    web.ts             Web fallback: Web Speech API STT + SpeechSynthesis TTS
  ios/
    Sources/TalkModePlugin/
      TalkModePlugin.swift   Native iOS: AVSpeechSynthesizer + SFSpeechRecognizer + ElevenLabs PCM/MP3
  android/
    src/main/java/ai/eliza/plugins/talkmode/TalkModePlugin.kt   Android native implementation (Kotlin)
  ElizaosCapacitorTalkmode.podspec   CocoaPods spec (requires AVFoundation + Speech frameworks)
  rollup.config.mjs          Builds IIFE (dist/plugin.js) and CJS (dist/plugin.cjs.js) from ESM
  tsconfig.json
  package.json

Commands

Scripts are defined in package.json; run them from the repo root with bun run --cwd:

bun run --cwd plugins/plugin-native-talkmode clean           # remove build output
bun run --cwd plugins/plugin-native-talkmode build           # build package artifacts
bun run --cwd plugins/plugin-native-talkmode build:docs      # generate docs and build artifacts
bun run --cwd plugins/plugin-native-talkmode typecheck       # TypeScript typecheck
bun run --cwd plugins/plugin-native-talkmode lint            # mutating Biome check
bun run --cwd plugins/plugin-native-talkmode lint:check      # read-only Biome check
bun run --cwd plugins/plugin-native-talkmode format          # write formatting
bun run --cwd plugins/plugin-native-talkmode format:check    # read-only formatting check
bun run --cwd plugins/plugin-native-talkmode test            # run package tests
bun run --cwd plugins/plugin-native-talkmode prepublishOnly  # publish-time build hook
bun run --cwd plugins/plugin-native-talkmode build:unlocked  # bun run clean && tsc && bunx rollup -c rollup.config.mjs
bun run --cwd plugins/plugin-native-talkmode docgen          # docgen --api TalkModePlugin --output-readme README.md --output-json dist/docs.json

Config / env vars

Config is passed at runtime via TalkMode.start({ config }) or TalkMode.updateConfig({ config }). No process-level env vars are read by this package. The key config fields:

The speak() call also accepts a TTSDirective for per-utterance overrides (voice, speed, stability, language, seed, etc.).

How to extend

Add a new method to the plugin surface:

1. Declare the method signature in src/definitions.ts on TalkModePlugin.
2. Implement it in src/web.ts (web fallback).
3. Implement it in ios/Sources/TalkModePlugin/TalkModePlugin.swift (register in pluginMethods).
4. Implement it in the Android Kotlin source at android/src/main/java/ai/eliza/plugins/talkmode/TalkModePlugin.kt.
5. Run bun run --cwd plugins/plugin-native-talkmode build to verify TS compiles.

Add a new event:

1. Define the event payload interface in src/definitions.ts.
2. Add an addListener overload to TalkModePlugin in src/definitions.ts.
3. Call this.notifyListeners("eventName", payload) in the web and native implementations.

Conventions / gotchas

• Not an elizaOS Plugin object. There is no actions, providers, services, or evaluators array. It is a Capacitor plugin registered with registerPlugin("TalkMode", { web: loadWeb }). Import TalkMode from @elizaos/capacitor-talkmode in UI/app code.
• ElevenLabs on web is blocked by CORS. The web implementation always falls back to SpeechSynthesis; usedSystemTts will always be true in the browser. ElevenLabs streaming TTS only works in native (iOS/Android) and Electrobun contexts.
• iOS native frameworks required. The CocoaPods spec declares AVFoundation and Speech frameworks. iOS 13.0+ minimum deployment target.
• Web STT auto-restarts. recognition.onend restarts the recogniser whenever the session is still enabled (this.enabled), preventing silent dropout when the browser ends a recognition run. Recognition is never paused during TTS, so the restart must not be gated on state === "listening"; a spontaneous onend that lands while the state is "speaking" (mid agent reply) would otherwise be swallowed, permanently deafening the single-shot recogniser (#22369).
• speak() on web always forces lang to en-US unless directive.language is set — this prevents browser-locale drift (e.g. numbers read in Chinese on Chinese-locale systems).
• Silence detection is stateful. On iOS, silenceWindow (default 0.7 s) drives a Task timer that finalises in-flight transcripts. Adjust via silenceWindowMs in config.
• Peer dep: @capacitor/core ^8.3.1 is required at the app level.
• See the root CLAUDE.md for repo-wide architecture rules, logger conventions, and git workflow.

Verification

Follow the repository-wide verification and evidence standard in the root CLAUDE.md. Run the package's relevant build, typecheck, lint, and test commands, then exercise the real integration boundary changed by the work. Inspect the produced domain artifacts and failure behavior; do not substitute mocked success for the system under test.