Instruction file imported from shivamkb17/memoria (
.cursor/rules/avatar-streaming.mdc). Copyright stays with the author.
Avatar streaming rules
WebRTC
'use client'for stream components- Clean up on unmount:
RTCPeerConnection.close(), clearsrcObject, abort in-flight fetches - Handle disconnect with reconnect UI — don't leave zombie connections
State machine
Align UI with DESIGN.md: Idle, Listening, Thinking, Speaking, Reacting.
- Thinking while awaiting GPT-4o; Speaking during ElevenLabs TTS + D-ID lip-sync
- Reacting when vision context changes materially (compare previous exhibit string)
D-ID (renderer only)
- Confirm paths via d-id MCP (
search-endpoints→get-endpoint) before editing stream code - Init session via
app/api/did— not directapi.d-id.comfrom browser with API key - Lip-sync via stream +
script.type: "audio"— do not use/agents/{id}/chat - Expression hints: curiosity (new exhibit), warmth, solemnity per GPT reply tone
Performance
- Target <2.5s speech-to-response — see ARCHITECTURE.md
- Pre-warm D-ID WebRTC stream after camera granted
- Stream GPT tokens → ElevenLabs TTS → D-ID audio inject (no full-reply buffering)