Multi-lingual-Nepali-Voice-AI

Imported from Susandhungana1/Multi-lingual-Nepali-Voice-AI (AGENTS.md). Install upstream with npx skills add Susandhungana1/Multi-lingual-Nepali-Voice-AI. Copyright stays with the author.

agents.md — Multilingual Voice AI Assistant for Nepal

Project Overview

A voice-based AI assistant that understands Nepali and English speech, answers questions intelligently, and responds in both languages. Built for real-world deployment with a privacy-first, locally-runnable architecture.

Agent Architecture

User (Voice Input)
       │
       ▼
┌─────────────────────┐
│  Streamlit Frontend  │  ← mic recording, language toggle, chat UI
└─────────┬───────────┘
          │ audio bytes (WAV/MP3)
          ▼
┌─────────────────────┐
│   FastAPI Backend    │
│                      │
│  ┌────────────────┐  │
│  │  STT Agent     │  │  ← faster-whisper (local, CUDA optional)
│  │  (Whisper AI)  │  │     detects language automatically
│  └───────┬────────┘  │
│          │ transcript + detected_lang
│          ▼            │
│  ┌────────────────┐  │
│  │ Translation    │  │  ← OPUS-MT (MarianMT) if input is Nepali → English
│  │ Agent (OPUS-MT)│  │     skips if input is already English
│  └───────┬────────┘  │
│          │ english_text
│          ▼            │
│  ┌────────────────┐  │
│  │  LLM Agent     │  │  ← OpenRouter (free tier: freemodels)
│  │  (OpenRouter)  │  │     answers in English
│  └───────┬────────┘  │
│          │ english_answer
│          ▼            │
│  ┌────────────────┐  │
│  │ Translation    │  │  ← OPUS-MT: English → Nepali (if user spoke Nepali)
│  │ Agent (OPUS-MT)│  │     returns both English + Nepali response
│  └───────┬────────┘  │
└──────────┼───────────┘
           │ { nepali_answer, english_answer, transcript, detected_lang }
           ▼
┌─────────────────────┐
│  Streamlit Frontend  │  ← displays bilingual response + plays TTS (optional)
└─────────────────────┘

Agents & Their Roles

1. STT Agent — stt_agent.py

Model: faster-whisper (tiny or base model for lightweight CPU use) Responsibility: Transcribe audio input → text. Automatically detects whether the user spoke Nepali (ne) or English (en).

from faster_whisper import WhisperModel

model = WhisperModel("base", device="cpu", compute_type="int8")

def transcribe(audio_path: str) -> dict:
    segments, info = model.transcribe(audio_path, beam_size=5)
    transcript = " ".join([s.text for s in segments])
    return {
        "transcript": transcript.strip(),
        "detected_lang": info.language  # "ne" or "en"
    }

2. Translation Agent — translation_agent.py

Model: Helsinki-NLP/opus-mt-ne-en + Helsinki-NLP/opus-mt-en-ne (MarianMT) Responsibility: Translate Nepali → English (before LLM) and English → Nepali (after LLM response). Skips translation if the detected language is already English.

from transformers import MarianMTModel, MarianTokenizer

MODEL_NAMES = {
    ("ne", "en"): "Helsinki-NLP/opus-mt-ne-en",
    ("en", "ne"): "Helsinki-NLP/opus-mt-en-ne",
}

_tokenizers = {}
_models = {}

def _get_pair(src_lang: str, tgt_lang: str):
    key = (src_lang, tgt_lang)
    if key not in MODEL_NAMES:
        raise ValueError(f"Unsupported translation pair: {src_lang}->{tgt_lang}")
    model_name = MODEL_NAMES[key]
    if key not in _tokenizers:
        _tokenizers[key] = MarianTokenizer.from_pretrained(model_name)
        _models[key] = MarianMTModel.from_pretrained(model_name)
    return _tokenizers[key], _models[key]

def translate(text: str, src_lang: str, tgt_lang: str) -> str:
    if src_lang == tgt_lang:
        return text
    tokenizer, model = _get_pair(src_lang, tgt_lang)
    inputs = tokenizer(text, return_tensors="pt", truncation=True)
    output = model.generate(**inputs, max_new_tokens=256)
    return tokenizer.decode(output[0], skip_special_tokens=True)

3. LLM Agent — llm_agent.py

Model: OpenRouter — meta-llama/llama-3.1-8b-instruct:free (free tier, no credit card needed) Responsibility: Receive the English-translated user query, generate a helpful, context-aware answer in English. A system prompt instructs it to keep answers concise and suitable for voice output.

Why OpenRouter? It provides access to free models (freemodels) via a single OpenAI-compatible API. Sign up at openrouter.ai — free tier requires no credit card.

import requests
import os

OPENROUTER_API_KEY = os.getenv("OPENROUTER_API_KEY")

SYSTEM_PROMPT = """
You are a helpful bilingual assistant for Nepali and English speakers.
Always answer in clear, concise English. Avoid markdown formatting.
Your answers will be translated to Nepali, so use simple, natural language.
"""

# Free models available on OpenRouter (as of 2025):
# - meta-llama/llama-3.1-8b-instruct:free
# - google/gemma-3-12b-it:free
# - mistralai/mistral-7b-instruct:free
FREE_MODEL = "openrouter/free"

def get_answer(english_query: str, conversation_history: list) -> str:
    messages = [{"role": "system", "content": SYSTEM_PROMPT}]
    messages += conversation_history
    messages.append({"role": "user", "content": english_query})

    response = requests.post(
        url="https://openrouter.ai/api/v1/chat/completions",
        headers={
            "Authorization": f"Bearer {OPENROUTER_API_KEY}",
            "Content-Type": "application/json",
            "HTTP-Referer": "https://your-app-name.streamlit.app",  # optional but recommended
        },
        json={
            "model": FREE_MODEL,
            "messages": messages,
            "max_tokens": 512,
        }
    )
    response.raise_for_status()
    return response.json()["choices"][0]["message"]["content"]

4. Orchestrator Agent — orchestrator.py

Responsibility: Coordinates the full pipeline. Called by the FastAPI endpoint. Persists conversation history in SQLite for multi-turn context across restarts.

import os
import sqlite3
from stt_agent import transcribe
from translation_agent import translate
from llm_agent import get_answer

DB_PATH = os.getenv("SQLITE_PATH", "./voice_assistant.db")

def _get_db():
    conn = sqlite3.connect(DB_PATH)
    conn.execute(
        """
        CREATE TABLE IF NOT EXISTS conversation_history (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            session_id TEXT NOT NULL,
            role TEXT NOT NULL,
            content TEXT NOT NULL
        )
        """
    )
    return conn

def load_history(session_id: str, limit: int = 20) -> list:
    conn = _get_db()
    rows = conn.execute(
        "SELECT role, content FROM conversation_history WHERE session_id = ? ORDER BY id DESC LIMIT ?",
        (session_id, limit),
    ).fetchall()
    conn.close()
    return [{"role": r[0], "content": r[1]} for r in reversed(rows)]

def save_turn(session_id: str, user_text: str, assistant_text: str) -> None:
    conn = _get_db()
    conn.execute(
        "INSERT INTO conversation_history (session_id, role, content) VALUES (?, ?, ?)",
        (session_id, "user", user_text),
    )
    conn.execute(
        "INSERT INTO conversation_history (session_id, role, content) VALUES (?, ?, ?)",
        (session_id, "assistant", assistant_text),
    )
    conn.commit()
    conn.close()

def process_voice_query(audio_path: str, session_id: str = "default") -> dict:
    # Step 1: Speech to text
    stt_result = transcribe(audio_path)
    transcript = stt_result["transcript"]
    detected_lang = stt_result["detected_lang"]

    # Step 2: Translate to English if needed
    english_query = translate(transcript, src_lang=detected_lang, tgt_lang="en")

    # Step 3: Load context + get LLM answer
    conversation_history = load_history(session_id)
    english_answer = get_answer(english_query, conversation_history)

    # Step 4: Persist conversation history
    save_turn(session_id, english_query, english_answer)

    # Step 5: Translate answer back to Nepali
    nepali_answer = translate(english_answer, src_lang="en", tgt_lang="ne")

    return {
        "transcript": transcript,
        "detected_lang": detected_lang,
        "english_answer": english_answer,
        "nepali_answer": nepali_answer
    }

5. API Layer — main.py (FastAPI)

Responsibility: Expose a /voice-query endpoint that accepts audio uploads and returns bilingual responses.

from fastapi import FastAPI, UploadFile, File
from orchestrator import process_voice_query
import shutil, uuid, os

app = FastAPI(title="Multilingual Voice AI — Nepal")

@app.post("/voice-query")
async def voice_query(audio: UploadFile = File(...), session_id: str = "default"):
    tmp_path = f"/tmp/{uuid.uuid4()}.wav"
    with open(tmp_path, "wb") as f:
        shutil.copyfileobj(audio.file, f)
    
    result = process_voice_query(tmp_path, session_id=session_id)
    os.remove(tmp_path)
    return result

6. Frontend — app.py (Streamlit)

Responsibility: Provide a clean UI with mic recording, language display, and bilingual response rendering.

import streamlit as st
import requests
from audio_recorder_streamlit import audio_recorder

st.title("🇳🇵 Multilingual Voice AI Assistant")
st.caption("Speak in Nepali or English — get answers in both!")

audio_bytes = audio_recorder()

if audio_bytes:
    st.audio(audio_bytes, format="audio/wav")
    with st.spinner("Processing..."):
        response = requests.post(
            "http://localhost:8000/voice-query",
            files={"audio": ("recording.wav", audio_bytes, "audio/wav")}
        )
        data = response.json()

    st.markdown(f"**🎙️ You said:** {data['transcript']} `({data['detected_lang']})`")
    st.markdown("---")
    col1, col2 = st.columns(2)
    with col1:
        st.markdown("### 🇬🇧 English")
        st.write(data["english_answer"])
    with col2:
        st.markdown("### 🇳🇵 नेपाली")
        st.write(data["nepali_answer"])

Project Structure

multilingual-voice-ai/
│
├── backend/
│   ├── main.py              # FastAPI app
│   ├── orchestrator.py      # Pipeline coordinator
│   ├── stt_agent.py         # Whisper transcription
│   ├── translation_agent.py # OPUS-MT translation
│   └── llm_agent.py         # OpenRouter (free LLM) integration
│
├── frontend/
│   └── app.py               # Streamlit UI
│
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
└── agents.md                # This file

Environment Variables

OPENROUTER_API_KEY=your_openrouter_api_key_here   # free at openrouter.ai
WHISPER_MODEL=base            # tiny | base (recommended on free CPU)
WHISPER_DEVICE=cpu            # cpu | cuda
OPUS_NE_EN_MODEL=Helsinki-NLP/opus-mt-ne-en
OPUS_EN_NE_MODEL=Helsinki-NLP/opus-mt-en-ne
SQLITE_PATH=./voice_assistant.db

Requirements

fastapi
uvicorn
faster-whisper
transformers
sentencepiece
requests
streamlit
audio-recorder-streamlit
python-multipart
torch

Running Locally

# 1. Install dependencies
pip install -r requirements.txt

# 2. Start FastAPI backend
cd backend
uvicorn main:app --reload --port 8000

# 3. Start Streamlit frontend (new terminal)
cd frontend
streamlit run app.py

Deployment Notes

• Backend: Deploy FastAPI on Render free tier or Koyeb free tier
• Frontend: Deploy Streamlit on Streamlit Community Cloud — completely free
• LLM: OpenRouter free tier — no credit card, no cost
• GPU: For faster Whisper inference, Render free tier is CPU-only but works fine with tiny or base Whisper model
• Model caching: OPUS-MT models are much smaller than NLLB, but still cache Hugging Face models in Docker for faster startup

💸 Full Cost Breakdown

Future Enhancements