Forem: Konstantin

I Built a Voice AI with Sub-500ms Latency. Here's the Echo Cancellation Problem Nobody Talks About

Konstantin — Sun, 05 Apr 2026 08:50:00 +0000

When I started building GoNoGo.team — a platform where AI agents interview founders by voice to validate startup ideas — I thought the hard part would be the AI reasoning. The multi-agent orchestration. The 40+ function-calling tools.

I was wrong.

The hard part was echo. Specifically: how do you stop an AI agent from hearing itself talk, freaking out, and interrupting its own sentence?

After 500+ voice sessions and too many late nights staring at RMS waveforms, here's what I actually learned.

The Setup: Speech-to-Speech, Not STT → LLM → TTS

GoNoGo runs on Gemini 2.5 Flash Live API — a true speech-to-speech pipeline. There's no intermediate transcription step, no text-to-speech synthesis layer bolted on afterward. Audio goes in, audio comes out. Direct.

This is important because it changes everything about how you handle audio on the client. You're not working with text buffers. You're working with raw PCM, 16kHz input from the browser mic, 24kHz output from the agent voice. Base64-encoded over WebSocket.

The browser capture side looks roughly like this:

// ScriptProcessorNode in browser — 512-sample chunks (~32ms each)
const scriptProcessor = audioContext.createScriptProcessor(512, 1, 1);

scriptProcessor.onaudioprocess = (event) => {
  const inputBuffer = event.inputBuffer.getChannelData(0);

  // Calculate RMS for VAD
  const rms = Math.sqrt(
    inputBuffer.reduce((sum, sample) => sum + sample * sample, 0) / inputBuffer.length
  );

  // VAD threshold: 0.05 RMS
  if (rms < VAD_THRESHOLD) return;

  // Convert Float32 PCM to Int16
  const int16Buffer = new Int16Array(inputBuffer.length);
  for (let i = 0; i < inputBuffer.length; i++) {
    int16Buffer[i] = Math.max(-32768, Math.min(32767, inputBuffer[i] * 32768));
  }

  // Base64 encode and send over WebSocket
  const base64Audio = btoa(String.fromCharCode(...new Uint8Array(int16Buffer.buffer)));
  ws.send(JSON.stringify({ type: 'audio_chunk', data: base64Audio }));
};

Simple enough. Until the AI starts talking.

The Echo Problem (And Why Browser AEC Isn't Enough)

Browsers have built-in acoustic echo cancellation. You enable it when you call getUserMedia:

const stream = await navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: true,
    noiseSuppression: true,
    autoGainControl: true
  }
});

This works great for video calls between humans. It was designed for that. But it has a fundamental assumption baked in: the "far end" audio is coming through a <audio> element or Web Audio API that the browser knows about.

When you're playing 24kHz PCM chunks from a WebSocket, decoded manually and scheduled through AudioContext buffers? The browser's AEC has no idea that audio exists. It can't cancel what it can't see.

So your AI agent starts speaking. The microphone picks up the speaker output. The agent hears itself. In the best case, it gets confused and repeats something. In the worst case — and this happened constantly in early builds — you get a feedback loop where the agent interrupts itself mid-sentence, hears the interruption, tries to respond to it, hears that, and the whole session collapses.

I called these 1011 disconnects, because that was the WebSocket close code I kept seeing in logs.

The Two-Tier RMS Gate

The fix is a two-tier RMS (Root Mean Square) gate on the audio capture side. The idea is simple: measure the loudness of what the mic is picking up, and if it's probably just the speaker playing back, don't send it.

But "simple" hides a lot of edge cases.

Tier 1: Hard suppress during agent speech

While the agent is actively speaking, I track that state server-side and send it to the client. During this window, incoming audio is suppressed entirely — no chunks sent to Gemini.

let agentSpeaking = false;
let cooldownTimer: ReturnType<typeof setTimeout> | null = null;
const COOLDOWN_MS = 1500;
const COOLDOWN_THRESHOLD = 0.03; // Higher threshold during cooldown
const NORMAL_THRESHOLD = 0.05;   // Normal VAD threshold

// Called when agent audio stream starts/stops
function setAgentSpeakingState(speaking: boolean) {
  if (speaking) {
    agentSpeaking = true;
    if (cooldownTimer) clearTimeout(cooldownTimer);
  } else {
    agentSpeaking = false;
    // Start cooldown period
    cooldownTimer = setTimeout(() => {
      cooldownTimer = null;
    }, COOLDOWN_MS);
  }
}

function shouldSendAudioChunk(rms: number): boolean {
  if (agentSpeaking) return false; // Hard suppress

  if (cooldownTimer !== null) {
    // In cooldown: use higher threshold
    return rms > COOLDOWN_THRESHOLD;
  }

  return rms > NORMAL_THRESHOLD;
}

Tier 2: The 1.5-second cooldown

This is the part that took me longest to figure out. When the agent stops talking, there's still speaker resonance in the room. The RMS of captured audio doesn't drop to zero immediately — it decays. The background noise in a typical home office sits at 0.01–0.02 RMS. But for 1-2 seconds after playback stops, you're seeing 0.025–0.04 RMS — above the normal VAD threshold.

The cooldown period uses a higher threshold (0.03 vs 0.05) for 1.5 seconds after agent speech ends. This catches the decay without cutting off a founder who immediately starts talking back.

Was this threshold tuned empirically? Absolutely. I spent days listening to session replays measuring exactly how fast room resonance decays in different mic setups.

Session Resumption: The Other Half of the Problem

Echo cancellation solved the quality problem. Session resumption solved the reliability problem.

Gemini Live sessions drop. Network hiccups, mobile handoffs, Chrome deciding to do something aggressive with memory — connections fail. Early on, a dropped connection meant starting the entire 30-minute interview over. Founders would ragequit. I would understand completely.

The fix: store session handles in Firestore and resume on reconnect.

# FastAPI backend — session management
from google.genai.live import AsyncSession
from firebase_admin import firestore

async def get_or_create_session(
    project_id: str, 
    user_id: str
) -> tuple[AsyncSession, bool]:
    db = firestore.client()
    session_ref = db.collection('sessions').document(f'{user_id}_{project_id}')
    session_doc = session_ref.get()

    if session_doc.exists:
        session_data = session_doc.to_dict()
        handle = session_data.get('resumption_handle')

        if handle:
            try:
                # Attempt resume — Gemini picks up exactly where it left off
                session = await resume_gemini_session(handle)
                return session, True  # resumed=True
            except Exception:
                pass  # Fall through to new session

    # Create new session
    session = await create_gemini_session(project_id)
    session_ref.set({
        'created_at': firestore.SERVER_TIMESTAMP,
        'project_id': project_id
    })
    return session, False  # resumed=False

async def store_resumption_handle(user_id: str, project_id: str, handle: str):
    db = firestore.client()
    session_ref = db.collection('sessions').document(f'{user_id}_{project_id}')
    session_ref.update({'resumption_handle': handle})

When a session resumes, Gemini restores full context — every tool call result, every piece of market research, every persona in the synthetic focus group. The founder reconnects and the agent says "Sorry about that, where were we?" and genuinely knows where you were.

The Filler Audio Problem

One more thing nobody talks about: what do you play while the AI is thinking?

Gemini 2.5 Flash is fast. 300-500ms end-to-end is genuinely fast. But when the agent is executing a tool call — crawling a competitor site with Playwright, running Reddit scraping, calculating unit economics — you can have 3-8 second gaps.

Silence in a voice conversation feels broken. Users assume the connection dropped.

Solution: pre-computed filler audio. Short phrases like "one moment please" or "let me look that up" in 17 languages, stored as PCM chunks, played when tool execution exceeds ~800ms. The agent is triggered via text signal (not proactive_audio, which had a regression that caused double-playback — disabled entirely, use text triggers instead).

This sounds trivial. It removed about 40% of "the app is broken" support messages.

What I'd Do Differently

Start with the echo gate, not the AI logic. I spent weeks building beautiful multi-agent orchestration before I could demo it reliably. Wrong order.
Instrument RMS values from day one. Log them. Every session. You can't tune what you can't see.
Test on bad hardware. My dev setup has a good mic with physical distance from speakers. Most users have laptop mics 30cm from laptop speakers. Build for that.
Mobile is a different planet. iOS Safari handles AudioContext lifecycle in ways that will make you question your career choices. But that's an article for another day.

The Result

After solving these problems — the two-tier RMS gate, the 1.5s cooldown, the session resumption, the filler audio — GoNoGo runs 15-45 minute voice sessions with real founders, across 21 languages, with 3 AI agents handing off to each other mid-conversation. The 1011 disconnects essentially disappeared.

The voice infrastructure became invisible, which is exactly what it should be.

If you're building anything with browser mic + real-time AI audio: what's been your biggest challenge? I'm genuinely curious whether the echo problem is universal or whether I was doing something particularly wrong early on. Drop it in the comments.

How I Built a Real-Time Voice AI Interview System with Gemini Live API and WebSockets (and What Almost Broke Me)

Konstantin — Sat, 04 Apr 2026 15:45:52 +0000

When I started building GoNoGo.team -- a platform that uses AI to validate startup ideas through voice interviews -- I thought the hardest part would be the business logic. Turns out, the hardest part was keeping a duplex audio stream alive across three layers of abstraction without everything falling apart.

This is a technical post-mortem of the voice AI system I built solo. I'll cover the architecture, the ugly edge cases, and the specific patterns that finally made it stable enough to run 500+ validation interviews.

The Core Problem: Bidirectional Audio at Low Latency

The concept: a founder speaks, Gemini listens and responds with follow-up questions, in real time. No STT/TTS pipeline -- direct speech-to-speech using Gemini Live API (native audio). The pipeline looks like this:

Browser Mic -> ScriptProcessor (16kHz PCM) -> WebSocket (base64) -> Python FastAPI -> Gemini Live API
                                                                                      v
Browser Speaker <- AudioContext (24kHz PCM) <- WebSocket (base64) <- Audio Chunks <------+

Every arrow in that diagram is a potential failure point. And in production, every single one of them failed at least once.

Step 1: Capturing Audio in the Browser

The browser side uses a ScriptProcessorNode (yes, it's deprecated -- but AudioWorklet adds latency we can't afford for real-time conversation). We capture 16kHz mono PCM in 512-sample chunks -- roughly 32ms per chunk.

// Audio capture setup (simplified from useAudioInput.ts)
const stream = await navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: true,
    noiseSuppression: true,
    autoGainControl: true,
    sampleRate: 16000,
  }
});

const audioContext = new AudioContext({ sampleRate: 16000 });
const source = audioContext.createMediaStreamSource(stream);
const processor = audioContext.createScriptProcessor(512, 1, 1);

// Analyser for RMS-based voice activity detection
const analyser = audioContext.createAnalyser();
source.connect(analyser);
analyser.connect(processor);

processor.onaudioprocess = (event) => {
  const pcmData = event.inputBuffer.getChannelData(0);
  const rms = Math.sqrt(
    pcmData.reduce((sum, x) => sum + x * x, 0) / pcmData.length
  );

  // VAD gate: only send if voice detected (RMS > 0.05)
  if (rms > 0.05 && ws.readyState === WebSocket.OPEN) {
    // Convert Float32 to Int16 PCM, then base64 encode
    const int16 = new Int16Array(pcmData.length);
    for (let i = 0; i < pcmData.length; i++) {
      int16[i] = Math.max(-32768, Math.min(32767, pcmData[i] * 32768));
    }
    ws.send(JSON.stringify({
      type: "audio",
      data: btoa(String.fromCharCode(...new Uint8Array(int16.buffer)))
    }));
  }
};

The 32ms chunk interval was a hard-won choice. It gives Gemini enough data per packet to process efficiently while keeping perceived latency under 300ms end-to-end. The VAD threshold of 0.05 RMS filters out background noise without clipping soft speech.

Step 2: The Python Backend (FastAPI + WebSockets)

The backend is Python FastAPI, deployed on Google Cloud Run. Python was the right call because Gemini's client libraries are Python-first, and the entire analysis pipeline (market research, competitor scraping with Playwright, report generation) lives in the same codebase.

# WebSocket handler (simplified from server.py)
@app.websocket("/ws_live")
async def websocket_live(ws: WebSocket):
    await ws.accept()
    session = GeminiLiveSession(model="gemini-2.5-flash-exp")

    async def forward_to_gemini():
        # Browser audio -> Gemini
        async for msg in ws.iter_json():
            if msg["type"] == "audio":
                pcm_bytes = base64.b64decode(msg["data"])
                await session.send_audio(pcm_bytes)  # 16kHz PCM

    async def forward_to_browser():
        # Gemini audio -> Browser
        async for event in session.events():
            if event.type == "audio":
                # Gemini returns 24kHz PCM
                chunk_b64 = base64.b64encode(event.data).decode()
                await ws.send_json({
                    "type": "audio",
                    "data": chunk_b64
                })
            elif event.type == "tool_call":
                result = await execute_tool(event)
                await session.send_tool_response(result)

    await asyncio.gather(forward_to_gemini(), forward_to_browser())

The asymmetric sample rates (16kHz in, 24kHz out) aren't a mistake -- Gemini natively outputs at 24kHz, and downsampling would lose audio quality. The browser's AudioContext handles the sample rate mismatch transparently.

The Echo Cancellation Problem

Gemini hears its own output through the user's speakers and tries to respond to itself. Browser-level echo cancellation (echoCancellation: true) handles most cases, but not all -- especially on laptops with poor speaker-mic isolation.

My solution: a speaking-state gate. When Gemini is outputting audio, we suppress inbound audio at the application level:

# Echo gate in the session handler
class SessionState:
    def __init__(self):
        self.agent_speaking = False
        self.last_agent_audio_time = 0.0

    def should_forward_audio(self, rms: float) -> bool:
        # Suppress during agent speech + 1.5s cooldown after
        if self.agent_speaking:
            return False
        if time.time() - self.last_agent_audio_time < 1.5:
            return rms > 0.03  # Higher threshold during cooldown
        return rms > 0.01  # Normal threshold

This two-tier threshold was the key insight: background noise sits at RMS 0.01-0.02, so during the cooldown period after the agent stops speaking, we only forward audio that's clearly human speech (> 0.03).

Failure Mode: The 1011 Disconnects

For weeks, Gemini Live API would randomly close connections with status code 1011 (Internal Server Error). No pattern, no warning. Sessions would die mid-sentence.

The fix was layered:

# Reconnection with session resumption
async def handle_disconnect(session, ws):
    for attempt in range(3):
        try:
            # Gemini supports session resumption via handle
            new_session = await GeminiLiveSession.resume(
                session.resumption_handle
            )
            # Re-send last audio chunk as context
            await new_session.send_audio(session.last_chunk)
            return new_session
        except Exception:
            await asyncio.sleep(0.5 * (attempt + 1))
    # After 3 fails, notify user with audio message
    await ws.send_json({"type": "system", "message": "reconnecting"})

Session resumption handles (persisted to Firestore) were a game-changer. Instead of starting a new conversation, Gemini picks up exactly where it left off. Users barely notice the blip.

Step 3: Playing Audio Back in the Browser

Gemini returns 24kHz PCM chunks. Playing them without glitches requires the Web Audio API with a buffer scheduler:

// Audio playback (simplified from useAudioOutput.ts)
class AudioPlayer {
  private context: AudioContext;
  private nextStartTime = 0;
  private gainNode: GainNode;

  constructor() {
    this.context = new AudioContext({
      sampleRate: 24000,
      latencyHint: /Mobi/.test(navigator.userAgent)
        ? "playback" : "interactive"
    });
    this.gainNode = this.context.createGain();
    this.gainNode.connect(this.context.destination);
  }

  playChunk(pcmBase64: string) {
    const bytes = Uint8Array.from(atob(pcmBase64), c => c.charCodeAt(0));
    const int16 = new Int16Array(bytes.buffer);
    const float32 = new Float32Array(int16.length);
    for (let i = 0; i < int16.length; i++) {
      float32[i] = int16[i] / 32768;
    }

    const buffer = this.context.createBuffer(1, float32.length, 24000);
    buffer.getChannelData(0).set(float32);

    const source = this.context.createBufferSource();
    source.buffer = buffer;
    source.connect(this.gainNode);

    const startTime = Math.max(
      this.context.currentTime, this.nextStartTime
    );
    source.start(startTime);
    this.nextStartTime = startTime + buffer.duration;
  }
}

The nextStartTime scheduler ensures seamless playback regardless of network jitter. The latencyHint switch between mobile ("playback") and desktop ("interactive") was a subtle but important optimization -- mobile browsers handle audio buffers differently.

What I Learned Building This Solo

1. Build the unhappy path first. I spent week one on the happy path. Weeks two through four were entirely edge cases -- reconnection, echo suppression, barge-in handling. If I could redo it, I'd build error recovery before a single feature.

2. Voice is a different UX paradigm. Users don't read error messages mid-conversation. Every failure needs an audio fallback. We pre-compute "filler" audio chunks ("one moment please...") as 24kHz PCM, ready to stream instantly when Gemini is slow or reconnecting.

3. Speech-to-speech beats STT+TTS. We initially considered a Whisper -> Claude -> ElevenLabs pipeline. Gemini Live API's native audio mode is faster (sub-500ms round-trip), cheaper, and handles interruptions naturally. The trade-off: less control over the voice, but the latency gain is massive.

4. Cloud Run works for WebSockets, with caveats. We deploy on Google Cloud Run (me-west1 region). WebSocket connections survive container restarts thanks to session resumption handles saved in Firestore. The key setting: request timeout of 3600s (1 hour) for long interview sessions.

The Result

The system now runs validation interviews averaging 12 minutes of continuous voice conversation. Across 500+ sessions, hard failures dropped to under 1% after implementing the echo gate and session resumption. Each interview includes 3 AI agents (Alex for discovery, Sam for architecture, Maya for design) that use ~12 function-calling tools to research markets, analyze competitors, and generate reports -- all while maintaining a natural conversation.

Building this solo meant every failure landed directly in my Telegram inbox (via a monitoring bot). Which, honestly, is the fastest feedback loop possible.

What I'm Curious About

The biggest remaining challenge is audio quality on mobile browsers. iOS Safari handles AudioContext differently from Chrome, and some Android devices have aggressive echo cancellation that clips the AI's speech. We're currently using device-specific settings, but it feels like a hack.

Has anyone found a robust cross-browser audio playback strategy for real-time AI voice? Especially interested in experiences with AudioWorklet vs ScriptProcessorNode for this use case. Drop your thoughts in the comments.