Forem: Smallest AI

What Speech Recognition APIs Get Wrong About Human Speech

Smallest AI — Fri, 03 Apr 2026 10:39:29 +0000

We've spent decades teaching computers to read. It took considerably longer to teach them to listen and if you have the wrong accent, or work in a noisy room, the honest answer is, we haven't managed it yet. AI speech recognition is one of the most impressive technologies of the last decade and one of the most inconsistently experienced.

That gap between what your voice says and what the machine hears is the subject of this piece. Not because the technology isn't impressive it genuinely is but because the conditions under which it impresses are far narrower than the marketing suggests. Background noise, regional accents, technical jargon, multiple languages switching mid-sentence, each one chips away at headline accuracy numbers until what's left barely resembles the promise.

Understanding why this happens and what engineers are doing about it is worth the effort. Especially now, when voice commands are moving from novelty to infrastructure across healthcare, automotive, customer service, and industrial safety. When these systems fail, they don't fail quietly.

How a Machine Learns to Listen

Before we can understand why automatic speech recognition fails, it helps to understand what it's actually doing because it's stranger and more impressive than most people realise.

The process is not translation in the simple sense. It's closer to a high-frequency interpretation problem. A raw audio signal arrives as an analog sound wave. The system samples it digitally, then breaks it into tiny windows and converts each window into a visual representation called a log-Mel spectrogram. This spectrogram maps the intensity of frequencies over time, mimicking the way the human inner ear processes sound. The machine isn't listening to your words. It's looking at pictures of your voice.

In modern architectures like Smallest.ai's Pulse STT, the system scans these pictures for patterns consonants, vowels, the edges between them before anything resembling a word takes shape.

What comes next is the part that changed everything.

The Encoder-Decoder Transformer

The heart of a modern ASR system is an encoder-decoder transformer, and understanding it explains both the power and the fragility of what these systems do.

The encoder takes the sequence of audio features and transforms them into a context vector, a rich mathematical blueprint of the entire audio window. The critical mechanism here is self-attention, which lets the model look at the entire 30-second audio window simultaneously rather than processing it word by word. This global perspective matters, if a speaker says "bank" early in a sentence, the model uses context from the end of the sentence to determine whether the reference is financial or geographical.

The decoder then writes the transcript one token at a time, using cross-attention to refer back to specific parts of the audio blueprint as it goes. Each predicted word corresponds to an exact moment in the original sound.

What made this architecture a step-change was what it replaced. Earlier systems needed separate acoustic modeling, lexicon, and language modeling components each trained and maintained independently, each introducing its own failure modes. The encoder-decoder approach collapses all of this into a single end-to-end system, reducing development complexity and dramatically improving performance on well-represented speech. The tradeoff is that failures are also more holistic when the model doesn't know how to handle something, there's no fallback.

The Accent Problem Is a Data Problem

Here's the uncomfortable truth about speech-to-text accuracy statistics, they're almost always measured on audio that sounds like the training data.

Accents and dialects are not minor stylistic variations. They're complex shifts in phonetics, intonation, rhythm, and timing. A speaker from West Africa may use fundamentally different vowel lengths than a speaker from Appalachia, even while saying identical words in the same language. The model's job, what researchers call phonetic fuzzy matching, is to recognise that "savins" and "savings" are likely the same word despite a regional clip. When models aren't trained on sufficient diversity, they don't develop this tolerance.

The numbers tell the story clearly. A well-resourced English model might achieve a Word Error Rate (WER) of 3–5% in ideal conditions. Put that same model in a real-world environment with a non-standard accent, and WER can climb past 25%. For low-resource languages like Hindi or Mizo, real-world error rates of 30–50% are not uncommon.

Modern neural networks attempt to close this gap through continuous learning, feeding more diverse speech data into the system over time to expand its phonetic tolerance. Deep Neural Networks (DNNs) analyse audio signals for subtle variations in pitch and tone, learning to generalise across regional variation. The challenge is that this requires data, and collecting diverse, labelled speech data is expensive and slow. The communities most underserved by these systems are typically the communities least represented in training datasets. It's a self-reinforcing gap.

Code-Switching and the Multilingual Problem

The accent problem compounds significantly in multilingual recognition environments. Code-switching, where a speaker moves between languages in the same sentence, as hundreds of millions of people do naturally every day, breaks most conventional ASR pipelines entirely. The model expects one language at a time; it gets two, mixed without warning.

Modern systems like Smallest.ai's Pulse STT address this through auto-language detection and adaptive modeling, switching linguistic contexts mid-stream as evidence accumulates. The more advanced frontier is zero-shot performance, a model that can recognise or translate a language it has never explicitly trained on.

This is achieved by learning language-agnostic speech representations of the fundamental acoustic properties that all human speech shares regardless of language. By mapping these properties to a shared latent space, a model can extend support to new languages with minimal labelled data. Large Language Models (LLMs) increasingly act as the reasoning engine for this acoustic output, applying contextual understanding to bridge gaps where phonetic training is sparse.

What This Looks Like in Practice: The Multilingual Translator

Smallest.ai's Multilingual Translator is a working demonstration of these principles. The system provides real-time translation and voice output across multiple languages, a meaningful feature for educators and travellers in low-connectivity environments.

It's a useful case study because it makes the engineering tradeoffs visible. Supporting many languages isn't just a matter of adding more training data; it requires architectural decisions about how the model represents language, how it handles uncertainty, and how latency is managed when the system needs to detect, transcribe, and translate in near real-time. Privacy is handled by keeping inference local, no audio leaves the device.

Background Noise Is Not a Special Case. It's the Default.

If the accent problem is about variety, the noise problem is about interference. And interference is not the exception in real-world audio, it's the condition.

Traffic, machinery, HVAC systems, overlapping speakers, music bleeding from nearby rooms, these sounds contaminate almost every audio environment where voice-activated systems are actually deployed. Noise breaks speech-to-text by interfering with the acoustic cues a model depends on formants, pitch contours, the micro-pauses that signal word boundaries. At a Signal-to-Noise Ratio (SNR) below 10 dB, most conventionally-trained models begin to fail badly.

The instinct is to clean the audio before transcribing it. Spectral subtraction, Wiener filtering, noise gates, decades of preprocessing research. The problem is what engineers have started calling the noise reduction paradox where every filter designed to remove background hum also risks erasing the subtle speech harmonics the recogniser needs to identify a word. Spectral subtraction can improve SNR by 8 dB and simultaneously drive WER up by 15% through the distortion it introduces. You solve one problem and create another.

Current best practice has shifted toward noise-trained models systems trained on datasets that deliberately include chaotic acoustic conditions, rather than clean recordings. Instead of preprocessing the audio into something more tractable, the model learns to find stable acoustic features that persist even under heavy noise. The architecture learns noise tolerance rather than having it bolted on afterward.

Noise Robustness Method	Advantage	Disadvantage
Preprocessing (Denoising)	Works with legacy ASR backends	Can erase speech harmonics; adds latency
Noise-Trained Models	Handles chaotic audio without cascade errors	High training cost and data requirements
VAD Buffering	Trims 30–40% of compute costs	Introduces 20–50ms of additional latency
Multi-Channel Processing	Uses microphone arrays to isolate voice	Requires specialised hardware

Voice Activity Detection (VAD) plays a critical supporting role here identifying which segments of audio contain speech and which don't, reducing the computational load on the transcription model. But VAD introduces its own failure mode: if the frame window is too short, a low-energy consonant can be misclassified as silence, creating a deletion error in the final transcript that looks like a simple mishear but originates in preprocessing.

The sector tables from real deployments underscore how high the stakes are:

Sector	Primary Use Case	Critical Requirement
Healthcare	Real-time patient monitoring and documentation	High transcription accuracy for medical terms
Automotive	Voice-activated navigation and multimedia	Robustness to background noise and engine hum
Customer Service	Virtual assistants and automated triage	Low latency and accurate intent detection
Industrial Safety	Hands-free data collection and reporting	Resilience to 90+ dBA acoustic environments

The Latency Problem Nobody Talks About Enough

Accuracy is the metric people quote. Latency is the metric that determines whether anyone uses the product.

A conversation feels natural only when response time stays under 300ms. For a developer building a voice agent, the pipeline is to capture audio, transcribe it, pass it to an NLU layer, run it through an LLM, generate a response, synthesise speech, stream audio back. Every step costs time. The cumulative budget is brutal.

Modern systems prioritise Time to First Transcript (TTFT) the delay between a speaker stopping and the first words appearing as text. Pulse STT achieves a TTFT of 64ms, which creates the perceptual illusion of real-time interaction by returning partial transcripts while the speaker is still talking. These partials update continuously until the model commits to a final transcript at a natural pause, a process called endpointing.

Performance Dimension	Goal for Natural Conversation	Typical Cloud API
TTFT	< 100ms	200ms – 500ms
Total Response Latency	< 800ms	1500ms – 3000ms
Transcription Accuracy	> 95%	80% – 90% (in noise)
Endpointing Delay	< 300ms	500ms – 1000ms

Streaming via WebSockets

The architectural mechanism that makes low-latency real-time transcription possible is the WebSocket connection. Unlike REST APIs which require a new handshake for every audio packet WebSockets maintain a persistent, bidirectional link between client and server. The server pushes transcript fragments back as soon as they're processed, rather than waiting for the full audio to arrive.

A typical streaming architecture flows like this establish an authenticated WSS connection, stream 40ms audio packets (roughly 640 bytes at 8kHz sampling) at a continuous 1:1 real-time rate, then receive a stream of JSON objects containing partial results, final results, and word-level timestamps. The client gets a live view into what the model is thinking, not just a final answer. For a technical deep down refer to realtime audio transcription guide

Beyond Transcription: What Speech Intelligence Actually Means

Transcription is the starting point, not the destination. The more interesting question is what you can infer from speech that doesn't survive the conversion to text.

Speaker diarization: Answering "who spoke when?" is one of the most practically valuable capabilities. It's an unsupervised clustering problem, the system segments the audio, converts each segment into a high-dimensional numerical embedding of the speaker's unique vocal characteristics, estimates how many distinct speakers are present, then assigns labels (Speaker 1, Speaker 2, etc.). The output transforms a raw transcript into a structured conversation.

Word-level confidence scores: Each word in a transcript carries a probability score typically 0.0 to 1.0 representing how certain the model is about that prediction. A score of 0.95 is reliable; 0.60 is a flag. By setting a confidence threshold, an application can automatically route uncertain words to human review, ask the user for clarification, or simply annotate the output with uncertainty markers. In healthcare or legal contexts, where a single misheard word has real consequences, this metadata is not optional.

More advanced uncertainty estimation uses entropy-based measures that provide more calibrated estimates of correctness than raw probability scores alone.

Metadata Feature	Data Content	Key Use Case
Speaker ID	Integer / label for unique voices	Meeting minutes, interview archives
Emotion Tag	Sentiment (happy, angry, neutral, etc.)	Call centre coaching, sentiment analysis
PII Detection	Flagged sensitive data	HIPAA, PCI, GDPR compliance
Confidence Score	Probability (0.0 – 1.0)	Quality assurance and error correction

What Happens When You Chain These Systems Together

One of the more revealing experiments in ASR research isn't a benchmark, it's a failure mode. Smallest.ai's Voice Chinese Whispers demonstrates what happens when you chain transcription, translation, and speech synthesis in repeated loops.

In a single pass, a misheard word shifts meaning slightly. By the fifth iteration, the system is producing phrases that have no relationship to the original utterance. The model hasn't hallucinated in the classic LLM sense; it's been faithfully following the degraded output of the previous step. Each stage introduces a small amount of acoustic drift or contextual drift, and the errors compound geometrically.

It's a useful demonstration because it makes visible something that's easy to miss in production systems; the output of an ASR model is not a stable foundation. It's a probabilistic estimate, and downstream systems that treat it as ground truth will inherit and amplify its errors. Transcript stability ensuring that once a word is committed it stays committed, and that confidence scores accurately reflect uncertainty is an engineering discipline, not a given.

From Transcription to Action: The Real Ambition

The most significant shift happening in speech intelligence right now isn't about accuracy or latency. It's about what the transcript does.

In the speech-to-action paradigm, the ASR transcript is fed directly into an LLM that can call external tools, query databases, trigger workflows, and manage complex dialogue. The voice interface becomes a reasoning interface. The gap between "I said a thing" and "something happened" collapses.

This requires a level of integration between the speech layer and the reasoning layer that earlier architectures couldn't support. The emerging answer is full-duplex multimodal models where a single model handles voice input, reasoning, and voice output in one pipeline, rather than piping data between separate ASR, LLM, and TTS services. Smallest.ai's Hydra takes this approach, handling intent detection and voice synthesis together to eliminate the inter-service latency that makes stitched-together pipelines feel unnatural.

What Real-Time Voice AI Looks Like in Practice

Smallest.ai's Debate Arena is a working demonstration of how far orchestration has come. The system stages a philosophical debate between AI agents, Socrates arguing for and Aristotle arguing against any topic the user proposes with distinct voices, expressive vocal parameters (emotion, pitch, volume, prosody) predicted by the LLM each round, and an ancient Athenian judge scoring the exchange.

For a system like this to work through voice, the ASR layer needs to maintain multi-speaker tracking, support adversarial turn-taking without the agents talking over each other, and do all of this at low enough latency that the conversation feels alive. The Debate Arena uses Lightning TTS v3.2 WebSocket streaming, with voice parameters generated dynamically per round by GPT-4o-mini. It supports two modes, Philosophical and Roast Battle with escalating arguments and audience voting.

It's a playful project, but it demonstrates something serious: the engineering required to make multi-agent, multi-voice, real-time voice interaction work is now tractable. The primitives exist. The question is how to compose them well.

Where This Is Going

The next decade of AI speech recognition is likely to diverge along two paths that are pulling in opposite directions.

The first is to scale a massive cloud model trained on ever-larger and more diverse datasets, capable of handling more languages, more accents, more acoustic conditions. The second is compression, hyper-efficient on-device models that run locally on a phone or an industrial edge device without sending audio to the cloud. Privacy, data sovereignty, and latency concerns are all pushing toward the second path, even as raw capability improvements come from the first.

Adaptive and personalised speech models represent a third direction that cuts across both. Rather than building a single model that tries to be equally good at everything, future systems will adapt in real-time to an individual speaker's specific pitch, pace, and vocabulary. Zero-shot adaptation, learning to recognise a specific voice from a few seconds of reference audio makes this tractable without requiring per-user retraining at scale.

Building Things That Actually Work

For developers, the translation from research benchmarks to production systems requires moving past Word Error Rate as the primary metric. WER tells you how accurate the model is on a test set. It doesn't tell you whether users can trust it.

The metrics that matter in production:

Tail latency (P99): Does the system respond quickly under heavy load, or does it occasionally spike in ways that break the conversation?
Calibrated confidence: When the model reports 90% certainty, is it actually right 90% of the time? Overconfident models are more dangerous than uncertain ones.
Domain-specific adaptation: Does the system handle your vocabulary? Medical terms, product names, and technical jargon that don't appear in general training data can be addressed through word boosting and custom dictionaries.

Best Practice	Implementation	Outcome
Handle low confidence	Flag words below 0.90 for human review	Reduced error rate in high-stakes documents
Use WebSockets	Implement persistent WSS connections	Sub-500ms response times for voice agents
Adopt noise-trained models	Skip preprocessing in chaotic environments	Better performance in factories and vehicles
Monitor RTF	Track the Real-Time Factor of inference	Guaranteed responsiveness under load

Smallest.ai's ecosystem offers a set of tools built around these production constraints. Pulse STT delivers 64ms TTFT with built-in diarization across 30+ languages. Lightning ASR is optimised for sub-300ms latency, with particular strength in non-English languages. Hydra handles the full voice conversation pipeline such as input, reasoning, and output in a single model.

A Note on What "Working" Really Means

Market projections $19.5 billion by 2030, 27% of the global population already using voice commands tend to measure adoption, not satisfaction. A system that works for one speaker in a quiet room and fails another speaker in a noisy one is not a solved problem, even if it ships with impressive accuracy numbers.

The history of automatic speech recognition is a history of systems getting impressively good at well-resourced voices and incrementally better at everyone else. The architecture has genuinely improved encoder-decoder transformers, end-to-end training, and noise-robust learning are meaningful advances over the rule-based systems of the 1990s. But the generalisation gap that makes a 95% accuracy number in a lab become a 75% accuracy number in the field is not a technical afterthought. It's the central problem.

Building voice interfaces that are worth trusting means taking that gap seriously in the training data you choose, the confidence metadata you expose, the noise conditions you test against, and the communities whose voices you treat as primary cases rather than edge cases.

The era of voice-first interfaces hasn't simply arrived. It's arriving unevenly. And the engineers who understand why have a real opportunity to build something better.

Tools referenced in this piece: Pulse STT, Lightning ASR, Hydra, Multilingual Translator, Voice Chinese Whispers, Debate Arena

What Speech Recognition APIs Get Wrong About Human Speech

Smallest AI — Fri, 03 Apr 2026 10:39:29 +0000

How a Machine Learns to Listen

Before we can understand why automatic speech recognition fails, it helps to understand what it's actually doing because it's stranger and more impressive than most people realise.

In modern architectures like Smallest.ai's Pulse STT, the system scans these pictures for patterns consonants, vowels, the edges between them before anything resembling a word takes shape.

What comes next is the part that changed everything.

The Encoder-Decoder Transformer

The heart of a modern ASR system is an encoder-decoder transformer, and understanding it explains both the power and the fragility of what these systems do.

The Accent Problem Is a Data Problem

Here's the uncomfortable truth about speech-to-text accuracy statistics, they're almost always measured on audio that sounds like the training data.

Code-Switching and the Multilingual Problem

What This Looks Like in Practice: The Multilingual Translator

Background Noise Is Not a Special Case. It's the Default.

If the accent problem is about variety, the noise problem is about interference. And interference is not the exception in real-world audio, it's the condition.

Noise Robustness Method	Advantage	Disadvantage
Preprocessing (Denoising)	Works with legacy ASR backends	Can erase speech harmonics; adds latency
Noise-Trained Models	Handles chaotic audio without cascade errors	High training cost and data requirements
VAD Buffering	Trims 30–40% of compute costs	Introduces 20–50ms of additional latency
Multi-Channel Processing	Uses microphone arrays to isolate voice	Requires specialised hardware

The sector tables from real deployments underscore how high the stakes are:

Sector	Primary Use Case	Critical Requirement
Healthcare	Real-time patient monitoring and documentation	High transcription accuracy for medical terms
Automotive	Voice-activated navigation and multimedia	Robustness to background noise and engine hum
Customer Service	Virtual assistants and automated triage	Low latency and accurate intent detection
Industrial Safety	Hands-free data collection and reporting	Resilience to 90+ dBA acoustic environments

The Latency Problem Nobody Talks About Enough

Accuracy is the metric people quote. Latency is the metric that determines whether anyone uses the product.

Performance Dimension	Goal for Natural Conversation	Typical Cloud API
TTFT	< 100ms	200ms – 500ms
Total Response Latency	< 800ms	1500ms – 3000ms
Transcription Accuracy	> 95%	80% – 90% (in noise)
Endpointing Delay	< 300ms	500ms – 1000ms

Streaming via WebSockets

Beyond Transcription: What Speech Intelligence Actually Means

Transcription is the starting point, not the destination. The more interesting question is what you can infer from speech that doesn't survive the conversion to text.

More advanced uncertainty estimation uses entropy-based measures that provide more calibrated estimates of correctness than raw probability scores alone.

Metadata Feature	Data Content	Key Use Case
Speaker ID	Integer / label for unique voices	Meeting minutes, interview archives
Emotion Tag	Sentiment (happy, angry, neutral, etc.)	Call centre coaching, sentiment analysis
PII Detection	Flagged sensitive data	HIPAA, PCI, GDPR compliance
Confidence Score	Probability (0.0 – 1.0)	Quality assurance and error correction

What Happens When You Chain These Systems Together

From Transcription to Action: The Real Ambition

The most significant shift happening in speech intelligence right now isn't about accuracy or latency. It's about what the transcript does.

What Real-Time Voice AI Looks Like in Practice

Where This Is Going

The next decade of AI speech recognition is likely to diverge along two paths that are pulling in opposite directions.

Building Things That Actually Work

The metrics that matter in production:

Tail latency (P99): Does the system respond quickly under heavy load, or does it occasionally spike in ways that break the conversation?
Calibrated confidence: When the model reports 90% certainty, is it actually right 90% of the time? Overconfident models are more dangerous than uncertain ones.
Domain-specific adaptation: Does the system handle your vocabulary? Medical terms, product names, and technical jargon that don't appear in general training data can be addressed through word boosting and custom dictionaries.

Best Practice	Implementation	Outcome
Handle low confidence	Flag words below 0.90 for human review	Reduced error rate in high-stakes documents
Use WebSockets	Implement persistent WSS connections	Sub-500ms response times for voice agents
Adopt noise-trained models	Skip preprocessing in chaotic environments	Better performance in factories and vehicles
Monitor RTF	Track the Real-Time Factor of inference	Guaranteed responsiveness under load

A Note on What "Working" Really Means

The era of voice-first interfaces hasn't simply arrived. It's arriving unevenly. And the engineers who understand why have a real opportunity to build something better.

Tools referenced in this piece: Pulse STT, Lightning ASR, Hydra, Multilingual Translator, Voice Chinese Whispers, Debate Arena

Why Speech Recognition API Requires a Different Architecture

Smallest AI — Wed, 01 Apr 2026 09:18:26 +0000

Speech Recognition API: Streaming, WebSockets and Latency

A speech recognition API that accepts a file and returns a transcript is a solved problem.The architecture is simple because the constraints are simple.

Real-time transcription is different. The audio doesn't exist yet when processing needs to begin. The user is still speaking while the system needs to be building a hypothesis about what they said. The application needs a partial answer now, not a complete answer in two seconds. These constraints change the architecture at every layer, from how audio is captured and transmitted to how the recognition model processes it and how results flow back to the client.

This piece walks through that architecture end to end. Not as an API reference, but as an explanation of what is actually happening inside a streaming speech recognition system and why each component is designed the way it is.

The fundamental problem with batch transcription for real-time use

Before looking at how streaming ASR works, it helps to understand precisely why the batch approach breaks down when applied to real-time audio.

In a batch system, the flow is straightforward. Audio is captured, buffered until complete, sent to a recognition service via an HTTP POST request, processed server-side, and a transcript is returned in the response body. The model sees the entire utterance before producing any output. This gives it full context, which tends to produce accurate results.

The problem is time. If a user speaks for five seconds, the system cannot return any transcript until those five seconds of audio have been captured, transmitted, and processed. Even with a fast model, the user experiences a dead pause after finishing their sentence before anything happens. In a voice agent or real-time captioning system, that pause breaks the interaction.

The deeper problem is that buffering introduces a fundamental floor on latency that no amount of model optimization can eliminate. Even an infinitely fast model cannot return a transcript before the audio has been collected and sent. The latency is baked into the architecture.

Streaming ASR removes this floor by changing the fundamental contract. Rather than collecting audio and then processing it, the system processes audio as it arrives.

How a streaming ASR API receives audio

The first architectural shift in a streaming system is the transport layer. HTTP request-response is the wrong shape for continuous audio delivery.

A new HTTP connection carries significant overhead including DNS resolution, TCP handshake, TLS negotiation, and HTTP headers on every request. For a file upload, this overhead is negligible relative to the payload. For 20-millisecond audio packets arriving fifty times per second, it is prohibitive. The connection overhead would dominate the actual audio data.

A WebSocket connection solves this by establishing a single persistent connection that remains open for the duration of the session. The initial handshake happens once. After that, both sides can send data at any time without per-message overhead. The client pushes audio packets as they arrive from the microphone. The server pushes transcript events as they are produced by the recognition model. Neither side waits for the other to finish.

Python Batch Processing Example

Prerequisite: SmallestAI's API Key

import os
import requests

API_KEY = os.getenv("SMALLEST_API_KEY")
audio_file = "meeting_recording.wav"

url = "https://api.smallest.ai/api/v1/pulse/get_text"
params = {
    "model": "pulse",
    "language": "en",
    "word_timestamps": "true",
    "diarization":"true",
    "emotion_detection": "true"
}
headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "audio/wav"
}

with open(audio_file, "rb") as f:
    audio_data = f.read()

response = requests.post(url, params=params, headers=headers, data=audio_data)
result = response.json()

print("Transcription:", result.get("transcription"))

for word in result.get("words", []):
    speaker = word.get("speaker", "N/A")
    print(f"  [Speaker {speaker}] [{word['start']:.2f}s - {word['end']:.2f}s] {word['word']}")

if "emotions" in result:
    print("\nEmotions detected:")
    for emotion, score in result["emotions"].items():
        if score > 0.1:
            print(f"  {emotion}: {score:.1%}")

Output

The audio capture and network transmission run concurrently. The recognition server receives a continuous stream of small packets rather than waiting for a complete file.

Inside the recognition model: how streaming inference works

Once audio packets arrive at the recognition server, the ASR model needs to produce transcript output without waiting for the utterance to complete. This requires a different inference architecture from batch transcription.

Modern streaming ASR systems use a buffer that accumulates incoming audio packets and runs the recognition model against overlapping windows of that buffer. The window size is typically 250 to 500 milliseconds, much longer than the 20ms packet size, because the model needs enough acoustic context to make meaningful predictions. Each time new audio arrives, the window advances and the model produces an updated hypothesis.

The model's job at each step is to answer the same question. Given all the audio seen so far, what is the most likely transcript? The answer changes as more audio arrives. A word that looked like "their" in the first pass might resolve to "there" when the following words provide context. These updates produce the partial transcript stream.

The internal architecture of the recognition model is typically an encoder-decoder transformer. The encoder converts the incoming audio frames into a sequence of dense vector representations capturing phonetic and prosodic features. The decoder attends to those representations to produce token predictions, one sub-word token at a time, building the transcript incrementally.

What makes this work in streaming mode is a technique called chunked attention, where the encoder is constrained to attend only to audio within a rolling window rather than the full utterance. This means the model can produce outputs without waiting for the sentence to end, at the cost of slightly reduced accuracy on words near the end of the window where context is limited.

WebSocket Connection Example

# file name: websocket.py
import asyncio
import json
import os
import websockets

API_KEY = os.environ.get("SMALLEST_API_KEY")

async def transcribe_stream():
    # Build the WebSocket URL with query parameters
    params = {
        "language": "en",
        "encoding": "linear16",
        "sample_rate": "16000",
        "word_timestamps": "true",
    }
    query_string = "&".join(f"{k}={v}" for k, v in params.items())
    uri = f"wss://waves-api.smallest.ai/api/v1/pulse/get_text?{query_string}"

    # Connect with Bearer token authentication header
    headers = {"Authorization": f"Bearer {API_KEY}"}

    async with websockets.connect(uri, extra_headers=headers) as ws:
        print("Connected to Pulse STT")

        # Launch concurrent tasks: send audio & receive transcripts
        send_task = asyncio.create_task(send_audio(ws))
        recv_task = asyncio.create_task(receive_transcripts(ws))

        await asyncio.gather(send_task, recv_task)

async def send_audio(ws):
    """Read audio from a source and stream it to the WebSocket."""
    with open("audio_16k_mono.raw", "rb") as f:
        while True:
            chunk = f.read(4096) # Recommended chunk size
            if not chunk:
                break
            await ws.send(chunk)
            await asyncio.sleep(0.05) # Pace stream to simulate real-time

async def receive_transcripts(ws):
    """Receive and process transcript responses from the server."""
    async for message in ws:
        response = json.loads(message)
        if response.get("transcript"):
            status = "FINAL" if response.get("is_final") else "PARTIAL"
            lang = response.get("language", "unknown")
            print(f"[{status}] ({lang}): {response['transcript']}")

            # Access word timestamps if enabled
            if "words" in response:
                for word in response["words"]:
                    print(f"  {word['word']}: {word['start']:.2f}s - {word['end']:.2f}s")

asyncio.run(transcribe_stream())

Output

Word timestamps are a byproduct of the encoder's attention alignment. The model learns which audio frames correspond to which output tokens during training, and this alignment is surfaced as timing metadata without additional inference cost.

Endpointing and deciding when an utterance ends

One of the harder problems in streaming ASR is endpointing, which is detecting when the speaker has finished a turn rather than simply paused mid-sentence. This matters because the system needs to know when to commit a final transcript and when to keep accumulating audio for the current hypothesis.

Getting endpointing wrong in either direction has visible consequences. An endpointer that fires too early cuts off sentences, producing truncated transcripts. One that fires too late adds perceptible delay after the speaker finishes, because the application has to wait for the endpointer before it can act on what was said.

The simplest approach is energy-based voice activity detection. If the audio energy drops below a threshold for a fixed duration, the system assumes the speaker has finished. This works adequately in quiet environments but fails under noise, where energy never drops cleanly to silence, and for speakers who naturally pause mid-thought.

Better endpointing systems combine acoustic signals with semantic signals. The acoustic layer watches for energy drops and spectral changes that characterize sentence endings. The semantic layer, often a small language model running on the partial transcript, checks whether the utterance is syntactically complete. A partial transcript ending mid-noun-phrase is unlikely to represent a complete turn. One ending with a complete declarative sentence is more likely to be a real turn boundary.

The output of the endpointer determines when partial transcript events transition to final transcript events in the client. A partial event is a hypothesis update. A final event is a committed result that the application can act on.

The transcript event stream and how to handle it

A streaming ASR API produces a continuous stream of events rather than a single response. Each event carries a type field distinguishing partial from final results, the current transcript text, word-level metadata if requested, and a timestamp indicating where in the audio this result corresponds.

Partial Response (`is_final: false`)

{
  "session_id": "sess_12345abcde",
  "transcript": "the customer said they want",
  "is_final": false,
  "is_last": false,
  "language": "en",
  "words": [
    {"word": "the", "start": 0.00, "end": 0.12},
    {"word": "customer", "start": 0.14, "end": 0.52},
    {"word": "said", "start": 0.54, "end": 0.74},
    {"word": "they", "start": 0.76, "end": 0.90},
    {"word": "want", "start": 0.92, "end": 1.10}
  ]
}

Final Response (`is_final: true`)

{
  "session_id": "sess_12345abcde",
  "transcript": "the customer said they want a refund",
  "is_final": true,
  "is_last": false,
  "language": "en",
  "words": [
    {"word": "the", "start": 0.00, "end": 0.12},
    {"word": "customer", "start": 0.14, "end": 0.52},
    {"word": "said", "start": 0.54, "end": 0.74},
    {"word": "they", "start": 0.76, "end": 0.90},
    {"word": "want", "start": 0.92, "end": 1.10},
    {"word": "a", "start": 1.10, "end": 1.14},
    {"word": "refund", "start": 1.14, "end": 1.48}
  ]
}

Notice that the word "want" had a confidence of 0.72 in the partial event. The model was uncertain whether more audio would follow and change the interpretation. In the final event, with the complete context, it scores 0.91. The word did not change, but the model's certainty about it did.

This is why acting on partial transcript content is architecturally risky. A word at low confidence in a partial might resolve to something different in the final. Any downstream action triggered by the partial would have been based on an unstable input.

The correct pattern is to use partial transcripts for user-facing display only, where visible corrections feel natural and expected, and to gate all application logic on final transcripts.

Confidence scores and what they actually measure

Every word in a streaming transcript carries a confidence score, typically a probability between 0.0 and 1.0, representing how certain the model is about that prediction.

The confidence score is not a measure of whether the word is correct. It measures how much probability mass the model assigned to this particular output versus the alternatives it considered. A score of 0.95 means the model strongly preferred this word over all others it evaluated. A score of 0.60 means there were plausible alternatives that the model considered seriously.

Words with low confidence scores are disproportionately likely to be wrong, but the relationship is not one-to-one. A model can be highly confident and wrong, particularly on proper nouns or domain-specific terms not present in training data. And it can be somewhat uncertain and still produce the correct output.

The most useful application of confidence scores is flagging rather than filtering. Rather than discarding low-confidence words, mark them for downstream attention. In a customer service context, a low-confidence stretch in a critical part of a call is a signal to route the transcript for human review. In a voice agent, a low-confidence final transcript is a signal to ask for clarification rather than proceeding.

async def transcribe_with_flagging(file_path: str):
    async with AsyncWavesClient(api_key="YOUR_API_KEY") as client:
        result = await client.transcribe(
            file_path=file_path,
            word_timestamps=True,
        )


    words = result.get("words", [])

    return result["transcription"], words

Paralinguistic signals alongside the transcript

The acoustic signal carries information that survives the conversion to text and information that does not. Tone, emotional register, estimated speaker age and gender are all present in the audio and absent from the transcript. A recognition service that surfaces these as structured metadata gives the application something the transcript alone cannot provide.

Emotion detection tags the emotional register of each segment. Gender and age detection provide demographic signals. These are extracted during the same inference pass as the transcript, using acoustic features the encoder computes regardless. They come without the cost of a separate processing pipeline.

async def full_acoustic_analysis(file_path: str):
    async with AsyncWavesClient(api_key="YOUR_API_KEY") as client:
        result = await client.transcribe(
            file_path=file_path,
            language="en",
            word_timestamps=True,
            age_detection=True,
            gender_detection=True,
            emotion_detection=True,
        )

    print("Transcript:", result.get("transcription"))

    # Handle emotions as a dictionary
    if 'emotions' in result:
        print("Detected emotions:")
        for emotion, score in result['emotions'].items():
            print(f"  {emotion.capitalize()}: {score:.2f}")

    print("Detected gender:", result.get("gender"))
    print("Estimated age range:", result.get("age"))

These are probabilistic estimates. They work well in aggregate and should be treated as signals rather than ground truth, particularly for individual utterances where the acoustic evidence may be ambiguous.

Streaming TTS and closing the audio loop

A streaming ASR system rarely lives alone. In a voice agent, the transcript from the recognition service feeds a language model, which generates a response that must be converted back to audio and played to the user. The latency of that full loop determines whether the agent feels conversational.

The dominant contributor is LLM reasoning, typically 300 to 500ms even with a fast model. The highest-impact optimization is therefore starting TTS synthesis before the LLM has finished generating, feeding the streaming token output rather than waiting for the complete response.

Smallest.ai's WavesStreamingTTS connects to the synthesis service via a persistent WebSocket and accepts text chunks as they arrive from the LLM token stream. Audio chunks come back as each sentence is ready, so playback can begin in under 100ms from the first LLM token.

Basic Setup

from smallestai.waves import TTSConfig, WavesStreamingTTS
import wave

config = TTSConfig(
    voice_id="magnus",
    api_key="YOUR_SMALLEST_API_KEY",
    sample_rate=24000,
    speed=1.0,
    max_buffer_flush_ms=100,
)

streaming_tts = WavesStreamingTTS(config)

Standard Streaming

text = "Streaming delivers audio in real-time for voice assistants and chatbots."
audio_chunks = list(streaming_tts.synthesize(text))

with wave.open("streamed.wav", "wb") as wf:
    wf.setnchannels(1)
    wf.setsampwidth(2)
    wf.setframerate(24000)
    wf.writeframes(b"".join(audio_chunks))

Multilingual streaming and code-switching

Streaming ASR systems designed around a single language make assumptions that break in multilingual environments. A model trained primarily on English will produce poor results on Hindi and may fail on code-switching utterances that mix the two within a single sentence.

Smallest.ai's Lightning ASR model supports 30 languages including Hindi, German, French, Spanish, Italian, Portuguese, Russian, Arabic, Polish, Dutch, Tamil, Bengali, Gujarati, Kannada and Malayalam, with a multi mode for code-switching environments where speakers alternate between languages.

async def multilingual_transcription(file_path: str):
    async with AsyncWavesClient(api_key="SMALLEST_API_KEY") as client:
        result = await client.transcribe(
            file_path=file_path,
            language="auto",   # Auto Language Detection
            word_timestamps=True,
            emotion_detection=True,
        )
    return result["transcription"]

Code-switching is architecturally harder than single-language recognition because the model cannot assume a stable phoneme inventory or language model distribution. The multi mode handles detection and switching internally, removing the need for the application to route audio to different models based on detected language.

What the architecture means for how you build

Understanding how streaming ASR works at each layer changes what you build around it.

Because partial transcripts are revised as more audio arrives, any application logic that needs to act on what the user said must wait for a final transcript event. Displaying partials in a UI is fine because visible corrections feel natural. Triggering downstream actions, database lookups, tool calls, or routing decisions on partial content is architecturally unsound. The final event is the contract.

Because word confidence scores reflect model uncertainty rather than correctness, the right use is flagging rather than filtering. A word with 0.60 confidence is a candidate for human review or a clarification prompt, not something to silently drop from the transcript.

Because the endpointer determines when final transcripts are emitted, the responsiveness of the system to turn endings is determined by endpointing quality, not model speed. A fast model behind a slow endpointer still feels slow. Endpointing latency is worth measuring explicitly and separately from overall model throughput.

Because WebSocket connections carry state, connection management becomes an application concern. Dropped connections need reconnection logic. Audio buffered during a reconnect gap needs to either be replayed or explicitly acknowledged as lost. These failure modes do not exist in batch transcription and need to be designed for in streaming systems from the start.

The technology is capable. The architecture around it determines whether that capability reaches the user.

*Tools referenced in this piece: AsyncWavesClient, WavesStreamingTTS, Pulse STT

Forem: Smallest AI

What Speech Recognition APIs Get Wrong About Human Speech

How a Machine Learns to Listen

The Encoder-Decoder Transformer

The Accent Problem Is a Data Problem

Code-Switching and the Multilingual Problem

What This Looks Like in Practice: The Multilingual Translator

Background Noise Is Not a Special Case. It's the Default.

The Latency Problem Nobody Talks About Enough

Streaming via WebSockets

Beyond Transcription: What Speech Intelligence Actually Means

What Happens When You Chain These Systems Together

From Transcription to Action: The Real Ambition

What Real-Time Voice AI Looks Like in Practice

Where This Is Going

Building Things That Actually Work

A Note on What "Working" Really Means

What Speech Recognition APIs Get Wrong About Human Speech

How a Machine Learns to Listen

The Encoder-Decoder Transformer

The Accent Problem Is a Data Problem

Code-Switching and the Multilingual Problem

What This Looks Like in Practice: The Multilingual Translator

Background Noise Is Not a Special Case. It's the Default.

The Latency Problem Nobody Talks About Enough

Streaming via WebSockets

Beyond Transcription: What Speech Intelligence Actually Means

What Happens When You Chain These Systems Together

From Transcription to Action: The Real Ambition

What Real-Time Voice AI Looks Like in Practice

Where This Is Going

Building Things That Actually Work

A Note on What "Working" Really Means

Why Speech Recognition API Requires a Different Architecture

Speech Recognition API: Streaming, WebSockets and Latency

The fundamental problem with batch transcription for real-time use

How a streaming ASR API receives audio

Python Batch Processing Example

Output

Inside the recognition model: how streaming inference works

WebSocket Connection Example

Output

Endpointing and deciding when an utterance ends

The transcript event stream and how to handle it

Partial Response (is_final: false)

Final Response (is_final: true)

Confidence scores and what they actually measure

Paralinguistic signals alongside the transcript

Streaming TTS and closing the audio loop

Basic Setup

Standard Streaming

Multilingual streaming and code-switching

What the architecture means for how you build

Partial Response (`is_final: false`)

Final Response (`is_final: true`)