Voice Agent Runtime

A voice agent that runs entirely on a 4 GB laptop GPU.

Speech in, speech out, with no cloud inference. The median reply starts 2.07 s after you stop talking, measured at the client over 179 turns.

Watch the demoGitHubBenchmark report

Measured

2.07 s
Median, end of speech to first audio
p95 2.52 s
1.27 s
Same, without the 0.8 s silence window
median
0.28 s
LLM time to first token
p95 0.44 s
2.1 GB
Peak VRAM, of 4 GB
the LLM only

Where the 2.07 seconds go

Each part is its own median, so they add to about 2.07 s and not exactly. About 40% of the delay is waiting to be sure you have stopped speaking, and about a third is transcription on the CPU. The models are not the bottleneck.

179 of 180 turns across three fresh-start runs, 12 pre-recorded utterances, RTX 3050 Laptop 4 GB, Ryzen 5 6600H. The speech is synthetic and single-voice, and the client is a gRPC client, not a browser, so browser capture and WebRTC playout are not included. One call at a time. Faster-Whisper base on the CPU, Qwen2.5 3B through Ollama on the GPU, Piper on the CPU. How each number is measured.

How it is built

A browser connects to AetherRTC, a WebRTC gateway written in Go, integrated with this runtime. It passes audio to a Go orchestrator over gRPC, and the orchestrator streams it to a Python inference engine over a second bidirectional gRPC stream. Audio keeps arriving while a slow model is busy, and replies are spoken sentence by sentence as the model writes them.

The GPU has 4 GB, so only the language model uses it. Speech recognition, speech synthesis and voice activity detection run on the CPU.

  1. 1BrowserMicrophone and speakerclient
  2. 2AetherRTCGo, WebRTC, G.711 audioCPU
  3. 3OrchestratorGo, session state, gRPCCPU
  4. 4InferenceSilero VAD, Faster-Whisper, PiperCPU
  5. 5Language modelQwen2.5 3B through OllamaGPU

Demo recording

A recording of a local run.

What it does not do yet

No barge-in
You cannot interrupt a reply. The microphone is muted while the agent speaks.
Short words can be missed
In a 50-trial replay, the one-word utterance "Hello." was missed 14% of the time by the voice activity detector. The cause is narrowed to its onset confirmation rule and is not fixed.
An unexplained stall
In browser testing, a call has stopped answering after about two exchanges. It did not reproduce through the gRPC path, and the cause is unknown.
Not tested
Concurrent calls, background noise, and real accents. The benchmark audio is synthetic.

Write-ups