A distributed, real-time voice agent runtime.
Built from scratch on consumer hardware.
Go orchestrator. Python inference engine. gRPC bidirectional stream. Sub-1.1s end-to-end latency on an RTX 3050 Mobile.
Capable of true bidirectional duplex, intelligent barge-in, and dynamic domain switching.
STT to first audio byte, warm, on consumer hardware. No cloud APIs.
Faster-Whisper on CPU. qwen2.5:3b on GPU via Ollama. Piper TTS on CPU. Nothing fights for VRAM. Everything runs completely locally without external dependencies.
Architecture
A clean separation of concerns. Go owns the concurrency, session lifecycle, and state machine. Python owns the heavy inference workloads.
Neither knows the other's internals. They communicate exclusively over a bidirectional gRPC stream, ensuring rock-solid stability even under heavy load.
- Low-overhead transport layer
- Strict boundary enforcement
- Independent environment scaling
Go Orchestrator
Python Inference Engine
Latency Breakdown
Every stage measured. No synthetic benchmarks.
| Stage | Metric | Value |
|---|---|---|
| VAD | Speech boundary detection | real-time, per 32ms |
| STT | Transcription (warm) | 0.70 – 0.82s |
| LLM | Time to first token | 0.43 – 0.58s |
| TTS | First chunk (warm) | 0.06 – 0.18s |
| End-to-end | STT → first audio byte | ~1.0 – 1.1s |
Measured on RTX 3050 Mobile 4GB VRAM · Ryzen 5 6600H · 16GB DDR5. Warm model, fresh boot, idle GPU.
What's Been Built
The milestone timeline. Green is shipped. Amber is actively building.
gRPC Bridge
Bidirectional stream, proved transport layer
Local Inference Pipeline
STT → LLM → TTS, no external APIs
Dynamic Agent Profiling
YAML personas, runtime domain switching
Session State Machine
Mutex-protected FSM, validated transitions
Token Streaming
Sentence-boundary chunking, parallel TTS
True Bidirectional Duplex
Open stream across utterance boundaries
VAD Integration
Silero VAD, four-state debounce, lookback buffer
Monitor Goroutine
Barge-in, timeout, context cancellation
AetherRTC Bridge
WebRTC edge gateway → orchestrator
Public Demo
Hosted end-to-end voice session
Hardware-Aware by Design
Most voice pipelines fail locally because they cram STT, LLMs, and TTS entirely into VRAM, causing out-of-memory crashes on consumer cards.
The RTX 3050 Mobile has just 4GB of VRAM.
By intentionally placing Whisper and Piper on the CPU, the LLM has exclusive access to the GPU. Nothing competes. Nothing OOMs.
| Stage | Component | Device | VRAM |
|---|---|---|---|
| VAD | Silero VAD (ONNX) | CPU | — |
| STT | Faster-Whisper int8 | CPU | — |
| LLM | qwen2.5:3b via Ollama | GPU | ~2.2GB |
| TTS | Piper TTS | CPU | — |
Stack
Transport & Orchestration
- Go 1.26
- gRPC / Protocol Buffers
- Pion WebRTC (AetherRTC)
- gorilla/websocket
Inference
- Python 3.11
- Faster-Whisper 1.2.1
- Ollama 0.6.2 + qwen2.5:3b
- Piper TTS 1.4.2
- Silero VAD (ONNX)
- onnxruntime 1.26.0
Infrastructure
- grpcio 1.81.0
- Protocol Buffers 6.33.6
- numpy 2.4.6
- scipy (resampling pipeline)
From the blog
Building True Bidirectional Duplex in a Distributed Voice Pipeline
How a three-worker Python architecture, a threading.Event gate, and an explicit proto signal replaced half-duplex voicemail exchange with a real concurrent call model.
AetherRTC
In DevelopmentA standalone WebRTC edge gateway that terminates SDP negotiation, ICE traversal, DTLS handshakes, and G.711 codec handling at the edge — so the Voice Agent Runtime never touches WebRTC internals. AetherRTC is the front door. VAR is the brain.
Stay in the loop
The runtime is not publicly hosted yet. Leave your email — I'll reach out when a live demo is available.