Aeion AI for Voice Products
Voice AI lives and dies by end-to-end latency. The right stack — Deepgram (STT) + Groq Llama 3.1 70B (LLM) + Cartesia Sonic (TTS) — hits p50 700ms voice-to-voice latency, below the ~1-second threshold where users perceive "real conversation" vs "stilted interaction." Aeion's engine orchestrates the stack with per-segment streaming for further latency cuts.
The Latency Budget
A "natural-feeling" voice interaction has a budget of ~1 second from "user stops speaking" to "AI starts speaking." Above 1.5s, users perceive lag; above 2s, the interaction breaks down.
Recommended Stack
| Component | Provider | Model |
Streaming Wiring
End-to-end streaming requires careful coordination across the three providers. Aeion's Realtime module gives you the WebRTC transport (via its LiveKit adapter); the AI engine gives you the per-modality primitives (transcribe, chatCompletionStream, speak). Wiring the three into a single low-latency voice loop — chunked audio in, streamed LLM tokens, sentence-boundary TTS kickoff — is integration work you do on top of these primitives; there's no single createVoiceSession() call that does it end-to-end today.
Cost Math for Voice Products
A voice product running 100K minutes/month of conversation.
FAQ
OpenAI Realtime is convenient but 10× more expensive AND ~3× slower (one provider's STT+LLM+TTS in series vs 3 specialists each tuned for their leg). For prototypes / low-volume, OpenAI Realtime is fine. For production scale, the specialist stack wins on both cost and latency.
There's no built-in interruption event today — you'd detect new speech via your own VAD/STT signal and, on the client or server side, stop playback of the current TTS audio and cancel/ignore the in-flight LLM stream before starting the new turn. Check current Cartesia API docs for their cancellation support rather than assuming a specific latency figure.
ElevenLabs is the leader for high-quality cloning (3-min sample → indistinguishable clone). Resemble + LMNT are alternatives. Aeion's `voice` field accepts a provider-specific voice ID; switch providers per-call.
Yes — Whisper-large-v3 on Ollama for STT, any LLM via Ollama for LLM, Kokoro on local GPU for TTS. Latency goes up to ~1.5s but data never leaves your hardware. Right for HIPAA / classified workloads.
Deepgram (Nova-3) + AssemblyAI both support diarization. Deepgram for streaming; AssemblyAI for batch with higher accuracy.
Aeion's Connect module (not Realtime) integrates with Twilio + Vonage for telephony. If you need this voice-AI stack reachable over a phone number, that's Connect's telephony transport, not Realtime's WebRTC path — the two modules serve different transports today rather than sharing one interchangeable layer.