Aeion AI + Groq — The Speed Leader
Configure Groq as a provider in Aeion AI. Groq runs Llama 4 (Scout + Maverick), Llama 3.3 70B, Mixtral 8x7B, and other open-weights models on their custom LPU silicon — at sub-100ms first-token latency. Best fit for any workload where latency dominates UX: real-time UI, voice interfaces, live agent loops, streaming chat.
Setup
- Get an API key at
https://console.groq.com/keys. Free tier provides generous developer quota.
Why Groq — The Latency Argument
Groq's pitch is one number: time-to-first-token.
Model Family
| Model | Aeion model string | $/1M in | $/1M out | Context | Best for |
When Groq Wins — Three Concrete Use Cases
Voice interfaces (Aeion Realtime / voice apps):
Common Gotchas
- Rate limits per-key tighter than OpenAI — Groq's free tier is 30 RPM, paid tier varies. For production: contact Groq for higher limits OR use multiple keys + round-robin via Aeion's provider config.
FAQ
Custom silicon (LPU — Language Processing Unit). Tape-out designed for transformer inference at extreme throughput. The LPU's deterministic execution path eliminates the variance + overhead of GPU inference.
Not yet — Groq runs the model catalog they host (Llama, Mixtral, Gemma, etc.). For custom fine-tunes, use Together AI or Fireworks (slower but supports custom models).
No — LLM + STT (Whisper) only. For embeddings, use Voyage / Mixedbread; for images, use Stability / Ideogram. Aeion's engine routes per-modality automatically.
Aeion's per-call `latencyMs` field in `ai_usage_logs`. Run a comparison from your own deployment: `ctx.ai.complete()` with the same prompt across a few providers, average TTFT over 100 calls per provider.
Use Groq as primary for latency-critical paths; fall back to another provider on Groq errors. For frontier-quality fallback, chain to OpenAI or Anthropic.