Realtime Infrastructure with AI Participants
Every business object gets a real-time session. Every session can have an AI participant that takes real actions via 89 MCP tools. Self-hosted on your own Hetzner VPS — zero per-minute fees, ever.
One Primitive, Every Module
Seven Session Types, One Engine
Meetings, review rooms, match rooms, spatial scenes, live streams, support sessions, and telehealth visits all run on the same session primitive — your team learns one permission model, one recording pipeline, and one audit trail instead of five different broadcasters bolted together over time.
Four Packet Lanes, Tuned Per Message
Reliable-ordered, reliable-unordered, unreliable-ordered, and unreliable-unordered delivery lanes let each message type move at the right trade-off — chat and file transfers arrive guaranteed and in order, while live cursor positions and voice-activity pings drop stale packets instead of queuing behind them.
True Peer-to-Peer DataChannel Mesh
On the unreliable lanes, WebRTC DataChannel mesh gives you real UDP semantics between participants — no server-relay tax on latency-sensitive traffic like live drawing or spatial position updates.
Live Connection Quality, Not Guesswork
RTT, jitter, packet loss, and MOS score are tracked per connection in real time, so when a participant says "the call was choppy," you have the telemetry to know whether it was their network or yours.
Sessions Attach to the Work, Not Just the Call
Link a session directly to a deal, ticket, project, or scene — the recording, transcript, and AI actions taken during that call live right next to the record they're about, not in a separate meetings app you have to cross-reference.
Summon an AI Participant, Not Just a Chatbot
Any session can pull in an AI participant with access to 89 MCP tools — it doesn't just watch and summarize, it takes real actions: creating notes, updating records, posting summaries, the same way a human co-host would.
Compliance Profiles Built In, Not Bolted On
Assign HIPAA, PCI, or GDPR-strict compliance profiles per tenant, and every session of that type automatically inherits the right recording rules, audit requirements, and data-handling behavior — no separate compliance layer to maintain.
The Realtime Architecture
AI Participants
Full lifecycle management — summon, list, and dismiss agents on demand, with cost accounting and conversation history tracked per invocation, and every MCP tool call routed to exactly the module it targets.
AI Turn Handling
Turn-by-turn orchestration keeps agents responsive without runaway loops — idle participants disconnect automatically, and progress checkpoints every 30 seconds so a platform restart never loses an agent's place mid-conversation.
Tool Routing
Resolves any of 89 MCP tools against the session's live context and dispatches each call to the right module — the agent doesn't need to know where a tool lives, just what it needs to do.
Universal Sessions
One room/peer/connection primitive replaces five separate per-module broadcasters that used to need independent maintenance, upgrades, and bug fixes.
Usage-Limit Engine
Enforces meeting, recording, transcription, and agent-budget limits per tenant automatically, so plan enforcement never depends on a human checking a spreadsheet.
Transcription Chain
Tries Whisper.cpp locally first (free, unlimited), then falls back through Cloud Whisper, OpenAI, and AWS automatically — you get the cheapest option that works, every time, with no manual provider juggling.
Transport Manager
Coordinates coturn, LiveKit, and Cloudflare transport under one roof, automatically resolving the best connectivity path so most users never have to think about NAT traversal at all.
AI Participants That Take Real Actions
Not a chatbot in a sidebar. An actual participant in the session.
AI participants join sessions, read context, take actions via MCP tools, and respond in real-time. They are not pre-scripted — they use the same AI infrastructure as the rest of Aeion OS.
Pre-built agents:
- Review Producer — monitors CUT video review sessions, auto-tags chapters, flags quality issues, generates post-session summary
- Helpdesk Triage — joins support sessions, classifies incoming issues, suggests KB articles, escalates appropriately
- AI Tutor — joins LMS class sessions, answers student questions, tracks comprehension gaps, generates post-class report
- Spatial NPC — joins Spatial architecture sessions, guides tours, explains building systems, answers spatial queries
- Telehealth Scribe — joins telehealth sessions, writes structured clinical notes, surfaces relevant history
Summon an agent into a session:
POST /v1/agents/summon
{
"agentDefinitionId": "review-producer",
"sessionId": "session_abc123",
"sessionContext": { "sessionType": "review", "projectId": "proj_xyz" },
"idleTimeoutSec": 900
}
The agent joins the session, reads context, and starts taking actions via 89 MCP tools — creating notes, updating records, posting summaries.
How agent turns work:
- Each message: the agent reads session context, an LLM generates a response, and any needed tools are called automatically.
- Up to 5 tool-call steps per turn, so an agent can't spin in a runaway loop.
- Progress is checkpointed every 30 seconds for crash-safe recovery.
- Idle agents automatically disconnect after their configured timeout.
- If the platform restarts, any agent still active in a session resumes automatically from its last checkpoint.
Self-Hosted — No Per-Minute Fees
| Provider | 100 users × 50 hrs/mo | Monthly cost |
|---|---|---|
| Twilio Video | 50K participant-minutes | ~$2,500 |
| Daily.co | 50K participant-minutes | ~$1,500 |
| Vonage Video API | 50K participant-minutes | ~$2,000 |
| **Aeion Realtime (self-hosted)** | Unlimited | **$0 infrastructure** |
Transcription — Free or Premium
Whisper.cpp local (free, unlimited): Runs on your own VPS. No API keys. No per-minute cost. Works offline. Accuracy ~85-90% for clean audio. Good enough for most use cases.
Cloud transcription chain (automatic fallback):
Recording completed
→ Try Whisper.cpp (local, free)
→ If fails: try Cloud Whisper (higher accuracy)
→ If fails: try OpenAI Whisper API
→ If fails: try AWS Transcribe
→ On final failure: emit realtime.transcription.failed event
→ Operator can retry manually
→ Recording itself remains intact
The transcription pipeline auto-selects the best available provider and handles retries — recordings that are queued when a provider goes down get retried automatically. Results are stored with speaker labels, timestamps, and word-level confidence scores, and are fully queryable via the API.
Compliance That Passes Audits
| Profile | Use Case | Requirements |
|---|---|---|
| **HIPAA** | Telehealth, health providers | BAA-ready, audit logging, PHI isolation |
| **PCI** | Payment sessions, subscription management | Cardholder data isolation, encrypted recording |
| **GDPR-strict** | EU residency, strict data residency | EU-only storage, right-to-erasure cascade, no US transfers |
Pick Your Transport, Honest No-Ops Everywhere
| Option | When to use |
|---|---|
| **Embedded LiveKit** | Self-hosted SFU on your own VPS — the default. Zero per-minute fees. Full LiveKit feature surface. |
| **Coturn relay metrics** | Live visibility into your self-hosted TURN relay — bytes relayed, allocation rate, active sessions — for the 20-30% of users behind NAT. |
| **Recording & egress** | Composite or per-track export to RTMP / HLS / your own S3-compatible storage — the same pipeline used for Cut review-room recording. |
| **Coturn relay control** | Start/stop, health checks, and a crash circuit-breaker so a TURN outage degrades gracefully instead of looping. |
| **Room administration** | Server-side room operations — kick participants, mute/unmute, dispatch admin events. |
| **External LiveKit** | Point at LiveKit Cloud or a separately hosted LiveKit cluster instead of running it on the same VPS. |
| **AI-powered transcription** | Plugs your tenant's configured AI provider (OpenAI, Anthropic, custom) into the transcription pipeline. |
| **Signed participant tokens** | Time-bounded, room-scoped tokens control who can publish or subscribe. |
| **Managed TURN lifecycle** | Configuration and lifecycle management for your self-hosted TURN relay, generated from your tenant settings. |
| **Cloudflare Calls** | Opt-in alternative when you want Cloudflare's global edge — still zero per-minute fees on the free tier. |
| **Webhook verification** | Every room event (created, participant joined/left) is signature-verified before it reaches your event stream. |
| **Bring-your-own TURN** | Point at an existing TURN server (Twilio, Vonage, or your own infrastructure) with your own credentials. |
| **Server config generation** | Generates server config from tenant settings — egress storage credentials, recording bucket, ingress URLs. |
| **Honest fallback** | If no TURN relay is configured, the platform says so plainly rather than silently degrading. STUN-only mode still works for ~70-80% of users on permissive NATs. |
Five Pre-Built Agent Seeds — Plus the Admin/Budget Plumbing
| Agent seed | What it does |
|---|---|
| **Spatial NPC** | Joins Spatial 3D sessions as an NPC participant — guides tours, explains building systems, answers spatial queries. |
| **Review Producer** | Monitors Cut video review sessions, auto-tags chapters, flags QC issues, generates post-session summary. |
| **Helpdesk Triage** | Joins support sessions, classifies incoming issues, suggests KB articles, escalates appropriately via the Helpdesk module. |
| **AI Tutor** | Joins LMS class sessions, answers student questions, tracks comprehension gaps, generates post-class report. |
| **Telehealth Scribe** | Joins telehealth sessions, writes structured clinical notes, surfaces relevant history. HIPAA-compliant audit trail. |
REST API Surface + Collections + Cron
| Data | What it holds |
|---|---|
| **Recording Sessions** | Recording lifecycle — pending / recording / processing / archived / failed |
| **Sessions** | Universal session record — type, linked business object (deal / ticket / project / scene), participants, compliance profile |
| **Transcripts** | Transcript text with speaker labels, word-level timestamps, confidence scores |
| **Agent Invocations** | Per-invocation cost ledger, tool-call history, checkpoint state for crash recovery |
| **Consent Records** | Per-participant consent grants + revocations, signed audit trail |
| **Compliance Events** | HMAC-SHA256 signed events — GDPR cascades, retention enforcement, admin bypass usage, key rotations |
| **Agent Definitions** | Agent registry — name, system prompt, allowed tools, idle timeout, budget caps |
| **Budget Alerts** | 50% / 80% / 100% threshold alerts per tenant per period |
| **Audit Signing Keys** | Signing keys with 90-day rotation (checked daily); rotated keys retained 7 years for verification |
| **Session Participants** | Participant snapshot — joined-at, left-at, role, role changes, network quality samples |
| **Session Resources** | Resources attached to a session — uploaded files, shared screens, whiteboards |
Usage Limits — Included Free, Configurable Per Tenant
Realtime is platform plumbing — included free with any Aeion module, no separate price and no tier of its own. Every tenant gets the full engine: unlimited concurrent meetings, unlimited participants, unlimited recording and transcription hours. Admins set their own per-tenant caps (concurrent meetings, max participants, monthly recording/transcription hours, AI agent budget) from the Realtime admin panel to match their own usage policy — not a vendor-imposed ceiling.
Budget alerts: when a tenant's monthly AI agent budget crosses 50% and 80% consumption, alerts fire. At 100%, new agent invocations are blocked until the budget resets the next month.
Connection rate limiting: up to 100 new WebSocket connections per minute per tenant, protecting against connection-flood attacks. Requests beyond the limit receive a 429 response.
Frequently Asked Questions
Every collaborative experience in Aeion — video meetings, live chat, real-time collaborative editing, live dashboards, signage updates — runs on the same Realtime engine. Centralizing it means consistent RBAC, audit, recording, and cost-management. Module surfaces the engine for direct use beyond what the higher-level modules cover.
Yes. Aeion deploys LiveKit on your tenant's VPS (Hetzner / OVH / DigitalOcean / your own hardware). All session traffic stays on your infrastructure. Zero per-minute fees, zero per-participant fees — your only cost is the underlying compute + bandwidth. Tested to 100+ concurrent participants per session on modest hardware.
Yes — recordings target your S3 / R2 / Wasabi / MinIO bucket. Aeion never holds recording bytes. Encryption at rest configurable (server-side or customer-managed key). Retention policies per session type.
An AI participant joins a session like a human participant — picks up the audio stream, transcribes in real time, can speak responses via TTS, can take actions via the platform's 89 MCP tools. Useful for AI tutor in education sessions, AI meeting-summarizer that takes notes + assigns action items, AI medical-scribe in telehealth.
HIPAA-strict (no recording to external storage, audit-log on every access), PCI-strict (no card data in audio/text), GDPR-strict (consent gating at session start, right-to-erasure on session data). Compliance profile per session type; once enabled, sessions can't downgrade out of it.
Whisper.cpp runs on-device (Bridge GPU) for HIPAA / privacy-strict sessions; cloud Whisper (or AssemblyAI / Deepgram) for non-sensitive. Per-session-type configurable. Multilingual transcription with auto-language detection.