Realtime Module Technical Specifications

What an invocation tracks: each time an agent is summoned into a session, the platform tracks its status (running / completed / failed / budget exceeded / idle-disconnected / cancelled), a running cost total, the number of tool calls made, start/end times, and the full conversation history — checkpointed every 30 seconds so nothing is lost if the process restarts.

What an agent definition holds: a name and description, the system prompt that governs its behavior, which LLM provider and model it uses, the list of MCP tools it's allowed to call, a human-readable skill list for the admin UI, an optional monthly budget cap, an active/inactive flag, and an optional compliance profile (HIPAA / PCI / GDPR-strict) that restricts which providers and storage it can use.

Summoning an agent runs through a set of pre-checks, in order:

1. The agent definition exists and is active — otherwise the request is rejected.
2. The tenant's overall monthly agent budget isn't exhausted.
3. The specific agent's own monthly budget cap (if any) isn't exceeded.
4. The target session exists and hasn't ended or been cancelled.
5. The requested compliance profile is actually allowed for that provider.
6. If the same agent is already running in that session, the existing invocation is returned rather than creating a duplicate.

Once summoned, the platform starts the agent's runtime; if startup fails, the invocation is marked failed with a clear reason.

Dismissing an agent can be immediate (disconnects right away, marked cancelled) or graceful (the agent runs one final summary turn, then disconnects, marked completed).

Every tool call an agent makes is recorded against the invocation: cost is added to the running total, the tool-call count increments, and budget thresholds are checked. If the agent's budget is exceeded mid-session, it's disconnected automatically.

Budget threshold alerts fire at 50%, 80%, and 100% of an agent's monthly budget — each threshold only fires once per invocation. At 100%, new invocations for that agent are blocked; at 50% and 80%, you're just notified.

Boot recovery: if the platform restarts while agents are mid-conversation, running invocations resume automatically from their last checkpoint. Any invocation whose last checkpoint is more than 5 minutes stale is marked failed rather than left running indefinitely. The built-in agent seeds (Review Producer, Helpdesk Triage, AI Tutor, Spatial NPC, and the Telehealth Scribe) are re-installed automatically at boot if missing.

How a conversational turn works: when a message arrives for an active agent, the platform loads that agent's running invocation, gathers session context (participants, attached resources, recent history), appends the new message, and sends the full conversation to the LLM. If the model's response calls for tools, each tool is executed and its result is appended back into the conversation before the LLM produces its final reply. The reply is broadcast to the session and the conversation state is checkpointed.

Guardrails on every turn:

• Up to 5 tool-call steps per message — prevents a model from looping on tool calls indefinitely without producing a final response. If the limit is hit, the agent's last response (or a brief apology) is used instead.
• State is checkpointed every 30 seconds, so a crash or restart loses at most 30 seconds of conversation state — and resumes automatically afterward.
• Idle timeout is configurable per invocation and resets on every activity; agents that go quiet for longer than their timeout are automatically dismissed.
• Agents can participate in voice/video sessions or in text-only, data-only sessions (e.g. Spatial NPC guides, Collab chat) without needing a full video connection.

When a session ends, any agent still active in it runs one final summary turn, broadcasts that summary, and is dismissed gracefully — so a conversation's takeaways aren't lost just because the call ended.

How a tool call is handled: the platform resolves the requested tool to the right module and method, loads the session's business context (linked deal/ticket/project/scene, participants, recent activity), calls the underlying service with that context, and returns a structured result (success/failure, data, error, timing) back into the conversation. Every tool call is recorded to the invocation's audit trail.

Every tool call automatically carries full session awareness — which deal, ticket, project, or scene the session is attached to, who's participating, and what's happened recently — so the agent's actions are always grounded in the right business context, not just the current message.

Categories of tools available:

• Collab: send message, list messages, update message, add reaction, create thread, list threads, update participant role
• CRM: get contact, update contact, create activity, get deal, update deal stage
• Helpdesk: get ticket, update ticket status, add reply, escalate, get KB articles
• Commerce: get product, check inventory, get order, update order status
• Notes: create note, get notes, update note, attach to session resource
• Tasks: create task, get tasks, update task, assign to session participant
• LMS: get course, get student progress, create enrollment, post announcement
• Spatial: get scene, get asset, get spatial query response, update annotation

Every recording is transcribed through an automatic fallback chain: Whisper.cpp (local, free, unlimited) is tried first; if it fails, Cloud Whisper (higher accuracy) is tried; then OpenAI Whisper ($0.006/min); then AWS Transcribe ($0.024/min). If every provider fails, the recording stays intact and you can retry transcription manually — nothing is lost. Each completed transcript records which provider produced it, the detected language, the full text, and per-word timestamps and confidence scores.

Speaker diarization (separating who said what) can be enabled per agent, up to 8 distinguishable speakers. Whisper.cpp doesn't support diarization on its own; Cloud Whisper and OpenAI do, for an additional $0.002/min.

Reliability: cloud transcription jobs are queued and processed by dedicated workers with automatic retries (3 attempts with backoff). Jobs that still fail after retries are held for manual review rather than silently dropped.

Consent states: pending (waiting for a response), given (participant accepted), declined (participant rejected — their audio is server-muted), revoked (mid-recording withdrawal — muted immediately), and admin bypass (an admin started recording without consent — always logged as an audit event).

How it works: a recording can't start until every participant has given consent. A participant who declines has their audio muted server-side, but can change their mind mid-session and be unmuted. A participant who revokes consent mid-recording is muted immediately and can't be unmuted without giving fresh consent — audio already captured before the revocation is retained, and nothing further from that participant is captured for the rest of the recording.

Admin bypass is an explicit, opt-in project setting. When used, the recording proceeds for all participants, but the bypass is always recorded as an audit event and flagged on the compliance dashboard for review.

Every compliance event is signed. Each signed record carries what happened (GDPR cascade, key rotation, retention enforcement, consent change, etc.), who or what it was about, who performed it, a human-readable summary, structured details (encrypted at rest), a timestamp, a cryptographic signature, and the ID of the key that produced it.

Key rotation: the active signing key rotates automatically every 90 days (daily check at 03:00 UTC, before the 04:00 archive and 05:00 retention runs, so everything written that day is signed under the current key). When a key rotates, the previous key is marked rotated rather than deleted — it's retained for 7 years so old signatures can still be verified — and the rotation itself is recorded as a signed compliance event. Signing keys are themselves encrypted at rest.