Media Architecture & Technical Specs
Local FFmpeg via the Aeion Bridge, multi-provider AI transcription, Twelve Labs semantic search, and enterprise storage abstraction.
Capability Breakdown
Local Processing
— FFmpeg transcoding, image processing, OCR, PDF generation, waveform extraction
Storage Abstraction
— Multi-backend routing (S3, GCS, Azure, R2, MinIO, local)
Unified Media Analysis
— AI orchestrator (tags, description, NSFW, objects, faces, OCR)
Image Processing
— Resize, crop, focal point, WebP/AVIF conversion, optimization
Watermarking
— Text/image watermarks, presets, batch apply
Transcription
— Whisper, Deepgram, AssemblyAI, Google STT, speaker diarization
Semantic Video Search
— Video semantic indexing, natural language search, scene detection
AI Video Pipeline
— Full pipeline: transcription, NSFW, objects, chapters, activities
Asset Management
— Upload, variants, watermark, focal point, storage management
Upload Validation
— MIME types, sizes, magic bytes, virus scanning hooks
Media Analytics
— Views, downloads, shares, storage usage, bandwidth
Video Delivery
— Thumbnails, streaming URLs, HLS/DASH manifests
1. Local Media Processing via Bridge
| Task | Tool | Output |
|---|---|---|
| Video transcoding | FFmpeg | MP4/MKV/MOV with codec/bitrate |
| Image resize/convert | ImageMagick/sharp | WebP, AVIF, JPEG |
| Thumbnail extraction | FFmpeg | JPEG/PNG/WEBP at timestamp |
| Media probe | FFprobe | Duration, codecs, resolution, bitrate |
| PDF generation | Headless browser | PDF from HTML |
| OCR | Tesseract | Extracted text from images |
| Audio waveform | FFmpeg | JSON peaks array, normalized -1 to 1 |
2. Multi-Provider AI Transcription
| Provider | Model | Features |
|---|---|---|
| OpenAI Whisper | whisper-1 | Word-level timestamps |
| Deepgram | Nova-2 | Real-time, pretrained/fine-tuned |
| AssemblyAI | Latest | Speaker diarization, PII redaction |
| Google STT | Latest | Multi-language, streaming |
3. Twelve Labs Semantic Video Search
Aeion integrates with Twelve Labs to power natural language video search.
What indexing returns: a video ID, the index timestamp, and the video's duration, so every indexed video is trackable and searchable going forward.
What search returns: for each match — video ID, confidence score, the matching start/end timestamps, and a thumbnail.
Indexing Pipeline:
- Upload video URL to Twelve Labs
- Wait for indexing (async, status polling)
- Index complete → semantic search available
Natural Language Search Examples:
- "find videos showing a person running"
- "scenes with blue sky and ocean"
- "clip of someone typing on keyboard"
- "moment when a car drives through a puddle"
Search Parameters:
- A natural language query description
- A minimum confidence threshold (default: 0.1)
- Search granularity: clip (segment) or shot (camera change)
Multi-Index Support:
Multiple indexes are supported (e.g. marketing, training, archive), each acting as a separate namespace for search — so a training-video archive and a marketing asset library never surface irrelevant cross-matches.
4. NSFW Detection
Aeion uses vision AI for content moderation, checked per category against two global thresholds.
Configuration:
Moderation can be turned on or off per tenant, with a category list — adult, racy, violence, spoof, medical — each independently toggled on or off (adult and violence are on by default; racy, medical, and spoof are opt-in). Two thresholds apply across whichever categories are enabled: a warn threshold (default 0.6) and a block threshold (default 0.9). You choose whether flagged uploads are auto-rejected or routed to a review queue.
Result:
Every check returns a per-category score, an overall score (the highest score among enabled categories), whether the content is allowed, and a reason string when it's flagged or blocked — so downstream systems can act without re-implementing the logic.
Detection Flow:
- Vision AI analyzes the image and returns a score per category (adult, racy, violence, spoof, medical)
- The overall score is the max across enabled categories
- If the overall score is at or above the block threshold, the content is blocked
- Otherwise, if it's at or above the warn threshold, it's flagged for review
- If detection itself fails, the content is allowed by default (fail-open) rather than blocking uploads on an AI outage
5. Unified Media Analyzer
One call runs the full AI analysis pipeline across image, video, and audio — no need to orchestrate each provider separately.
What you can request (images): tags, a generated description, NSFW check, object detection, face detection, OCR text extraction, and dominant colors.
What you can request (video): semantic indexing (Twelve Labs), transcription, and auto-generated chapters.
What you can request (audio): transcription.
Each analysis type can be toggled independently, with its own configuration (e.g. NSFW thresholds, transcription language), and every result — tags, description, NSFW status, detected objects/faces, OCR text, transcript, chapters, detected activities — comes back in a single unified response.
Provider Auto-Selection:
The analyzer routes each request to the right configured AI provider automatically, and falls back gracefully if a provider is unavailable.
6. Storage Backend Abstraction
| Backend | Config Fields |
|---|---|
| Amazon S3 | bucket, region, access key ID, secret access key |
| Google GCS | bucket, project ID, credentials |
| Azure Blob | container, connection string or account name + key |
| Cloudflare R2 | account ID, access key ID, secret access key, bucket |
| MinIO | endpoint, port, access key, secret key, bucket |
| Local | base path |
7. Image Processing & Watermark
Image Variants:
Every variant is defined by a name, target dimensions, a fit mode (cover, contain, or fill), quality, and output format — generated in batch on upload. Focal point detection (a normalized x/y coordinate) keeps important content in frame during smart cropping, and WebP/AVIF conversion with perceptual quality tuning keeps file sizes down.
Watermarking:
Watermarks can be text or image-based, placed at any of six positions (four corners, center, or diagonal), with configurable opacity and rotation. Text watermarks support font, size, color, and family; image watermarks support a custom overlay image and width.
Four built-in presets cover the common cases: corner (subtle, bottom-right), center (light overlay), diagonal (classic stamp), and subtle (barely-there bottom-right) — each with sensible default opacity so most teams never need to tune it by hand.
Batch Watermark:
- Apply to entire collections
- Save as new or replace original
- Custom filenames with pattern support
8. Supporting Capabilities
Full AI Video Pipeline: On upload, a video runs through the complete AI pipeline automatically:
- Verify the file is a valid video
- Check which AI providers are available
- Index the video for semantic search (Twelve Labs)
- Run transcription
- Moderate for NSFW content — checked against the indexed video across categories (explicit nudity, sexual activity, graphic violence, self-harm, hate symbols), aggregated into a single confidence score. If the video hasn't been indexed yet, or no video-intelligence provider is configured, moderation is honestly reported as skipped rather than silently marked safe — no fabricated "all clear" result.
- Detect objects (vision AI)
- Generate chapters (AI)
- Detect activities (Twelve Labs)
- Save all results to the asset record
Every AI operation is tracked for usage-based billing and quota management.
Asset Management:
- Upload with focal point detection
- Variant generation (all sizes)
- Watermark application
- Storage management
- Delete with tracking
Upload Validation:
- MIME type whitelist
- Max size per type
- Magic byte verification
- Virus scanning hooks
- Chunked upload progress
Media Analytics:
- View/download tracking
- Popular assets ranking
- Storage usage per folder
- Bandwidth consumption
- Aggregate and per-asset metrics
Video Delivery:
- Thumbnail extraction (FFmpeg)
- Streaming URL generation
- HLS/DASH manifest generation
- Quality variant management
Upload Progress:
- Real-time progress callbacks
- Chunked upload support
- Resume interrupted uploads
- Per-file progress tracking
Safe Deletion:
- Soft delete with grace period
- Background cleanup
- Orphaned file tracking
- Storage quota enforcement
9. AST Studios — AudioWorklet DSP + Broadcast Matrix
Two visual studios in the Media module compile to real artifacts via the Universal AST Engine:
AudioWorklet Synth Studio
- A node graph (Oscillator / ADSR / Filter / Gain / MIDI) compiles to a real
AudioWorkletProcessor— generated code that runs on the browser's dedicated audio thread, per-sample, not on the main thread. - The oscillator does genuine phase-accumulation synthesis:
phase += (freq / sampleRate) * 2πper sample, with sine / saw / square waveforms — not a sample playback trick. - Web MIDI input is wired in, so a controller can drive the generated synth live.
Broadcast Matrix
- A node graph of inputs + operations (overlay, watermark,
scale, LUT, RTMP output) compiles to a real FFmpeg invocation with a-filter_complex "[…]overlay=…;[…]scale=…"graph. - Output is a portable, inspectable FFmpeg command — the visual graph is a _definition_, not a runtime lock-in. Note: the studio generates and previews the filtergraph; live "dispatch to an ingest server" execution is not yet wired (the compiler is the shipped capability).
Aeion Bridge exposes local processing — video/audio transcoding, image resize/convert, OCR, and HTML-to-PDF generation — to the platform over a lightweight local protocol.
Bridge runs these locally, avoiding cloud upload for large files. Status checks are cached briefly so the admin UI stays responsive without polling Bridge constantly.
When Whisper is the selected provider, audio is sent directly to OpenAI, which returns the transcript with word-level timestamps.
1. Submit a video for indexing — returns a video ID and polls until indexing completes
2. Search with a natural language query and confidence threshold — returns matching video segments
3. Results include: video ID, confidence, start/end timestamps, thumbnail
Two global thresholds apply across all enabled categories (adult, racy, violence, spoof, medical):
- Warn threshold: 0.6 by default — flags content for review
- Block threshold: 0.9 by default — blocks the upload outright
The overall score is the highest per-category score among the categories a tenant has enabled; individual categories don't carry their own threshold.
When user uploads an image with focal point selection:
1. Store x/y coordinates (0-1 normalized)
2. On variant generation, crop from focal point
3. Ensures important content (face, product) stays in frame