Singularity Architecture & Specs
An industrial-grade ETL backend featuring batch processing, dynamic Transformer Registries, and Multi-Collection Foreign Key resolution.
System & Ingestion Metrics
Import Speed
10k+ records/sec via batch inserts
Dry-Run Validation
Sub-second schema validation checks
Data Transformation
O(1) performance per field mapper
Connector Support
102 native API integration templates
1. The Resilient Import Engine
Migrating gigabytes of data requires an architecture built to survive network interrupts, dirty data, and memory constraints. - Batch Streaming: Singularity never loads massive datasets entirely into memory. Files are streamed and chunked into optimal batches (100 records by default) as they're processed, so a multi-gigabyte export doesn't blow your import job's memory budget. - Graceful Error Handling: By default, the engine keeps going even if a specific row fails. Rows that don't pass validation are logged with the exact row number and failure reason, and exported as a report so an admin can fix and re-run them later.
2. Dynamic Field Mapping & Transformation
Legacy databases are messy. Singularity cleans data on the fly.
- The Transformer Pipeline: Developers and admins can map incoming fields through a sequence of built-in cleaning and formatting operations. Convert "yes/no" text columns to strict booleans, parse complex European date formats (DD/MM/YYYY) into a standard format, or concatenate "First Name" and "Last Name" columns into a single string.
- Multi-Collection Resolution: A single CSV row from a legacy system often contains data that belongs in more than one place in Aeion. The field mapper can fan a single import job out across multiple destination collections at once — for example, creating a company record first, capturing its newly assigned ID, and using it to automatically link the related contact record, resolving relationships as it goes.
Frequently Asked Questions
A single monolithic insert has to hold the entire dataset in memory and commit as one all-or-nothing transaction — one bad row past validation could roll back the whole migration. Batching keeps the memory footprint flat regardless of file size (a multi-gigabyte export streams through the same 100-record window), lets per-row failures be logged and skipped without killing the run, and gives the progress bar a real, continuously advancing checkpoint to report against.
It runs a sample of your data through the entire pipeline — parsing and format detection, every field transformer you've attached, and the same strict Zod schema validation the target collection enforces on real writes — with zero side effects. The output is a per-column report of what would pass and what would fail, so you fix the mapping against the real destination rules rather than discovering a type mismatch halfway through a live import.
You nominate a matching field (an external SKU, an email address, a legacy ID). On each run the engine looks that value up in the destination collection first; a match becomes an update to the existing record, a miss becomes an insert. This is a per-run lookup, not a delta/watermark tracker — Singularity is an on-demand engine, so idempotency comes from the upsert match, which is what makes re-running a saved job safe to repeat.
Yes. The field mapper resolves multi-collection targets in dependency order within the same job — it writes the parent (e.g., a company), captures its assigned ID, and links the child (e.g., a contact) to it as it streams, establishing the foreign key at write time. Parent and children commit together in the batch, and either side failing validation shows up in the dry-run report first.
By default the engine keeps going. A failing row is logged with its exact row number and the specific reason it failed, and collected into an errors report you can export, correct, and re-run — the rest of the import isn't held hostage to a handful of dirty records.