Aeion AI + Ollama — On-Prem, Air-Gapped LLM
Configure Ollama as a provider in Aeion AI to run open-weights models entirely on YOUR hardware. Supports Llama 3.1 / DeepSeek / Mistral / Qwen / Phi / Gemma / any GGUF-format model. OpenAI-compatible endpoint = drop-in for any code path. The right path for HIPAA / FedRAMP / classified / regulated deployments where data egress is prohibited.
When Ollama Wins
Self-hosted LLM via Ollama is the right choice when:
Setup — Ollama Side
Ollama runs on Linux / macOS / Windows. For production deployments, Linux server with NVIDIA GPU (A100, H100, RTX 4090, etc.) or AMD MI300.
Setup — Aeion Side
- In Aeion, go to
/admin/ai/providersand create a new AI Provider record: set Provider to Ollama, Category to Text.
Model Selection — What to Run
Open-weights model landscape (2026-05).
Hardware — What You Actually Need
Cost-of-deployment math for typical setups.
Common Gotchas
- Aeion uses Ollama's native API, not the OpenAI-compatible surface — the adapter calls
/api/chatand/api/embeddirectly. Tool-calling reliability still varies by model; smaller models are less consistent at producing well-formed tool calls.
FAQ
Yes — air-gapped deployment with Ollama as the only AI provider is included free with any module, no separate tier required. No outbound network calls, no phone-home telemetry, embedded license. For dedicated on-prem hardening support, talk to us about a provisioning session to package the deployment manifest.
Not directly through the Ollama provider — Aeion's Ollama adapter calls Ollama's native `/api/chat` and `/api/embed` routes, not the generic OpenAI-compatible `/v1` surface those runtimes expose. If you're running vLLM or TGI, front it with an Ollama-API-compatible shim, or register it as a custom OpenAI-compatible provider instead.
Fine-tune in your training framework (Unsloth, axolotl, etc.), export to GGUF, drop into Ollama's models directory, and call it with whatever tag you gave it in Ollama. If you're running multiple text providers, double-check your model naming doesn't collide with another provider's models.
Yes — configure a fallback chain across providers. Default to Ollama for sensitive data, fall back to a cloud provider like OpenAI for hard reasoning tasks it can't handle as well.
Llama 405B-class open models benchmark within striking distance of mid-tier cloud models on general tasks. DeepSeek R1 is competitive on hard reasoning + math. For frontier-only workloads (very hard reasoning, advanced agent loops), cloud frontier models still win — fall back accordingly.
Run the math: GPU hardware monthly amortization + power + ops salary + idle capacity vs. cloud per-token rates × your monthly volume. Crossover is usually ~50-100B tokens/month for general workloads; lower for reasoning workloads where cloud is expensive.