An AI agent orchestration architecture is the layered design that turns many model calls into one dependable system: ingress normalizes the request, routing decides what runs, execution runs it, aggregation folds candidate outputs into one answer, verification checks that answer, and observability records what happened. Most production systems are one of four shapes built from those layers. This page diagrams all four and states where each one breaks, with measurements where we have them.
Disclosure: we build a managed mixture endpoint (Moamao), and pattern four is the architecture we run in production. Methods, replications, and retractions are published in full. For the wider category context, start with the complete orchestration guide.
The six layers every architecture shares
Before the diagrams, the vocabulary. Whatever shape the graph takes, requests pass through the same six concerns — and each layer has a characteristic way of failing.
| Layer | Job | Typical failure |
|---|---|---|
| Ingress | Auth, quotas, prompt assembly, session context | Instruction clutter — auxiliary instructions anywhere in a prompt measurably degrade knife-edge, sequential answers |
| Routing | Decide which models, agents, or pools run | Silent misroutes only traces can catch |
| Execution | Run the calls, parallel or sequential | Latency stacking; partial failures mid-graph |
| Aggregation | Fold N candidates into one answer | Selection bottleneck when a model judges candidates |
| Verification | Check the answer deterministically before it ships | Skipped, or replaced by another model’s opinion |
| Observability | Per-request traces: route, candidates, merge decision, timings | Token logs with no per-task view |
Aggregation is where architectures genuinely diverge, so the four patterns below are named by how — or whether — they fold multiple candidates.
Pattern 1: Pipeline
One lane, fixed stages, each stage trusting the previous one. Pipelines are the easiest architecture to debug — every request has exactly one trace — and latency is simply the sum of the stages. The structural weakness is error compounding: a wrong plan poisons everything downstream, and no later stage re-opens an earlier decision. A second, less obvious weakness sits at ingress: because every stage prepends its own instructions, pipelines accumulate auxiliary instructions — and we measured that instruction presence anywhere in a prompt degrades knife-edge, sequential answers. Keep stage prompts minimal, and put a deterministic verify stage before egress rather than a third model opinion.
Pattern 2: Router
The router inspects each request and dispatches it down exactly one path — a code specialist, a reasoning specialist, a cheap fallback. Disagreement never arises because only one candidate is ever produced, which makes routers the standard answer for cost control over heterogeneous traffic. The catch: the router is itself a model decision, and a misroute produces a confidently wrong answer with nothing to compare it against. Routers therefore lean harder on the observability layer than any other pattern — the route taken must be in every trace, or bad routing is undetectable until users complain.
Pattern 3: Ensemble with a fold
Run several candidates in parallel — different models, or repeated samples of one — then fold them into a single answer. This is the shape behind the mixture-of-agents paper and its layered variants, and Self-MoA showed that repeatedly sampling one strong model can beat mixing weaker ones. (Primer: what is mixture-of-agents?)
The fold is the load-bearing box, and it is where this pattern fails. In our published season, six mixture architectures with the correct answer provably present in the candidate pool all scored at or below their own best member when a small model did the judging — a result we call the selection bottleneck, documented with hash-locked benchmark files and pre-registered predictions in Never Let a Small Model Choose. Generating good candidates is not the hard part; selecting among them is.
Pattern 4: Dual pool with a mechanical merge
The selection bottleneck suggests a design rule: never let a model compare answers. The dual-pool pattern applies it. Pool A produces direct answer candidates. Pool B produces program-of-thought candidates in the lineage of PAL— the model writes code, and deterministic execution produces the value. The merge is then not a judgment but a single bit: did deterministic code substitute a value? If yes, that result wins; if no, Pool A’s answer ships. No model ever reads two candidates side by side, so the aggregation and verification layers collapse into one deterministic step.
This is the architecture behind Moamao: one API call runs the dual-pool mixture server-side, on hardware we own — a single consumer 8GB GPU with a qwen3:8b base. Measured on our frozen benchmark suite, it scores 10/10 on the frozen ten-task tier-1 reasoning, context, and memory suite, replicated eight consecutive times; claude-opus-5 scores 10/10 on the same suite under the same harness. Composite across our frozen suites: 14.75/17 — 87%, where Opus scores 17/17 — at roughly ten seconds per task. In the same season, a single well-orchestrated 8B matched a two-model panel at 4.6× the speed. All of that is scoped to our frozen suites, and the season included three same-day retractions of results that did not survive scrutiny.
Choosing between the four patterns
| Pattern | Disagreement resolved by | Dominant failure | Use when |
|---|---|---|---|
| Pipeline | Never arises — one lane | Errors compound across stages | Fixed multi-step transforms |
| Router | Avoided — one path per request | Silent misroutes | Heterogeneous traffic, cost control |
| Ensemble + fold | A model judges, votes, or synthesizes | Selection bottleneck: fold at or below best member | Single-turn quality, with a trusted fold |
| Dual pool + mechanical merge | One deterministic bit; no model compares | Both pools must produce usable candidates | Single-turn tasks where execution can verify |
These compose. A router can front an ensemble; a pipeline stage can be a dual pool. The invariant worth keeping as you compose them: every place two candidate answers meet, know exactly what resolves the tie — and prefer that thing to be deterministic. If you are evaluating vendors rather than building, the platform comparison lists the questions to ask; if you want the dual-pool architecture as a service, plans start at $0.
FAQ
What are the layers of an AI agent orchestration architecture?
Six cover nearly every design: ingress (normalize the request), routing (decide what runs), execution (run the calls), aggregation (fold candidates into one answer), verification (check before shipping), and observability (per-request traces).
What is the difference between a router and an ensemble?
A router chooses one path per request, so no disagreement ever exists — but a misroute goes unnoticed without traces. An ensemble runs several candidates and folds them, so its quality is exactly the quality of its fold step.
Why use a mechanical merge instead of an LLM judge?
Because judged folds measurably underperform: in our published season, six architectures with the correct answer provably in the pool scored at or below their own best member when a small model judged. One deterministic bit — did code substitute a value — removes the comparison entirely.
Related: the complete orchestration guide · orchestration platforms compared · what is mixture-of-agents? · one model vs a panel