An LLM ensemble benchmark measures one thing vendor pages rarely state plainly: whether a mixture of models beats the best single member of its own pool, on a frozen task suite that cannot quietly move underneath the result. Not whether the ensemble’s answers read better — whether they score better, under a harness that stays identical from run to run. This is the account of a season spent taking that measurement seriously: six architectures dead, three claims retracted the same day they failed, one scorer caught drifting, and an ending in which no model judges anything at all.
The setting, for context: Moamao runs a mixture architecture server-side, behind one API call, on hardware we own — a single consumer 8GB GPU with a qwen3:8b base (product details here). The season’s question was never “can a small ensemble look impressive once.” It was whether any of the mixture designs we could build would hold up on a suite it could not overfit, rerun until the answer stopped being interesting.
The rules, and the day the measuring stick moved
The rules were set early: frozen task suites, pre-registered predictions written down before runs, and a standing commitment that a claim which failed replication would be retracted publicly rather than quietly deleted.
The rules got harder mid-season, for an embarrassing reason: the scorer drifted. Between two runs that should have been identical, the harness was no longer grading quite the same thing it had graded before. The models had not changed. The measuring stick had. Every downstream number was suspect until the runs were redone, and the fix was mechanical rather than aspirational: the benchmark files are now hash-locked, so a suite that changes by one byte announces itself. Catching your own instrument drifting is unpleasant. It is also the single best argument for freezing suites that we can offer — if it happened to a team actively looking for it, it is happening to teams that are not.
The 10/10 that collapsed on replication day one
Early in the season, a configuration scored 10/10 on the frozen ten-task tier-1 reasoning/context/memory suite. The celebration lasted until replication day one, when the same configuration collapsed on the same suite. Nothing about the harness had moved — the first result was simply not stable. That collapse set the season’s standard: a score does not exist until it replicates, and one replication is a coincidence.
The architecture that eventually survived scored 10/10 on that same frozen tier-1 suite, replicated eight consecutive times. Under the same harness, on the same suite, claude-opus-5 also scores 10/10. Across the composite of frozen suites the mixture reaches 14.75/17 — 87% — where Opus scores 17/17, at roughly ten seconds per task. All of these numbers are scoped to our frozen benchmark suite, and deliberately so; the season’s retractions, covered below, were exactly the claims that had been stated bigger than their measurement.
| Measurement | Result | Scope |
|---|---|---|
| Tier-1 suite (ten tasks, frozen) | 10/10, replicated 8 consecutive times | Our harness, hash-locked files |
| claude-opus-5, same suite | 10/10 | Same harness, same suite |
| Composite across frozen suites | 14.75/17 (87%); Opus 17/17 | Our frozen benchmark suites |
| Latency | ~10s per task | Single consumer 8GB GPU, qwen3:8b base |
| Single well-orchestrated 8B vs 2-model panel | Matched it at 4.6x the speed | Our frozen benchmark suite |
| Judge-configured mixtures (six architectures) | At or below their own best member | Correct answer provably in the pool |
Six judge-shaped failures
The middle of the season was a graveyard, and every headstone had the same epitaph. Six different mixture architectures — different pool compositions, different aggregation prompts, different orchestration shapes — shared one property: a small model was asked, at some point, to compare candidate answers and choose. In every one of the six, the correct answer was provably present in the candidate pool. In every one, the final score came out at or below the pool’s own best member.
That is a specific and uncomfortable finding: generation was not the bottleneck. Selection was. A pool that contains the right answer is worthless if the component doing the choosing cannot recognize it, and a small judge reliably could not. This is consistent with the direction of the broader literature — the original mixture-of-agents work established that aggregation can help, and Self-MoA later showed that mixing in more models is not automatically better than sampling your best one — but the sharpness surprised us anyway. The full writeup, with the architectures and the numbers, is published as Never Let a Small Model Choose.
The finding nobody was looking for
Failure analysis on the judge experiments produced the season’s strangest result. Answers to knife-edge and sequential tasks — the ones where a single substitution or ordering error flips the result — degraded when auxiliary instructions were present anywhere in the prompt. Not adversarial instructions. Ordinary ones: formatting requests, role framing, meta-guidance. Their mere presence, in any position, was enough to poison exactly the answers that had no tolerance for drift. We call it instruction-presence poisoning, and once seen it is hard to unsee: much of standard prompt engineering amounts to adding instructions, and on this class of task, adding was the problem.
The ending: one bit, no judge
The surviving architecture follows from the two findings above. If a small judge cannot pick the right answer out of a pool, stop asking any model to compare answers. If instruction presence poisons knife-edge work, stop decorating prompts. The design that replicated runs two pools and merges them on a single bit of deterministic evidence — did code actually substitute a value — with no model ever ranking, scoring, or choosing between candidate answers. The one comparison in the system is performed by code, and code does not have opinions. Background on how this differs from classic judge-topped mixtures is in What is mixture-of-agents?, and the honest map of when any panel beats a single model is in One model vs. a panel.
Why we publish retractions
Three claims from this season were retracted, publicly, the same day they failed. Each shared a property: the claim was bigger than the measurement underneath it. The correction was not to defend them harder; it was to withdraw them and keep only what a frozen, hash-locked suite would certify on demand.
The uncynical reason to publish retractions is that it is what the evidence requires. The strategic reason is nearly as good: in a market where every vendor claims benchmark wins, a retraction log is the one artifact that cannot be faked cheaply, because it costs something real to produce. It is also the fastest due-diligence test available to a buyer. Ask any vendor whose benchmark claims you are evaluating one question: what did you retract when it did not replicate? A team that measures honestly will have an answer, because honest measurement produces failures. Silence is an answer too. That test, and the rest of the evaluation questions we would ask of anyone in the category — including ourselves — are laid out in the orchestration platforms comparison.
FAQ
What is an LLM ensemble benchmark? A test of whether a mixture of models outperforms the best single member of its own pool on a frozen task suite. Comparing the ensemble against its own best member — not an unrelated baseline — is what keeps the measurement honest.
Why did the judge-based mixtures fail?In six architectures where the correct answer was provably present in the candidate pool, a small model acting as judge selected at or below the pool’s own best member. Selection, not generation, was the bottleneck.
Why publish retractions at all? A published retraction is the cheapest verifiable evidence that the surviving numbers were measured rather than curated. Three claims were retracted same-day; what remains replicated eight consecutive times.
Related: Never Let a Small Model Choose, What is mixture-of-agents?, AI orchestration, explained, and Moamao pricing.