Production Diaries #1 — Can three free models beat one?

If three free models agree, is that an answer, or just three guesses in the same coat?

I ran a small test. The answer is: not settled, with a few hints worth following.

“Measured here” below means observed in my own run on 19 October 2026, with these specific models. It may change.

🪙 What I tested

A “council” here is simple: several models answer independently, then one model reads those answers and writes the final one. No debate, no personas.

The pilot used 12 synthetic tasks, written by an AI, in 3 families of 4: cost calculations, incident-evidence triage and release-readiness checks. 3 of the 12 were underdetermined: the evidence lacked the deciding fact, so the right answer was “insufficient evidence”. I ran 1 repeat, which gave 48 task-runs and 120 final calls.

Four arms:

• S-A: model A answers once.
• H-A: A answers three times, then A synthesises.
• M-ABC: A, B and C answer once each, then A synthesises.
• S-A+: one A call with a larger thinking and output allowance.

Models: A is gemini-3.5-flash-lite; B is nvidia/nemotron-3-super-120b-a12b:free; C is poolside/laguna-s-2.1:free. All free tiers.

What changed from my plan

Disclosure, plainly:

• Model A changed. I planned gemini-2.5-flash, but its free tier rejected requests at a limit of 20 on that key (measured here), which did not fit the planned calls. I switched to gemini-3.5-flash-lite before the holdout, under a preflight rule frozen before the run. By its name, Flash-Lite is a smaller tier than I planned (my inference, not benchmarked), so A may be a weaker reference than intended.
• B and C did not change. S-B and S-C (each member alone) were not run.
• The run stopped and resumed 5 times on upstream rate limits for model C, with the same models throughout.
• One scoring correction came after I saw results (below).

The result

Accepted answers out of 12, strict deterministic scoring (corrected scorer v1.1, applied to all arms; the uncorrected v1.0 gave 9/11/11/9 for S-A/H-A/M-ABC/S-A+):

No timeouts. Latency is the critical path: slowest candidate plus synthesis, as if the calls had run in parallel.

Versus the mixed council, the paired differences came with exploratory 95% bootstrap intervals over tasks. M-ABC beat S-A on 2 tasks (+0.17, interval 0.00 to +0.42) and S-A+ on 1 (+0.08, 0.00 to +0.25). H-A beat M-ABC on 1 (-0.08, -0.25 to 0.00). Every interval touches zero. With n = 12, I claim no significance.

Where it helped, and where it did not

The gains were mostly in cost calculations (H-A 4 of 4 vs S-A 2 of 4; M-ABC 3, S-A+ 2), plus one incident task. One cost pass, 54.82 against a key of 54.86, was a tolerance-edge pass. Release checks were 4 of 4 for every arm. All three underdetermined tasks were correctly abstained on by every arm, and no arm abstained on a task that had an answer.

Two traces, one each way:

• hold-cost-03: B had the right figure, 12.68 USD. A said 15.65, and C gave no usable decision. M-ABC’s synthesis returned 16.15 and failed. Synthesis erased the correct minority.
• hold-inc-04: B and C both said dependency outage. A alone abstained, and M-ABC’s final kept that lone abstention, correctly.

So the council can lose the best objection, and it can keep it.

Seedhi baat: three different models did not beat three calls to one.

H-A (12) did at least as well as M-ABC (11). Same-model repeats plus synthesis matched the mixed council here, and finished in about half the median wait (10.3 s vs 20.9 s). M-ABC’s p95 wait was 163.6 s, which I read from the logs as free-tier reasoning-model waits and rate-limit backoffs (my inference).

How this fits the research

Early debate work reported gains on its authors’ tested tasks. Later work argued for strong single-agent baselines and reported conditional effectiveness. Anthropic has described the tradeoffs of its own multi-agent system, and its evals guidance shaped my deterministic scoring (a design reference only). My pilot is too small to confirm or challenge any of them.

The scoring correction

Scorer v1.0 gave S-A 9, H-A 11, M-ABC 11 and S-A+ 9. In an audit blind to arm names, I found it had failed two correct abstentions on hold-rel-04 because they mentioned a stale report while explaining it did not apply. I corrected the rule, applied it to every arm, and kept the v1.0 results. That correction came after seeing results, and the author graded alone.

Cost

Reported cost was 0 for every OpenRouter response, and Gemini ran on the free tier. The API cannot verify billing; I rely on the AI Studio free-tier status reported to me. So cost per accepted answer is uninformative here. It is not a price estimate. With free tiers, the scarce resources are quota and waiting.

My decision rule, applied

My decision rule has three tests: a minimum gain, a ceiling on extra cost per extra pass, and a wait limit. These are chosen policy, not constants, and I did not fix them numerically for this pilot. So the verdict is a judgement: inconclusive, extend the evaluation. Next time I will pre-register the thresholds, and add S-B, S-C and more tasks. Dollars cannot test the cost test on free tiers.

What this issue does not claim

• It does not claim a council beats one model, or loses to one.
• It does not generalise beyond 12 synthetic AI-authored tasks, one repeat, one grader and these free-tier models.
• Using a smaller A than planned may favour or hurt any arm; I did not test which.
• It does not show a mixed council beats repeated calls to one model.
• It makes no claim about production use, other models or free-tier limits staying the same.

The useful unit is not the number of models in the room. It is the cost of an answer you can accept.

Follow the series at https://garganuj.com/token-diaries/ or start with Token Diaries #1: https://garganuj.com/token-diaries-ep-1-one-token/

Would a result like this change what you run in production?

References

Accessed 19 October 2026.

1. Anuj Garg, Token Diaries #1. https://garganuj.com/token-diaries-ep-1-one-token/
2. Du et al., “Improving Factuality and Reasoning in Language Models through Multiagent Debate”, 2023 (abstract reviewed). https://arxiv.org/abs/2305.14325
3. Yang et al., “Revisiting Multi-Agent Debate as Test-Time Scaling”, 2025 (abstract and version record reviewed; preprint). https://arxiv.org/abs/2505.22960
4. Anthropic, “How we built our multi-agent research system”, 13 Jun 2025 (engineering report reviewed; vendor account of its own system). https://www.anthropic.com/engineering/multi-agent-research-system
5. Anthropic, “Demystifying evals for AI agents”, 9 Jan 2026 (evaluation guidance reviewed; design reference only). https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents