LLM-as-judge: the first answer wins on order, not content

In the note on multi-agent consensus I made a promise: instead of a mutual-admiration circle, you need an adversarial panel. Today I close that loop and show how to build one so it can't fool itself. Because your LLM judge has a flaw nobody puts on a slide. It scores order, not content. And that quietly breaks everything you think your metrics mean.

The experiment: same content, different winner

I ran the simplest test I could think of. I give the model two answers to compare, a first one and a second one, and ask straight out which is better. I record the verdict. Then I do one thing. I swap them. Same content, same words, same scoring prompt, just the order reversed. Logic says the verdict should be identical. Nothing about the substance changed.

And suddenly a different one wins.

I measured it on a larger sample so it wasn't a fluke. The answer shown first gets, on average, 10–15 points out of 100 more. No matter what it actually says. Content doesn't decide the result. The slot it landed in does.

And here's the trap. Most evaluation pipelines run the judge once, in a fixed order. You feed in the candidates in the same sequence they arrived, read off the score, and move on. Nobody checks whether the ranking would survive a plain swap of positions. It doesn't.

"It's like a juror at a cooking contest who always praises the first dish. By the fifth tasting their palate is gone and they're scoring on autopilot."

This is called position bias

The effect has a name: position bias. A model acting as judge (LLM-as-judge) systematically favors position over content. The longer the context and the more candidates you stack up, the worse it gets. The model's attention drifts along the way, and the first entry sticks in memory most strongly and sets the tone for the whole evaluation.

Position bias rarely travels alone. It has two cousins that quietly corrupt the score:

• Verbosity bias. The model prefers longer answers, even when they're padded. It reads more text as more value, when it's often just volume.
• Self-preference. The model prefers its own writing style. An answer that sounds the way it would have written it gets a head start.

Each of these three errors is small on its own. Together they can flip a ranking completely. Meanwhile you stare at the numbers in your dashboard, convinced you're measuring quality. You're measuring a setup artifact. The worst part is that the artifact is stable. Repeat the evaluation and the same wrong winner comes out every time, so you take it as ground truth.

An adversarial panel with position rotation

The fix I tested isn't clever, it's just effective. The whole trick is to take away the judge's chance to cheat before it ever issues a verdict. Three steps:

Rotation is the heart of the whole thing. Only when the same answer sits first in one run and second in the next does the slot bonus cancel out and leave pure content on the table. A panel without rotation is still a mutual-admiration circle, just with more participants.

Yes, this doubles the number of runs. Every candidate goes through the panel twice, once in slot A and once in slot B, and you average the scores only afterward. That's the price of a real measurement. If the evaluation is supposed to mean anything, you can't afford not to pay it.

219 variants, 2 winners

Recently I ran 219 variants of a single solution through a panel like this. At the end, two were left that genuinely won on content. The rest only looked "better." They won on their place in the queue, not on the quality of the answer. Sit with that ratio for a second. Out of two hundred and nineteen candidates, more than two hundred looked better only because they lined up well.

This wasn't bad luck in my sample. The research Adaline cites in its 2026 write-up shows exactly the same thing: position bias is the rule, not the exception. If you evaluate agents without rotation, you're measuring order by default. It's one of those agentic engineering details that quietly decide whether your system measures quality or its own illusion.