Multi-agent consensus: how to verify the result

Three agents choosing the same answer gives you a candidate to check. It does not establish that the answer meets your requirements. For generated code, ask for an executable counterexample; for a factual claim, open the source; for a proposed URL, check the built page and its HTTP response. The useful question is: what observation could reject this answer even if every agent agrees?

A real failure: the best page was missing from the choices

In September 2026, the query router for this site was offering three URLs that returned 404. It assembled its catalogue from translation dictionary names. The dictionary called consultant-program described a real service, but that was not the service's public address. The Polish training page also used a translated slug.

A model could correctly recognise the topic of the service and still select an unusable URL. Asking three reviewers to choose from that same catalogue would leave the correct address unavailable to all three. This incident was a catalogue bug, not a controlled experiment on agent voting. It gives us a concrete shared assumption to test: does every candidate actually exist?

The correction was recorded in pull request #59. The follow-up check compares every routing candidate with the build's sitemap and verifies that each sitemap entry has an HTML file. A catalogue containing any of those three old addresses is rejected before the model is called.

What independence would buy you

Consider a simplified calculation, not a measurement of any particular model. Each of three voters has a 20% chance of being wrong on a binary question. If their errors are independent, majority voting fails with probability 3 × 0.2² × 0.8 + 0.2³ = 10.4%. If they always fail on exactly the same questions, the majority is still wrong 20% of the time.

You cannot choose between those cases by counting agent processes. Separate runs can share missing context, misleading instructions or the same faulty catalogue. Different models can also receive that catalogue. Independence has to be supported by observed errors on representative tasks; it does not follow from having three names in a review panel.

This is also the limit of model self-evaluation: another judgment may help locate a mistake, but a confident judgment does not supply the evidence it was missing.

Debate can help without becoming proof

Du and colleagues report improvements in reasoning and factuality from multiagent debate in their 2023 study. That is evidence that debate can improve answers on the evaluated tasks. It is not a guarantee for every answer a production workflow accepts.

Give the reviewer a deliverable beyond agreement: a failing input, a source passage that contradicts the claim, or an observed response from the proposed endpoint. For the routing bug, “the title sounds relevant” adds no new evidence. “This candidate is absent from the sitemap” does. When an LLM judge ranks alternatives, also check for position bias; presentation order is another input to the judgment.

An acceptance check you can run

From this repository's root, after npm run build, the router's check can be run without an API key:

node --input-type=module <<'JS'
import { inventory } from './tools/typesafe/corpus.mjs';
import { assertIndexable, indexablePaths } from './tools/typesafe/routing-data.mjs';
const pages = [...inventory('en'), ...inventory('pl')];
assertIndexable(pages, indexablePaths());
console.log(`${pages.length} candidate URLs checked`);
JS

This check has a narrow contract: all offered URLs belong to the built indexable set. It does not prove that a page answers a query, and it does not test the deployed host's redirects or availability. Those need a content review and a separate HTTP check. Likewise, compilation establishes that code compiles; a passing test establishes only the behaviour its assertions exercise.

Repair the evidence as well as the code

Changing the catalogue does not repair a saved routing result. A nonexistent candidate can affect the runner-up and confidence even when the winning URL exists. Renaming three URLs in old JSON would hide the defect without recalculating that choice.

1. Keep the original input. Use the same Search Console export to rerun the comparison.
2. Record what produced the result. Save the catalogue fingerprint, input hash, generation time and responding model versions.
3. Reject incompatible evidence. The page audit must refuse an old report when its catalogue no longer matches.
4. Recompute downstream findings. Rerun the page audit with the new routing before deciding which page to rewrite.

For an agent review, require the same separation: a proposed answer, the observation that checks it, and the boundary of what that check establishes. More reviewers are useful when they bring additional evidence or find additional counterexamples. A unanimous vote alone leaves the acceptance condition untested.