What is LLM-as-a-judge?

LLM-as-a-judge is the practice of using a language model to evaluate the output of a language model. You give the judge model the thing being assessed, a set of criteria, and a scale. It returns a verdict and, usually, its reasoning.

The technique exists because the other options don't work. A summary, a translation or a support reply has no single correct string, so exact-match scoring tells you nothing. Human review answers the question properly but costs hours per hundred samples, which is too slow to sit inside a test suite.

The first hard numbers came from one paper. Lianmin Zheng and co-authors published Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena on arXiv in June 2023. They measured judge verdicts against human preference data. Strong judge models reached "over 80% agreement" with those preferences, the same level two humans reach with each other.

That comparison is what made the idea spread.

Three setups cover most real-world use.

Pairwise comparison. Show the judge two answers to the same input and ask which is better. This is the most reliable of the three, because relative judgements are easier than absolute ones.

Single-answer grading. Give the judge one answer and a rubric, and ask for a score. Cheapest to run and easiest to wire into a pipeline, but scores drift between runs unless the rubric is specific.

Reference-based grading. Supply a known-good answer and ask how far the candidate deviates. Useful when a correct answer exists but exact wording does not matter.

In all three, the quality of the rubric does more work than the choice of model. "Is this good?" produces noise. "Does this answer name a specific next action? And does every number in it appear in the source text?" produces something you can act on.

The technique is at its best where the volume is high and the cost of a single wrong verdict is low.

The MT-Bench paper names four failure modes, and they are all still live: position bias, verbosity bias, self-enhancement bias, and limited reasoning ability. In practice that means a judge will favour whichever answer it saw first, reward length over substance, prefer text that resembles its own output, and grade confidently on questions it cannot actually reason through. The survey literature still treats judge reliability as an open problem rather than a solved one. Jiawei Gu and co-authors set out the state of it in A Survey on LLM-as-a-Judge, published on arXiv in November 2024.

Mitigations exist and are cheap. Randomise the order in pairwise runs, or run both orders and discard disagreements. Never let a model grade its own output when you can afford a different one. Put a length ceiling in the rubric. Calibrate against a hundred human-labelled examples before trusting a score, and re-calibrate when you change the judge model.

There is one limitation worth spelling out, because we walked into it ourselves.

Every article in this knowledge base passes two LLM judges before it goes live. One checks each claim against its cited source. The other answers a set of editorial questions and returns pass or block.

In August 2026 a draft went through both carrying a number nobody had verified. The text said the AGENTS.md list had more than 30 tools. The list named 23. Both judges passed it. Our review caught the error before the article published, but only by luck: someone opened the source page to add a link, and adding the link meant counting the list.

The judges were not malfunctioning. They were asked whether the draft was well-sourced and internally consistent, and by any reading it was. Every claim carried a citation. Both language versions agreed with each other. Neither judge fetched the cited page, because neither had been asked to, and agreement between two versions of the same mistake looks exactly like corroboration.

So the rule we now apply is narrow. A judge grades the text in front of it, not the world that text describes. A number that was wrong going in comes out wrong. Every gate it clears just makes it look better sourced. Verifying a fact means opening the source. That step cannot go to a model handed nothing but prose.

The same rule applies to any evaluation setup you build. Ask what the judge can physically see. Consistency checks, tone checks and rubric compliance are inside that boundary. Ground truth is outside it.

Related articles: