2026-10-16T16:14:20Z https://lifeofresearch.com/oai2

oai:lifeofresearch.com:90 2026-10-07T10:21:50Z type:article openaire

10.5281/zenodo.23199460 https://lifeofresearch.com/articles/90 Pranay Mahendrakar 0009-0003-7224-029X SONYTECH Life of Research 2026 LLM-as-a-judge evaluation validity inter-rater agreement construct validity criterion validity reward model evaluation multitrait-multimethod measurement theory Journal article Creative Commons Attribution 4.0 (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. An automatic judge is usually accepted on one piece of evidence: it agrees with human preference labels about as often as humans agree with each other. That number is then read as permission to use the judge for ranking models, filtering data, selecting samples and supplying reward. This paper separates three claims the ritual runs together - that the judge can stand in for the labelers on a given item population, that it tracks the quality construct the labels were meant to measure, and that decisions made with it turn out well in a stated use - and argues that agreement evidence bears directly only on the first. A stylised two-loading model makes the reason exact: agreement with a single reference identifies one linear combination of a judge's sensitivity to the construct and to the surface features the labelers also respond to, so a judge that shares the labelers' shortcuts can match a judge that tracks quality. The published record is then read through that separation. Human labels carry documented surface sensitivities that judges reproduce; judges carry failures of their own that natural validation sets never exercise; and agreement is concentrated on the easy, wide-gap items where decisions matter least. In the corners where an external criterion exists - verified correctness, downstream policy outcomes, randomised business outcomes and verifiable task success - the criterion has repeatedly exposed failures that agreement did not show, and where it confirms an offline score the confirmation does not transfer across uses: a benchmark score that predicts best-of-N selection well gives only a rough signal for PPO. Validity, on this evidence, is a property of a judge in a use. The paper consolidates these published results, states a use-indexed validation procedure as an algorithm, and proposes the tests that would settle what remains open.