Journal article
Automatic Benchmarks Measure an Asymmetry: Privileged Information, Panel-Relative Difficulty, and Two Self-Biases That Pull in Opposite Directions
Automatic benchmark construction is presented as a response to saturation: a language model proposes topics, writes items, and supplies answer keys, and the resulting datasets are reported to be harder and more novel than human-built ones. The standard objection is that a model cannot examine…