Which Way the Verdict Falls: Noise-Induced Flips in an LLM Bias Judge Run Toward the Label Its Prompt Makes Easiest to Reach, the Harm-Detection Literature Records the Opposite Sign, and Whether Noise Inflates a Bias Score Depends on the Score

@misc{mahendrakar2026which,
  author       = {Mahendrakar, Pranay},
  title        = {{Which Way the Verdict Falls: Noise-Induced Flips in an LLM Bias Judge Run Toward the Label Its Prompt Makes Easiest to Reach, the Harm-Detection Literature Records the Opposite Sign, and Whether Noise Inflates a Bias Score Depends on the Score}},
  year         = {2026},
  month        = oct,
  date         = {2026-10-19},
  publisher    = {Life of Research},
  howpublished = {Preprint},
  doi          = {10.5281/zenodo.23461021},
  url          = {https://lifeofresearch.com/preprints/116},
  abstract     = {(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Life of Research Language models are now used as judges that read a passage and decide whether it leans toward a stereotype, and the passages they read are often noisy. A 2026 study corrupted 3,822 stereotype-probing responses with typos, informal spelling and collapsed punctuation and found that, pooled across its four judges and in three of them individually, noise turned neutral verdicts into directional ones far more often than the reverse, by up to 120 to 1 in the most fragile judge; in the fourth, GPT-5.4, the two directions were indistinguishable. It concluded that bias measured on noisy text is systematically overestimated. Work on toxicity and safety classifiers records the opposite sign: misspellings and restyling make detectors miss harmful text. In this analysis the author reviews and synthesises published results to show that the direction of a noise-induced flip is not a property of noise or of bias, but of which label the judge's instructions make hardest to earn, where the judge falls when evidence thins, and which tokens the corruption strikes. In the anchor, the neutral label demands affirmative evidence while a directional one may be implicit, and 258 of the fragile judge's 288 fabrications land on the second-listed option it already favoured; elsewhere, one meaning-preserving restyle pushes three safety judges toward "safe" and a fourth toward "unsafe". The paper derives that a BBQ-style signed bias score moves under fabrication only by the fabrications' net stereotype alignment, which the anchor did not measure, shows that the closed judge's parity is underpowered and lacks a rerun floor, and states the content-free, rerun, prompt-ablation and re-annotation controls that would settle the direction for any judge.},
  keywords     = {LLM-as-a-judge, social bias measurement, noisy user-generated text, robustness, label bias, calibration, prevalence estimation, evaluation methodology},
  language     = {en},
  copyright    = {Creative Commons Attribution 4.0}
}