LLM-as-a-judge: open-access research

Journal article

INT4 Is Not One Treatment: Why the Quantisation-Safety Literature Cannot Be Reconciled as Reported, How Six Design Axes Separate Its Studies Before Bit-Width Does, and Why Two 2026 Studies of the Same Model Names Reach Opposite Verdicts

Pranay Mahendrakar 2026

Post-training quantisation to 4 bits is widely treated as nearly free, a judgement resting on perplexity and multiple-choice accuracy. Whether it is also free for safety is now contested: 2025-2026 studies report that 4-bit checkpoints lose refusal while their quality holds, that 3-bit weights…

Cite Related research LOR-00102 · 6 views · 5 downloads

PDF406 KB

Journal article

Agreement Is Not a Criterion: What Matching Human Labels Certifies About an LLM Judge, Why a Shared Shortcut Passes the Test, and Why Every External Criterion Found So Far Validates a Use Rather Than a Judge

Pranay Mahendrakar 2026

An automatic judge is usually accepted on one piece of evidence: it agrees with human preference labels about as often as humans agree with each other. That number is then read as permission to use the judge for ranking models, filtering data, selecting samples and supplying reward. This paper…

Cite Related research LOR-00090 · 15 views · 26 downloads

PDF376.2 KB

Journal article

Problem Choice Without a Referee: An Automated Novelty Check Certifies an Empty Search, a Rediscovery Benchmark Credits the Hypothesis Humans Found Next, and Every Located Referee of Worth Independent of Proposer and Field Needs the Objective Given

Pranay Mahendrakar 2026

Pipelines that claim to automate scientific discovery gate their search on a judgement that a proposed idea or problem is novel and worth pursuing. The best-controlled evidence on that judgement points two ways at once: in a blind study involving more than one hundred NLP researchers, expert…

Cite Related research LOR-00075 · 9 views · 13 downloads

PDF530.7 KB

Journal article

Refusal Is Not a Rate: What a Context-Conditioned Refusal Policy Would Have to Specify, Why the Benchmarks That Score Context Assume It Is Verified, and the Provenance Term No Instrument Prices

Pranay Mahendrakar 2026

A language model that refuses a harmful request and a model that refuses a harmless one produce the same event, and most of the literature on the jailbreak/over-refusal trade-off counts both as one refusal rate. The standard proposal for escaping the trade-off is to make refusal depend on context…

Cite Related research LOR-00055 · 8 views · 20 downloads

PDF6.3 MB

Journal article

Scored Before the Question Exists: What a Write-Time Importance Value in LLM Agent Memory Predicts, Why an Additive Term Is Not a Prior, and the Ablation the Canonical Architecture Did Not Run

Pranay Mahendrakar 2026

An agent that stores what happens to it must decide what is worth storing and, later, what is worth reading back. The most-copied mechanism for the first decision is a scalar written at storage time: a language model is shown a record and asked how important it is. (2023) introduced this in the…

Cite Related research LOR-00032 · 7 views · 12 downloads

PDF556.6 KB

Journal article

Automatic Benchmarks Measure an Asymmetry: Privileged Information, Panel-Relative Difficulty, and Two Self-Biases That Pull in Opposite Directions

Pranay M. Mahendrakar 2026

Automatic benchmark construction is presented as a response to saturation: a language model proposes topics, writes items, and supplies answer keys, and the resulting datasets are reported to be harder and more novel than human-built ones. The standard objection is that a model cannot examine…

Cite Related research LOR-00037 · 11 views · 46 downloads

PDF511.7 KB

Journal article

A Second Model Is Not a Second Opinion: Peer Verification, Correlated Failure, and the Independence Variable Nobody Reports

Pranay M. Mahendrakar 2026

When a language model cannot reliably detect its own errors, the standard remedy is to have a second model check the first. That substitution is now the default across agent benchmarks, reward pipelines and safety evaluations, and it rests on an assumption that is rarely stated and that, in the…

Cite Related research LOR-00073 · 11 views · 37 downloads

PDF456.6 KB

Journal article

A Second Model Is Not a Second Opinion: Peer Verification, Correlated Failure, and the Independence Variable Nobody Reports

Pranay M. Mahendrakar 2026

When a language model cannot reliably detect its own errors, the standard remedy is to have a second model check the first. That substitution is now the default across agent benchmarks, reward pipelines and safety evaluations, and it rests on an assumption that is rarely stated and that, in the…

Cite Related research LOR-00043 · 7 views · 6 downloads

PDF451.8 KB

8 records