Agent evaluation: open-access research

Journal article

What Counts as a Control? A Scaffold Effect Is a Contrast Conditional on What Was Held Fixed, the 2026 Controlled Studies Each Held Something Different, and Only a Declared Spend Interval Makes a Harness Claim Refutable

Pranay Mahendrakar 2026

A language-model agent is a model wrapped in a harness that builds its context, exposes its tools, decides when to retry and when to stop. Several 2026 studies set out to measure how much the harness matters relative to the model, holding the model and the tasks fixed, and their answers do not…

Cite Related research LOR-00103 · 5 views · 6 downloads

PDF405.2 KB

Journal article

Three Things Called Budget Awareness: Observability, Forecasting and Allocation in LLM Agents, Why Every Published Allocation Gain Is Keyed to a Signal Measured After the Fact, and the Run-to-Run Variance No Forecast Is Scored Against

Pranay Mahendrakar 2026

Two 2024-2026 literatures make claims about resource use in language-model agents that look incompatible. One reports that allocating test-time compute according to problem difficulty beats spending it uniformly, by margins up to a factor of four in compute at equal accuracy. The other reports…

Cite Related research LOR-00016 · 5 views · 30 downloads

PDF619.5 KB

Journal article

A Second Model Is Not a Second Opinion: Peer Verification, Correlated Failure, and the Independence Variable Nobody Reports

Pranay M. Mahendrakar 2026

When a language model cannot reliably detect its own errors, the standard remedy is to have a second model check the first. That substitution is now the default across agent benchmarks, reward pipelines and safety evaluations, and it rests on an assumption that is rarely stated and that, in the…

Cite Related research LOR-00073 · 11 views · 37 downloads

PDF456.6 KB

Journal article

A Second Model Is Not a Second Opinion: Peer Verification, Correlated Failure, and the Independence Variable Nobody Reports

Pranay M. Mahendrakar 2026

When a language model cannot reliably detect its own errors, the standard remedy is to have a second model check the first. That substitution is now the default across agent benchmarks, reward pipelines and safety evaluations, and it rests on an assumption that is rarely stated and that, in the…

Cite Related research LOR-00043 · 7 views · 6 downloads

PDF451.8 KB

4 records

What Counts as a Control? A Scaffold Effect Is a Contrast Conditional on What Was Held Fixed, the 2026 Controlled Studies Each Held Something Different, and Only a Declared Spend Interval Makes a Harness Claim Refutable

Three Things Called Budget Awareness: Observability, Forecasting and Allocation in LLM Agents, Why Every Published Allocation Gain Is Keyed to a Signal Measured After the Fact, and the Run-to-Run Variance No Forecast Is Scored Against

A Second Model Is Not a Second Opinion: Peer Verification, Correlated Failure, and the Independence Variable Nobody Reports

A Second Model Is Not a Second Opinion: Peer Verification, Correlated Failure, and the Independence Variable Nobody Reports