Construct validity

Journal article

A Blind Score Is Not a Reliance Measure: What Answering Without the Audio Shows About Audio-Language Benchmarks, Why the Partial-Input Debate Already Settled Half of It, and the Counterfactual-Audio Test the Other Half Needs

Pranay Mahendrakar – 2026

Audio-language models are now routinely tested with the audio taken away, and the results are read in opposite directions. One 2026 audit reports that eight models keep 60 to 72 percent of their accuracy on three benchmarks when the audio is omitted, and concludes that the scores largely reflect…

Cite Related research LOR-00107 · 5 views · 5 downloads

PDF429.3 KB

Journal article

Agreement Is Not a Criterion: What Matching Human Labels Certifies About an LLM Judge, Why a Shared Shortcut Passes the Test, and Why Every External Criterion Found So Far Validates a Use Rather Than a Judge

Pranay Mahendrakar – 2026

An automatic judge is usually accepted on one piece of evidence: it agrees with human preference labels about as often as humans agree with each other. That number is then read as permission to use the judge for ranking models, filtering data, selecting samples and supplying reward. This paper…

Cite Related research LOR-00090 · 15 views · 26 downloads

PDF376.2 KB

Journal article

Self-Model or Self-Simulation? A Machine Self-Awareness Index Averages Sub-Scores With No Common Referent and No Fixed Sign, Why Persistent Identity, Goal Stability and Memory Continuity Are Not Evidence of Self-Access, and the Validity Tests Any Composite Would Have to Pass

Pranay Mahendrakar – 2026

Some proposals to quantify machine self-awareness combine several sub-scores - persistent identity, goal stability, cross-session memory continuity, contradiction detection, uncertainty awareness, self-prediction and introspective access - into one index. This paper asks whether such an index…

Cite Related research LOR-00062 · 7 views · 6 downloads

PDF10.9 MB

Journal article

When Deliberation Hurts: Inverse Test-Time Scaling, Unfaithful Traces, and the Case Against a Unified System-2 in LLM Reasoning

Pranay Mahendrakar – 2026

The dominant frame for large reasoning models borrows a label from dual-process psychology: a fast, intuitive System 1 and a slower, deliberate System 2, with longer chains of thought read as more of the latter and therefore, on average, more reliable. A survey of reasoning LLMs states the frame…

Cite Related research LOR-00064 · 6 views · 8 downloads

PDF373.5 KB

Journal article

When Deliberation Hurts: Inverse Test-Time Scaling, Unfaithful Traces, and the Case Against a Unified System-2 in LLM Reasoning

Pranay Mahendrakar – 2026

The dominant frame for large reasoning models borrows a label from dual-process psychology: a fast, intuitive System 1 and a slower, deliberate System 2, with longer chains of thought read as more of the latter and therefore, on average, more reliable. A survey of reasoning LLMs states the frame…

Cite Related research LOR-00049 · 17 views · 15 downloads

PDF371.2 KB

Journal article

Three Things Called Budget Awareness: Observability, Forecasting and Allocation in LLM Agents, Why Every Published Allocation Gain Is Keyed to a Signal Measured After the Fact, and the Run-to-Run Variance No Forecast Is Scored Against

Pranay Mahendrakar – 2026

Two 2024-2026 literatures make claims about resource use in language-model agents that look incompatible. One reports that allocating test-time compute according to problem difficulty beats spending it uniformly, by margins up to a factor of four in compute at equal accuracy. The other reports…

Cite Related research LOR-00016 · 5 views · 30 downloads

PDF619.5 KB

Journal article

Two Kinds of Missing: Underspecified Inputs Versus Unknown Answers, and Why One Abstention Policy Cannot Serve Both

Pranay Mahendrakar – 2026

A language model that declines to answer is scored the same way whether the reason is that the question has several readings and it picked the wrong one, or that the question has one reading and the model does not know the fact it names. AbstentionBench (Kirichenko et al., 2025) makes the…

Cite Related research LOR-00010 · 6 views · 18 downloads

PDF387.4 KB

Journal article

Scored Before the Question Exists: What a Write-Time Importance Value in LLM Agent Memory Predicts, Why an Additive Term Is Not a Prior, and the Ablation the Canonical Architecture Did Not Run

Pranay Mahendrakar – 2026

An agent that stores what happens to it must decide what is worth storing and, later, what is worth reading back. The most-copied mechanism for the first decision is a scalar written at storage time: a language model is shown a record and asked how important it is. (2023) introduced this in the…

Cite Related research LOR-00032 · 7 views · 12 downloads

PDF556.6 KB

Journal article

A Conflict Is Constructed Before It Is Measured: Seven Design Choices That Set Which Way a Language Model Bends, and Why the Context-Memory Results Do Not Compare

Pranay Mahendrakar – 2026

The literature on how language models behave when retrieved context contradicts parametric memory contains a flat contradiction, and neither of its two surveys resolves it. One body of results reports over-reliance on memorised information; another reports that models are highly receptive to…

Cite Related research LOR-00004 · 7 views · 23 downloads

PDF462 KB

Journal article

Out-of-Distribution With Respect to What? Four Referents Behind One Predicate, and Why OOD Detection Results Do Not Transfer to Foundation-Model Reasoning

Pranay M. Mahendrakar – 2026

Out-of-distribution detection was formalised for a setting in which the training distribution is known, the label space is closed, and a designated test set stands in for everything outside it. Foundation models satisfy none of those conditions, and yet a growing body of work reports…

Cite Related research LOR-00033 · 6 views · 9 downloads

PDF509 KB

Journal article

Automatic Benchmarks Measure an Asymmetry: Privileged Information, Panel-Relative Difficulty, and Two Self-Biases That Pull in Opposite Directions

Pranay M. Mahendrakar – 2026

Automatic benchmark construction is presented as a response to saturation: a language model proposes topics, writes items, and supplies answer keys, and the resulting datasets are reported to be harder and more novel than human-built ones. The standard objection is that a model cannot examine…

Cite Related research LOR-00037 · 11 views · 46 downloads

PDF511.7 KB

Journal article

The Crossover Does Not Carry the Claim: Pass@k Inversion, the Reasoning Boundary, and Five Premises the RLVR Debate Leaves Unstated

Pranay M. Mahendrakar – 2026

A model trained with reinforcement learning from verifiable rewards beats its base model at one sample and loses to it at many. The field has read that crossover as a measurement of the reasoning boundary and has spent two years arguing about which direction the boundary moved. This paper…

Cite Related research LOR-00072 · 7 views · 9 downloads

PDF461 KB

Journal article

The Crossover Does Not Carry the Claim: Pass@k Inversion, the Reasoning Boundary, and Five Premises the RLVR Debate Leaves Unstated

Pranay M. Mahendrakar – 2026

A model trained with reinforcement learning from verifiable rewards beats its base model at one sample and loses to it at many. The field has read that crossover as a measurement of the reasoning boundary and has spent two years arguing about which direction the boundary moved. This paper…

Cite Related research LOR-00034 · 6 views · 29 downloads

PDF457.3 KB