What Counts as a Control? A Scaffold Effect Is a Contrast Conditional on What Was Held Fixed, the 2026 Controlled Studies Each Held Something Different, and Only a Declared Spend Interval Makes a Harness Claim Refutable

@misc{mahendrakar2026counts,
  author       = {Mahendrakar, Pranay},
  title        = {{What Counts as a Control? A Scaffold Effect Is a Contrast Conditional on What Was Held Fixed, the 2026 Controlled Studies Each Held Something Different, and Only a Declared Spend Interval Makes a Harness Claim Refutable}},
  year         = {2026},
  month        = oct,
  date         = {2026-10-12},
  publisher    = {Life of Research},
  doi          = {10.5281/zenodo.23306459},
  url          = {https://lifeofresearch.com/articles/103},
  abstract     = {(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. A language-model agent is a model wrapped in a harness that builds its context, exposes its tools, decides when to retry and when to stop. Several 2026 studies set out to measure how much the harness matters relative to the model, holding the model and the tasks fixed, and their answers do not stack: within-model gaps of 0 to 8 points on one terminal benchmark alongside a 40-fold spread in tokens per solved task, 8.5 to 13 points on a coding subset, and up to 28 points on a question-answering benchmark. In this analysis the author reviews and synthesises published results to show that these are not noisy estimates of one quantity. A harness effect is a contrast between two systems conditional on a control set - what was held fixed and what was left free - and a readout, and each study fixed something different: native budget controls, a shared step cap, an attempt count, or nothing at all. The paper argues four points. A cap is not a spend, so a matched ceiling leaves consumption free and lets a harness win by spending. Accuracy is purchasable with attempts, time and retries, so a harness claim with no declared budget excludes no observation. Model and harness interact, so a harness main effect is a property of the model grid. And run-to-run variance is of the same order as many reported gaps. The paper consolidates the measured values in a table and a figure, separates three questions that call for three different controls, states a decision procedure as an algorithm, and names the studies that would settle what remains open.},
  keywords     = {agent evaluation, agent harness, scaffolding, compute-matched baselines, cost-controlled evaluation, experimental design, benchmark validity, LLM agents},
  language     = {en},
  copyright    = {Creative Commons Attribution 4.0}
}