A Ceiling Score Is Not a Physical Inference: Why Pretrained Video Encoders Look Physically Competent on One Benchmark and Near Chance on Its Successor, What a Clean Scene Lets a Readout Read Off, and the Rankability Checks a Physics Evaluation Owes

@misc{mahendrakar2026ceiling,
  author       = {Mahendrakar, Pranay},
  title        = {{A Ceiling Score Is Not a Physical Inference: Why Pretrained Video Encoders Look Physically Competent on One Benchmark and Near Chance on Its Successor, What a Clean Scene Lets a Readout Read Off, and the Rankability Checks a Physics Evaluation Owes}},
  year         = {2026},
  month        = oct,
  date         = {2026-10-29},
  publisher    = {Life of Research},
  howpublished = {Preprint},
  doi          = {10.5281/zenodo.23749317},
  url          = {https://lifeofresearch.com/preprints/156},
  abstract     = {(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Life of Research Pretrained video and image encoders are now chosen as the perception front end of world models for manipulation partly on the strength of evaluations that are said to measure physical understanding: violation-of-expectation benchmarks, outcome-prediction tasks, and linear probes for properties such as mass and friction. The published verdicts disagree sharply. A 2025 study reports that a model trained to predict masked video in a learned representation space separates possible from impossible events on the original IntPhys benchmark with 98 percent accuracy, and concludes that intuitive physics can emerge without built-in core knowledge; the same model scores 53.75 percent on IntPhys 2, and no model tested there clears 60 percent overall against a human majority vote of 96.44 percent. A September 2026 study finds that in a clean, fixed-camera scene eight representations, including a randomly initialised network and raw pixels, all land within 0.02 R\textasciicircum{}2 of the ceiling on a task that requires inferring mass and friction, and spread across 0.50 R\textasciicircum{}2 once camera, lighting and clutter vary. In this analysis the author reviews and synthesises published results to show that these verdicts do not answer one question. A physics score can certify four different things: that the outcome is legible in the input, that a latent property was identified from interaction history, that the identified property is used, and that the instrument can rank models at all. The positive results concentrate on the first. On the latent-property benchmarks (Physion++, ComPhy and ExPhy) the negative results concentrate where the second and third are required, while CALIPER's swap control shows a fitted readout using the object's own interaction history with every representation tested, an untrained one included; IntPhys 2 is the exception, a near-chance result on largely object-intrinsic properties in more complex scenes. Almost no benchmark has demonstrated the fourth. The paper consolidates 30 published measurements in a table, separates the shortcut a clean scene offers a linear readout from the different failure on harder violation-of-expectation scenes, states a rankability audit as an algorithm, and proposes five tests that would settle what remains open.},
  keywords     = {intuitive physics, video representation learning, world models, benchmark validity, linear probing, violation of expectation, latent physical properties, robot manipulation},
  language     = {en},
  copyright    = {Creative Commons Attribution 4.0}
}