A Recall Score Is Not a Use Forecast: What Factual-Memory Benchmarks Predict About Whether a Conversational Memory System Uses What It Stores, Why the Published Transfer Failure Runs Through the Harness and the Write Policy, and the Within-Harness Test That Would Settle It

@misc{mahendrakar2026recall,
  author       = {Mahendrakar, Pranay},
  title        = {{A Recall Score Is Not a Use Forecast: What Factual-Memory Benchmarks Predict About Whether a Conversational Memory System Uses What It Stores, Why the Published Transfer Failure Runs Through the Harness and the Write Policy, and the Within-Harness Test That Would Settle It}},
  year         = {2026},
  month        = nov,
  date         = {2026-11-03},
  publisher    = {Life of Research},
  howpublished = {Preprint},
  doi          = {10.5281/zenodo.23899944},
  url          = {https://lifeofresearch.com/preprints/181},
  abstract     = {(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Life of Research Long-term memory systems for conversational assistants are compared almost entirely on question-answering benchmarks that ask for a fact stated somewhere in a long history. A 2026 benchmark for what its authors call memory utilization reports that strength on those benchmarks does not reliably carry over to tasks that require combining scattered, implicit and distractor-laden evidence into a long answer. In this analysis the author reviews and synthesises published results to show that this conclusion bundles three claims that the evidence supports to very different degrees. The first, that retrieving the right evidence is necessary for good use but not sufficient, is well supported and is not new: it was already visible inside the recall benchmarks, which report correct retrieval followed by a wrong answer on 15 to 19 percent of questions, summaries retrieved with high recall that did not help, and models that recall a stated preference but fail to apply it unprompted. The second, that a system's rank on a recall benchmark does not predict its rank on use, rests on comparisons with scores the systems' developers reported for themselves. Against those, the five systems shared with the utilization benchmark largely reverse order (Spearman -0.6 by the author's arithmetic); against a shared harness maintained by the developer of one of them, MemOS, the three shared systems keep most of their order, and harnesses run by third parties place the system at the bottom of the utilization ranking low on recall as well. The third claim is implicit and is the one the evidence most undermines: a recall score is not a stable property of a system. The same system's published LoCoMo score spans 10 to 21 points across harnesses, and two harnesses using the same answering model rank four shared systems with zero rank correlation. The paper separates four places where use can fail after access succeeds, consolidates the measured values in a table, and states a within-harness transfer test as an algorithm.},
  keywords     = {long-term memory, conversational agents, memory utilization, LoCoMo, LongMemEval, benchmark validity, evaluation harness, retrieval-augmented generation},
  language     = {en},
  copyright    = {Creative Commons Attribution 4.0}
}