A Hit Rate Is a Retention Policy: Prefix-Cache Reuse in Agentic Coding Traces Compounds the Workload With the Server's Eviction Rule and the Harness's Invalidations, Which Serving Assumptions the Traces Actually Break, and the Rent-or-Buy Rule That Prices the Turn Boundary

@misc{mahendrakar2026hit,
  author       = {Mahendrakar, Pranay},
  title        = {{A Hit Rate Is a Retention Policy: Prefix-Cache Reuse in Agentic Coding Traces Compounds the Workload With the Server's Eviction Rule and the Harness's Invalidations, Which Serving Assumptions the Traces Actually Break, and the Rent-or-Buy Rule That Prices the Turn Boundary}},
  year         = {2026},
  month        = oct,
  date         = {2026-10-21},
  publisher    = {Life of Research},
  howpublished = {Preprint},
  doi          = {10.5281/zenodo.23519028},
  url          = {https://lifeofresearch.com/preprints/122},
  abstract     = {(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Life of Research Coding agents are becoming a major class of language-model traffic, and the first production-scale characterisation of that traffic, 13.5 million GitHub Copilot agent sessions, concludes that its findings "challenge assumptions underlying current LLM-serving systems and provide an empirical foundation for agent-native infrastructure." Its central evidence is the prefix-cache hit rate: about 90 percent within an agent turn, far lower across turn boundaries, 8 percent after a model switch. Other 2026 studies of agent traffic report hit rates from above 98 percent to below 17 percent. In this analysis the author reviews and synthesises published results to show that a measured hit rate is the product of three factors that the studies vary separately: the reuse the workload makes available, the probability that cached state survives until it is reused, which is set by the server's retention rule, capacity and load, and the probability that the agent harness has not invalidated the prefix by switching models or compacting context. Holding the workload and model fixed, one study's hit rate falls from 98.85 to 16.78 percent when concurrency rises from 15 to 20 agents; in another, sweeping the eviction timeout from one minute to one hour moves the achievable rate from 85.4 to 98.6 percent. The paper audits which serving assumptions the traces break, finds that statelessness was abandoned for multi-turn chat by 2024 and that context rewriting which invalidates the prefix predates agents, while the size of the state at stake, model switches made by automatic routing and LLM-written compaction are new, and separates three senses in which agent serving is said to be dominated by input, by decode or by cache charges. It states the turn-boundary retention decision as a rent-or-buy problem with a hazard-rate threshold, works a keep-alive break-even from published prices, and specifies the policy-held-fixed comparisons that would settle what is specific to agents.},
  keywords     = {LLM serving, KV cache, prefix caching, coding agents, workload characterization, cache retention, ski rental, context compaction},
  language     = {en},
  copyright    = {Creative Commons Attribution 4.0}
}