One Pass, Three Purchases: Deduplication's Efficiency, Generalisation and Privacy Arguments Each Hold in Their Own Regime, Rest on Different Properties of the Repetition Distribution, and Cannot All Be Bought With One Similarity Threshold

@misc{mahendrakar2026one,
  author       = {Mahendrakar, Pranay},
  title        = {{One Pass, Three Purchases: Deduplication's Efficiency, Generalisation and Privacy Arguments Each Hold in Their Own Regime, Rest on Different Properties of the Repetition Distribution, and Cannot All Be Bought With One Similarity Threshold}},
  year         = {2026},
  month        = oct,
  date         = {2026-10-10},
  publisher    = {Life of Research},
  doi          = {10.5281/zenodo.23271874},
  url          = {https://lifeofresearch.com/articles/95},
  abstract     = {(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Deduplication became a fixed stage of language-model pretraining on the strength of three claims made together: removing duplicates saves compute, improves what the model learns, and reduces memorisation and therefore privacy risk. A reading of the literature from 2024 to 2026 can make it look as if all three have collapsed, since repeating data for several epochs is nearly free, globally deduplicated web data can be worse than what it removed, and near-copies that deduplication leaves behind are memorised almost as well as exact ones. This paper argues that the collapse reading is wrong and that the original bundle was wrong too. Each argument survives, but each rests on a different property of the repetition distribution. The efficiency case is an argument against a heavy tail of moderately repeated low-value content, not against repetition, and it reverses when duplicate counts track quality. The generalisation case holds for verbatim repetition and is in open conflict for semantic duplicates: one 2026 line reports that capable models' gradients treat translations and light surface edits increasingly like copies, and argues that paraphrases behave the same way, while controlled knowledge studies find that paraphrased repetition is what makes facts extractable. The privacy case holds for count-driven verbatim extraction, which deduplication cuts by an order of magnitude or more, but the residual risk sits in fuzzy copies, paraphrases and relative frequency, where the standard attack instruments are weakest, so a quiet audit after deduplication is not evidence of safety. A single similarity threshold with a keep-one rule fixes all three choices at once. The paper sets out the three axes on which the arguments come apart, consolidates the measured values, gives a decision procedure for choosing a duplicate relation and a count policy per goal, and names five measurements that would settle the open parts.},
  keywords     = {deduplication, pretraining data, data repetition, memorization, privacy, data curation, language models},
  language     = {en},
  copyright    = {Creative Commons Attribution 4.0}
}