Journal article
One Pass, Three Purchases: Deduplication's Efficiency, Generalisation and Privacy Arguments Each Hold in Their Own Regime, Rest on Different Properties of the Repetition Distribution, and Cannot All Be Bought With One Similarity Threshold
(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Deduplication became a fixed stage of language-model pretraining on the strength of three claims made together: removing duplicates saves compute, improves what the model learns, and reduces memorisation and therefore privacy risk. A reading…