Which Signal Did the Synthesis Import? Information, Re-presentation and Selection in Synthetic Pretraining, Why 'the Verifier Is Doing the Work' Holds Only Where Correctness Is the Target, and the Controls That Would Assign the Credit

@misc{mahendrakar2026which,
  author       = {Mahendrakar, Pranay},
  title        = {{Which Signal Did the Synthesis Import? Information, Re-presentation and Selection in Synthetic Pretraining, Why 'the Verifier Is Doing the Work' Holds Only Where Correctness Is the Target, and the Controls That Would Assign the Credit}},
  year         = {2026},
  month        = oct,
  date         = {2026-10-14},
  publisher    = {Life of Research},
  doi          = {10.5281/zenodo.23353594},
  url          = {https://lifeofresearch.com/articles/109},
  abstract     = {(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. The expectation that language models will keep improving after the supply of human text runs out rests on a claim about synthetic data: that a model, or a family of models, can produce training text worth more than the text it was trained on. A competing reading holds that no synthetic-data success is self-generated at all, because every pipeline that works draws on something outside the generating model, most visibly a verifier. In this analysis the author reviews and synthesises published results on synthetic data for pretraining, continued pretraining and mid-training, and argues that both readings are posed at the wrong grain. Read as a conservation law, the imported-signal claim is true and uninformative, since every generator's weights were themselves fitted to human text. Read as a claim about what explains a measured gain, it splits. The paper separates three contributions a synthetic corpus can make - new information, re-presentation of information already in an organic seed, and selection within the generator's own distribution - and four places the signal can come from: the seed text, the generator's prior, a selection signal, and the designer's prompts or grammars. Against that ledger, the best-controlled pretraining gains are mostly re-presentation: they beat repetition of the same corpus, they persist when the generator is small, and summarisation of web pages nearly matches teacher-written textbooks. There is no verifier in those pipelines. The verifier does the work where the target is correctness, in geometry, code and arithmetic. The most self-contained bootstrapping results use held-out human text as the scorer. No located pretraining result shows a gain from self-generation with none of the four sources present, and the pipelines that come closest score below the real-data baseline on factuality. The paper consolidates the reported values, states a credit-assignment protocol, and names the five controls that would settle the open part.},
  keywords     = {synthetic data, language model pretraining, model collapse, data-constrained scaling, rephrasing, verification, self-improvement, credit assignment},
  language     = {en},
  copyright    = {Creative Commons Attribution 4.0}
}