2026-10-17T13:49:46Z https://lifeofresearch.com/oai2

oai:lifeofresearch.com:109 2026-10-15T08:25:59Z type:article openaire

10.5281/zenodo.23353594 https://lifeofresearch.com/articles/109 Pranay Mahendrakar 0009-0003-7224-029X SONYTECH Life of Research 2026 synthetic data language model pretraining model collapse data-constrained scaling rephrasing verification self-improvement credit assignment Journal article Creative Commons Attribution 4.0 (c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. The expectation that language models will keep improving after the supply of human text runs out rests on a claim about synthetic data: that a model, or a family of models, can produce training text worth more than the text it was trained on. A competing reading holds that no synthetic-data success is self-generated at all, because every pipeline that works draws on something outside the generating model, most visibly a verifier. In this analysis the author reviews and synthesises published results on synthetic data for pretraining, continued pretraining and mid-training, and argues that both readings are posed at the wrong grain. Read as a conservation law, the imported-signal claim is true and uninformative, since every generator's weights were themselves fitted to human text. Read as a claim about what explains a measured gain, it splits. The paper separates three contributions a synthetic corpus can make - new information, re-presentation of information already in an organic seed, and selection within the generator's own distribution - and four places the signal can come from: the seed text, the generator's prior, a selection signal, and the designer's prompts or grammars. Against that ledger, the best-controlled pretraining gains are mostly re-presentation: they beat repetition of the same corpus, they persist when the generator is small, and summarisation of web pages nearly matches teacher-written textbooks. There is no verifier in those pipelines. The verifier does the work where the target is correctness, in geometry, code and arithmetic. The most self-contained bootstrapping results use held-out human text as the scorer. No located pretraining result shows a gain from self-generation with none of the four sources present, and the pipelines that come closest score below the real-data baseline on factuality. The paper consolidates the reported values, states a credit-assignment protocol, and names the five controls that would settle the open part.