A Gap to a Reference Is Not a Loss: What Text-Capability Comparisons Between Vision-Language Models and Their Language Backbones Measure, Why Independent Audits and Vendor Reports Disagree in Sign, and the Controlled Trajectory That Would Assign the Cause

@misc{mahendrakar2026gap,
  author       = {Mahendrakar, Pranay},
  title        = {{A Gap to a Reference Is Not a Loss: What Text-Capability Comparisons Between Vision-Language Models and Their Language Backbones Measure, Why Independent Audits and Vendor Reports Disagree in Sign, and the Controlled Trajectory That Would Assign the Cause}},
  year         = {2026},
  month        = nov,
  date         = {2026-11-01},
  publisher    = {Life of Research},
  howpublished = {Preprint},
  doi          = {10.5281/zenodo.23858880},
  url          = {https://lifeofresearch.com/preprints/179},
  abstract     = {(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Life of Research A vision-language model is usually built by attaching an image encoder to a pretrained language model and training the combination on multimodal data, and a recurring claim is that this damages the language model's own text abilities. A 2026 study gives the strongest form of the claim: across six released model pairs, the language backbones of vision-language models score up to 18.7 points below a text-only reference on instruction following while knowledge benchmarks barely move, and a scalar that measures how strongly the reference model concentrates attention on an early "sink" position ranks the damage with a Spearman correlation of 0.97. Vendor reports for some of the same models say the opposite. InternVL3's report puts its backbone ahead of the same text-only reference on most language benchmarks, and Qwen3-VL's report puts its 8B model slightly ahead on instruction following. In this analysis the author reviews and synthesises published results to show that a gap between a vision-language backbone and a text reference is not, in general, a loss caused by vision-language training. The gap carries four indices that headlines leave implicit: lineage (whether the reference is the checkpoint the model was trained from), protocol (harness, template and answer extraction), mixture (how much text, and in what output format, the multimodal recipe contained) and construct (whether the benchmark probes knowledge, procedure or output format). For one pair, an independent audit and the vendor report agree on MMLU to within a tenth of a point but disagree in sign on GSM8K and on MBPP. With lineage and protocol held fixed, a loss does survive: in a matched 3B run, vision fine-tuning cost about 23 instruction-following points in the first thousand steps, against about 7 for text-only tuning. In the same run, however, the loss arrived before the attention sink weakened, and part of it came back while the sink kept falling. The paper consolidates the measured values in a table, sets out what four candidate mechanisms predict, and gives an attribution procedure that separates pipeline differences, protocol artefacts, recoverable losses of access and losses of capacity.},
  keywords     = {vision-language models, catastrophic forgetting, instruction following, attention sinks, QK normalization, evaluation methodology, multimodal training data, model merging},
  language     = {en},
  copyright    = {Creative Commons Attribution 4.0}
}