A Best Match Is Not a Twin: What Feature Similarity Between State Space Models and Transformers Certifies, Why the Null and the Shared Inputs Decide the Verdict, and the Controls That Would Show Two Architectures Learned the Same Thing

@misc{mahendrakar2026best,
  author       = {Mahendrakar, Pranay},
  title        = {{A Best Match Is Not a Twin: What Feature Similarity Between State Space Models and Transformers Certifies, Why the Null and the Shared Inputs Decide the Verdict, and the Controls That Would Show Two Architectures Learned the Same Thing}},
  year         = {2026},
  month        = oct,
  date         = {2026-10-26},
  publisher    = {Life of Research},
  howpublished = {Preprint},
  doi          = {10.5281/zenodo.23652519},
  url          = {https://lifeofresearch.com/preprints/142},
  abstract     = {(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Life of Research Selective state space models such as Mamba compress the past into a fixed-size state, while Transformers attend over the whole context, and a growing set of studies asks whether the two learn the same internal features. A 2026 study reports that 99.98\% of a Mamba model's sparse-autoencoder features have a strong Transformer counterpart and reads this as support for the universality hypothesis. In this analysis the author reviews and synthesises published results to show that such verdicts are set less by the models than by three choices the measurement makes: the reference point a score is judged against, the inputs the two models share, and the instrument that extracts features. The author derives the expected Jaccard similarity of two binary features that share nothing and shows that the study's reported band, 0.234 to 0.411 for every feature's best match, is what unrelated features would score if they fired on roughly a quarter to three fifths of tokens; firing rates were not reported, so this cannot be checked. Against same-architecture skylines, the strongest design supports a narrower claim than universality: features recur across the two families nearly as well as across random seeds, most strongly for token-level features, while the documented differences sit in context mixing and recurrent state, which residual-stream dictionaries do not measure. Sixteen published findings are consolidated in a table, a seven-step procedure licenses a convergence claim at one of four levels, and five studies are proposed that would settle the open part.},
  keywords     = {universality hypothesis, representational similarity, sparse autoencoders, state space models, Mamba, mechanistic interpretability, null calibration, Platonic representation hypothesis},
  language     = {en},
  copyright    = {Creative Commons Attribution 4.0}
}