Mechanistic Interpretability of In-Context Learning: A Survey of Known Circuits and Open Problems

@misc{mahendrakar2026mechanistic,
  author       = {Mahendrakar, Pranay},
  title        = {{Mechanistic Interpretability of In-Context Learning: A Survey of Known Circuits and Open Problems}},
  year         = {2026},
  month        = apr,
  date         = {2026-04-28},
  publisher    = {Life of Research},
  doi          = {10.5281/zenodo.19853292},
  url          = {https://lifeofresearch.com/articles/17},
  abstract     = {In-context learning (ICL) — the ability of a transformer language model to acquire a new input-output mapping from a handful of demonstrations in its prompt, with no weight updates — remains the most striking and least understood capability of large language models. Mechanistic interpretability has produced two influential partial explanations: induction-head circuits, which implement fuzzy pattern completion of the form [A][B]…[A]→[B], and function vectors, which compress an entire ICL task into a low-dimensional representation routed by a small number of attention heads. A third line of work argues that ICL implements implicit gradient descent on linear regression problems. This paper contributes (i) a structured synthesis of these three accounts, (ii) a sharper articulation of where each explanation succeeds and where it breaks down, and (iii) a concrete research agenda targeting five gaps that current work does not address: compositional ICL, ICL in instruction-tuned frontier models, the transient-versus-persistent learning distinction, the polysemanticity barrier at scale, and the deeper interpretive question of what it means to claim that a circuit "explains" a behaviour. We argue the field has solved the easy cases and now faces the hard ones.},
  language     = {en},
  copyright    = {Creative Commons Attribution 4.0}
}