2026-10-16T16:57:23Z https://lifeofresearch.com/oai2

oai:lifeofresearch.com:17 2026-09-22T17:34:08Z type:article openaire

Mechanistic Interpretability of In-Context Learning: A Survey of Known Circuits and Open Problems Pranay Mahendrakar In-context learning (ICL) — the ability of a transformer language model to acquire a new input-output mapping from a handful of demonstrations in its prompt, with no weight updates — remains the most striking and least understood capability of large language models. Mechanistic interpretability has produced two influential partial explanations: induction-head circuits, which implement fuzzy pattern completion of the form [A][B]…[A]→[B], and function vectors, which compress an entire ICL task into a low-dimensional representation routed by a small number of attention heads. A third line of work argues that ICL implements implicit gradient descent on linear regression problems. This paper contributes (i) a structured synthesis of these three accounts, (ii) a sharper articulation of where each explanation succeeds and where it breaks down, and (iii) a concrete research agenda targeting five gaps that current work does not address: compositional ICL, ICL in instruction-tuned frontier models, the transient-versus-persistent learning distinction, the polysemanticity barrier at scale, and the deeper interpretive question of what it means to claim that a circuit "explains" a behaviour. We argue the field has solved the easy cases and now faces the hard ones. Life of Research 2026-04-28 info:eu-repo/semantics/article https://lifeofresearch.com/articles/17 https://doi.org/10.5281/zenodo.19853292 eng https://lifeofresearch.com/articles/17/files/17/Mechanistic_Interpretability_of_ICL_Mahendrakar.pdf application/pdf https://creativecommons.org/licenses/by/4.0/ info:eu-repo/semantics/openAccess