{"id":17,"identifier":"LOR-00017","url":"https://lifeofresearch.com/articles/17","title":"Mechanistic Interpretability of In-Context Learning: A Survey of Known Circuits and Open Problems","abstract":"In-context learning (ICL) — the ability of a transformer language model to acquire a new input-output mapping from a handful of demonstrations in its prompt, with no weight updates — remains the most striking and least understood capability of large language models. Mechanistic interpretability has produced two influential partial explanations: induction-head circuits, which implement fuzzy pattern completion of the form [A][B]…[A]→[B], and function vectors, which compress an entire ICL task into a low-dimensional representation routed by a small number of attention heads. A third line of work argues that ICL implements implicit gradient descent on linear regression problems. This paper contributes (i) a structured synthesis of these three accounts, (ii) a sharper articulation of where each explanation succeeds and where it breaks down, and (iii) a concrete research agenda targeting five gaps that current work does not address: compositional ICL, ICL in instruction-tuned frontier models, the transient-versus-persistent learning distinction, the polysemanticity barrier at scale, and the deeper interpretive question of what it means to claim that a circuit \"explains\" a behaviour. We argue the field has solved the easy cases and now faces the hard ones.","authors":[{"name":"Pranay Mahendrakar","affiliation":"","orcid":"0009-0003-7224-029X"}],"type":"article","license":"CC-BY-4.0","keywords":[],"publication_date":"2026-04-28","version":1,"views":8,"downloads":5,"created_at":"2026-09-22 17:34:08","doi":"10.5281/zenodo.19853292","source":"Zenodo","source_url":"https://zenodo.org/records/19853292","files":[{"id":17,"name":"Mechanistic_Interpretability_of_ICL_Mahendrakar.pdf","size":91751,"mime":"application/pdf","downloads":5,"download_url":"https://lifeofresearch.com/articles/17/files/17/Mechanistic_Interpretability_of_ICL_Mahendrakar.pdf"}]}