Aleksandra Bakalova
Saarbrücken, Germany
I am Sasha (short for Aleksandra) Bakalova, PhD student in LaCoCo Lab at Saarland University supervised by Prof. Michael Hahn. I am also a member of RTG ‘‘Neuroexplicit Models of Language, Vision, and Action’’.
I did my Masters in Language Science and Technology at Saarland University, and my undergraduate in Applied Mathematics and Computer Science at HSE University. During my undergraduate, I once interned at Google as a Software Engineer, and once was a part of Student Training in Engineering Program.
I want to develop interpretability methods that can help us understand, debug and control AI models.
If you wish to chat, feel free to drop me an email!
News
| Aug 14, 2026 | Participated in ADFOCS 2026, the Max Planck summer school on Theory and Machine Learning. |
|---|---|
| Jun 22, 2026 | We gave a talk at FLaNN Seminars about our paper on decompiling transformers. The recording is available on YouTube. |
| May 15, 2026 | Our paper on mechanistic understanding of how LLMs learn in-context when some of the demostrations are ambiguious was accepted to ICML 2026! Read the paper on arxiv. |
| Apr 27, 2026 | Presented our work on decompiling transformers at Latent & Implicit Thinking workshop at ICLR 2026! |
| Mar 04, 2026 | We propose a way to rewrite trained models into D-RASP – a programming language that mimics transformers, and show that it works, at least in the controlled setting of small models and algorithmic and formal languages tasks. See the paper on arxiv and this blog post for more visualizations and examples of decompiled programs. UPD: accepted to ICML 2026! |
| Nov 01, 2025 | Excited to start a new chapter as a PhD student at the RTG ‘‘Neuroexplicit Models of Language, Vision, and Action’’ under the supervision of Prof. Michael Hahn! |
| Oct 15, 2025 | Born a Transformer – Always a Transformer? is accepted to Neurips 2025! |
| Jul 31, 2025 | Attended EEML 2025, got a best poster award! Walking around Sarajevo, getting inspired by the amazing people there and their work: it was a lot of fun. |
| Jul 07, 2025 | Our work on understanding in-context learning is accepted to COLM 2025! |
| May 30, 2025 | Born a Transformer – Always a Transformer?: new preprint is on arxiv! We try to bring theory closer to practice and answer a question: how do theoretical limitations of transformers manifest in pre-trained models? See Yana’s X post for a short explanation. |
| Mar 31, 2025 | Our preprint on understanding in-context learning in Gemma2-2b is on arxiv! We found a circuit in the model that performs in-context learning and interpreted the information flow in it. See this X post for a short explanation. |
| Aug 01, 2024 | Started my Erasmus semester at Charles University in Prague. In love with the city! |
| Apr 30, 2024 | Attended ALPS 2024 for some inspiration and breathtaking views :) |
| Oct 30, 2023 | Started a job as a Research Assistant at LaCoCo Lab with Dr. Michael Hahn. Very excited to work on LLM interpretability with such amazing colleagues! |
| Oct 01, 2023 | Started my Masters in Language Science and Technology at Saarland University. |
Publications
2026
- How Few-Shot Examples Add Up: A Causal Decomposition of Function Vectors in In-Context LearningIn Forty-third International Conference on Machine Learning (ICML), 2026We study the mechanisms behind how LLMs learn in context when some of the demonstrations are ambiguous.
- Discovering Interpretable Algorithms by Decompiling Transformers to RASPIn Forty-third International Conference on Machine Learning (ICML), 2026We rewrite trained transformers into D-RASP, a programming language that mimics their computations. We show that this works in the controlled setting of small models trained on algorithmic and formal language tasks.Talks: FLaNN Seminars (YouTube)
2025
- Born a Transformer–Always a Transformer? On the Effect of Pretraining on Architectural AbilitiesThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025We bring theory closer to practice by investigating how the theoretical limitations of transformers manifest in pretrained models.
- Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2BSecond Conference on Language Modeling, 2025We identify a circuit in Gemma-2 2B, 9B and 27B that performs in-context learning and interpret the information flow within it.