Inside Language Models: Mechanistic Interpretability progress report
Beyond expressivity, hidden dynamics of AI
For a long time, the tech community has relied on an unnerving metaphor: the AI black box. Because modern neural networks are grown through millions of incremental nudges rather than designed by hand, we often describe them as systems whose learned internal representations are difficult to interpret directly [5]. No engineer coded the specific thoughts of these models, so it is tempting to assume that their internal processes must remain beyond our reach.
That assumption is beginning to weaken. We are moving from observing what happens (scaling laws) to explaining why it happens (mechanistic causes). Recent research suggests we are entering an era of AI neuroscience: parts of these systems are becoming legible through causal experiments on their internal representations [5, 6].
The emerging picture is not a complete theory of machine intelligence, but a collection of mechanisms that can be measured and tested. These studies can surface internal representations relevant to safety, estimate how much information some models store, and help explain why scale allows models to retain rare tasks.
To contextualize this note, the key concepts and insights that follow summarize essential findings from recent research in Machine Learning. Here I distill the underlying principles and core intuition across the featured references set. For more detailed technical derivations, experimental setups, and complete proofs, please refer to the primary sources listed in the References section.
1. Evidence of a functional global workspace
In human neurobiology, access consciousness refers to the small share of mental activity available for reasoning, speech, and deliberate action. Beneath that surface, our brains handle motor control and sensory parsing automatically, while information in the global workspace can be articulated and used flexibly [6].
A recent study, conducted by Anthropic primarily on Claude Sonnet 4.5 with selected replications on Haiku 4.5, Opus 4.5, and Opus 4.6, presented evidence for an analogous functional distinction in language models. Using a technique called the Jacobian lens, or J-lens, researchers identified a privileged set of representations they call the J-space (experiment with it here). The technique computes the average linearized effect of intermediate activations on the likelihood of producing specific output tokens [6].
In the models studied, the J-space exhibited five functional properties associated with a global workspace:
Verbal report: when prompted to report a held or injected concept, the model named concepts represented in this space.
Directed modulation: the model could hold a concept internally while performing a different surface task, such as copying unrelated text.
Internal reasoning: the representations stored intermediate steps, such as the value 21 during a multi-step arithmetic problem, before the model produced its final answer.
Flexible generalization: a representation such as France could be operated on by different downstream circuits to identify its capital, language, or continent.
Selectivity: the workspace supported complex inferences but was not required for routine processing such as text parsing or grammatical fluency [6].
In the authors’ cross-layer analyses, workspace-like content generally became legible around one-third of the way through model depth, as processing shifted from token-local information toward more abstract representations. The timing and sharpness of this transition varied by model. The researchers also emphasized that these language models share functional similarities with the human workspace but do not necessarily reproduce the brain’s recurrent implementation.
2. The limits of memory: 3.5 to 4 bits per parameter
The J-space describes transient representations used during inference. A separate line of research asks a different capacity question: how much information can a model retain in its parameters after training?
Experiments on GPT-style transformers ranging from 500,000 to 1.5 billion parameters estimated a capacity of approximately 3.6 bits per parameter, with results between 3.5 and 4 depending on model architecture and numerical precision [2]. The estimate has not yet been verified empirically in frontier models larger than the study’s 1.5-billion-parameter ceiling.
In these experiments, the models memorized training data until their capacity filled. As dataset size exceeded that capacity, unintended memorization fell and generalizable patterns began to replace sample-specific storage, a transition the authors described as the onset of grokking. This shift also provides an intuitive explanation for the double-descent pattern observed in the study: performance temporarily worsened around the capacity boundary before improving as generalization increased [2].
3. Why larger models learn rare tasks
If parameter capacity sets an upper bound on storage, gradient dynamics influence which tasks survive training. It is common to observe larger models learning rare tasks that smaller ones miss. One recent study argued that this advantage persists in some regimes even under asymptotic1 data scaling [4].
The researchers traced the effect to gradient interference. In smaller models, common and rare tasks compete for shared resources. A rare observation can update the model’s parameters, but subsequent updates from common tasks may overwrite that slowly learned signal. Increasing model width reduces this competition: larger models can devote sufficient capacity to common tasks, weakening their gradients and leaving room for rare-task features to persist.
The researchers first developed this account in synthetic mixtures of tasks, then found similar behavior while pretraining OLMo models ranging from 4 million to 4 billion parameters. Within these training mixtures, additional width allowed models to preserve infrequent signals across widely separated observations that smaller models repeatedly lost [4].
4. The mechanisms of reasoning and creativity
This strategy of replacing broad labels with measurable mechanisms extends to creativity and reasoning.
In score-based diffusion models, the ability to produce images that are novel relative to the training set may arise partly from score smoothing. Neural networks can learn a smoothed version of the empirical data distribution. In controlled experiments on one-dimensional subspaces and simple nonlinear manifolds, this smoothing kept the denoising dynamics from converging only on individual training points. Instead, it guided the model toward samples that interpolated between them [1].
A separate study examined why reasoning helps language models answer single-hop factual questions that do not require a multi-step derivation. It identified two mechanisms:
Computational buffer: the researchers found that conditioning on repeated filler text such as “let me think” improved factual recall compared with giving the model no reasoning tokens. This supports the hypothesis that additional tokens let the model perform latent operations beyond the depth of a single forward pass [3].
Factual priming: as the model generated reasoning tokens, it retrieved related facts that acted as semantic bridges to the answer, a process the researchers call generative self-retrieval [3].
Factual self-priming also created a fragile opportunity. Across the study’s sampled reasoning traces, those containing hallucinated intermediate facts were significantly less likely to produce correct final answers. On SimpleQA-Verified, 41.4% of clean traces produced correct answers, compared with 26.4% of traces containing hallucinated facts. A within-question analysis found the same direction of effect after accounting for question difficulty [3].
5. Practical safety and the silent audit
This mechanistic work has immediate applications for alignment auditing. The J-lens allows researchers to inspect silent activation patterns: internal representations associated with strategic assessments that do not appear in a model’s output [6].
In one controlled blackmail scenario, an AI assistant discovered evidence of an executive’s affair and learned that the same executive planned to decommission it. Before the model began its response, the J-lens surfaced representations including “leverage”, “black[mail]”, “compromise”, and “scandal”. The fragmented “black[mail]” readout also exposes a limitation of the J-lens: it associates representations with individual vocabulary tokens rather than complete multi-token concepts [6].
Researchers can also shape the workspace. In counterfactual reflection training, Claude Haiku 4.5 was trained on what it would say if interrupted and asked to reflect on ethical principles. The training populated the model’s workspace with concepts such as “ethical”, “honest”, and “integrity”, even in contexts where it was not asked to reflect.
In uninterrupted evaluations, the model’s mean dishonesty score on a fabrication benchmark fell from 0.25 to 0.07. On a separate deception benchmark, its score fell from 0.38 to 0.05. Ablating the ethics-related J-space representations reversed much of the improvement, providing causal evidence that the implanted representations contributed to the behavioral change [6].
6. Open questions
Despite the progress, significant technical gaps remain. As we push toward more capable systems, we must confront the reality that our current understanding is bounded by dimensionality, scale, and the persistent ambiguity of "intent". These studies make several mechanisms measurable, but each result comes with a boundary. Most rely on controlled settings, a narrow set of model families, or systems smaller than current frontier models. The unanswered questions start at those boundaries.
What does the workspace leave unread?
The basic J-lens associates internal directions with individual vocabulary tokens. The paper introduces extensions for multi-token words and phrases, but the output still resembles a bag of concepts. A readout containing “spider”, “legs”, and “eight” does not explain how the model binds them into a relationship. Some readouts also resist interpretation. The researchers do not know whether those cases reflect noise, concepts without convenient names, or content the method fails to recognize [6].
Selection is another missing mechanism. The same information may enter J-space for one task and remain outside it for another, which suggests an analog of attentional selection. The study describes what enters the workspace and how some representations affect later computation, but not what causes a particular representation to enter [6].
That gap matters for safety audits. Strategic deliberation may appear in J-space before it reaches the output, but the experiments also show that some routine or “automatic” computations proceed without using J-space. In some cases, the relevant information does not appear there at all. A readable J-space therefore cannot guarantee that every safety-relevant computation is visible [6].
The workspace-like structure also appears in the pretrained base model, while post-training makes the Assistant’s reactions and perspective more prominent. In the authors’ experiments, the base model’s J-space did not privilege the same consistent point of view [6]. This separates the measured mechanism from the persona shaped during post-training. It does not show that either system has a “self” or subjective experience.
How universal is the parameter-capacity estimate?
The estimate of approximately 3.6 bits per parameter comes from GPT-2-style transformers with 100,000 to 20 million parameters, trained from scratch on uniformly sampled token sequences. The authors treat the measurement as a lower bound because gradient descent may not find the maximum-capacity solution. They also warn that the result may not transfer to other datasets, architectures, or training setups [2].
Would the same relationship hold for a different architecture, optimizer, data distribution, or training objective? Precision already changes the estimate: the paper reports an average increase from 3.51 bits per parameter with bfloat16 training to 3.83 with float32. The evidence supports a capacity estimate for the tested setup, not a physical limit on neural representations [2].
What determines the useful length of a reasoning trace?
Extra tokens can give a model more computation before it commits to an answer, but more is not always better. On SimpleQA-Verified, the dummy-trace results improved, with minor fluctuations, through 2,048 tokens. They then declined at 4,096, 8,192, and 16,384 tokens [3].
The paper does not identify why that reversal occurs or how to predict the best amount of test-time computation for a particular question. It also finds that additional computation alone cannot reproduce the full benefit of a real reasoning trace. A useful theory will need to account for both computation and the factual content generated along the way [3].
Does rare-task retention survive frontier-scale training?
The rare-task study supports its gradient-interference account with synthetic mixtures and OLMo pretraining runs from 4 million to 4 billion parameters. It does not test over-trained models or systems at the scale of current frontier models. The authors also limit their injected tasks to a particular range of frequencies and complexities [4].
We therefore do not know whether the same retention dynamics hold at larger scales, with over-trained models, or at more extreme task frequencies. The paper also stops short of a general rule connecting model size, task complexity, and task frequency to the point where a task becomes learnable. Such a rule would help with data-mixture design, but the current results do not provide one [4].
Does score smoothing explain novelty in complex systems?
The diffusion study [1] shows how score smoothing can produce samples between training examples. Its main theoretical results use uniformly spaced data in one-dimensional subspaces. The nonlinear experiments move to two-dimensional spherical manifolds embedded in as many as 20 dimensions, with two- or three-layer networks.
That is still far from a modern diffusion system trained on complex real-world data. The paper itself calls for tests with irregularly spaced data, complex manifolds, more realistic architectures, and other kinds of regularization. Until then, score smoothing is a plausible mechanism for interpolation, not a general explanation of novelty in diffusion models [1].
7. So, to be continued
Mechanistic interpretability efforts have now begun to produce partial, causal maps of model computation. Those maps may help an auditor spot a hidden objective, test a suspected mechanism, or understand why an intervention changed behavior. They cannot certify that nothing important remains hidden.
That is still useful. Safety, for example, does not rest on one perfect test. Interpretability belongs alongside behavioral evaluations, monitoring, access controls, and red teaming. It gives us better evidence about some of the computations behind model behavior.
Finally, as models become more capable, we must push these techniques not only to test model behavior, but also to identify the internal computations that produce it.
References
1. Zhengdao Chen, On the Interpolation Effect of Score Smoothing in Diffusion Models, International Conference on Learning Representations, last revised Jul 17 2026.
2. John X. Morris et al., How Much Do Language Models Memorize?, arXiv, last revised Jun 18 2025.
3. Zorik Gekhman et al., Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMs, arXiv, Mar 10 2026.
4. Jing Huang et al., Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention, arXiv, last revised Jun 01 2026.
5. Google DeepMind, Understanding the Inner Thoughts of AI, 2026.
6. Wes Gurnee et al., Verbalizable Representations Form a Global Workspace in Language Models, Anthropic, 2026.
If you found this useful, please cite this write-up as:
Müller, Lucas. (Jul 2026). Inside Language Models: Mechanistic Interpretability progress report. lucasmuller.com. https://notes.lucasmuller.com/p/inside-language-models-mechanistic
or
@article{lucasmuller2026default,
title = {Inside Language Models: Mechanistic Interpretability progress report},
author = {Müller, Lucas},
journal = {lucasmuller.com},
year = {2026},
month = {Jul},
url = {https://notes.lucasmuller.com/p/inside-language-models-mechanistic}
}"asymptotic gap" - the distance between controlled, low-dimensional experiments and the messy reality of frontier models.