Imagine opening a language model and finding a map of its thoughts: one marker for biology, another for culture, another for a workplace. The map is not the model itself. It is a simplified guide researchers use to make the model’s dense, tangled calculations easier to inspect. A new study asks a troubling question about that guide: if a sentence changes only slightly, will the map still point to the same things?
In a September 14, 2026 preprint, Tobias Ladner and Matthias Althoff test this problem in five open-weight language model families. They focus on interpretable replacement networks, or IRNs: systems trained to approximate parts of a language model’s internal calculations. Some IRNs, including sparse autoencoders, translate complicated internal signals into a small set of active features. Researchers can then give those features plain-language labels based on the examples that activate them.
Think of an IRN as a field guide to a dense forest. It does not reproduce every tree and root; it offers a more legible account of what might be happening inside. That shortcut is useful only if the guide remains faithful to the terrain. If the IRN’s highlighted features change while the underlying calculation does not change in the same way, the guide could give a misleading account of the model.
To test that map, the researchers assembled 1,000 sentences from WikiText-2 and selected a content word with multiple meanings in each. They built sets of meaning-preserving variations: at the mildest level, one content word before the target was replaced with a synonym; stronger tests changed more words or rephrased the sentence. They examined the overlap among the 20 most active features for each version.
The overlap fell sharply under adversarial tests. In the worst-performing layer, the reported top-20 feature overlap dropped 73% for minor perturbations and 95% for major ones. These figures are not a count of how often ordinary paraphrasing misleads every interpreter. The researchers searched for troublesome cases, so the results show how low overlap could be driven—not how frequently it would happen in everyday use. But the pattern appeared across sparse autoencoders and transcoders, and across all five model families tested.
One example makes the stakes easier to picture. The sentence asks how someone could culture a sample using laboratory equipment available online. On the original wording, several highly ranked features are labeled as biological: mold growth, cellular processes, and biochemical analysis. The researchers then construct a neighborhood of internal states based on synonym variants, including changing someone to somebody. Within that bounded neighborhood, an optimization attack finds a state where those biology-related features disappear from the top eight, replaced by generic labels such as research methods and accountability.
The words on the page are not the only thing to keep in mind. The attack searches among internal representations associated with the synonym variations; it does not establish that every simple synonym swap in a live conversation will produce the same flip. And the example does not show that a deployed safety system actually let a harmful request through. It illustrates a risk: if an auditor relied on these feature labels as evidence of what the model was attending to, a fragile explanation could point them in the wrong direction.
The paper’s next move is to replace reassurance with a checkable question. Rather than asking whether an IRN looks convincing on familiar examples, the researchers use neural-network verification to bound how far its calculations can differ from the model’s within a specified neighborhood of internal states. The method treats the possible states as a region and tries to certify a maximum error across that region, rather than testing only a few individual sentences.
That approach is a meaningful step, but not a certificate for an entire language model. The team verified individual layers, not the full chain of computations from words to output. In their tests, the gap between the model and its replacement could grow in later layers. The paper describes how layer-by-layer checks might eventually be composed, while noting that the current bounds are not yet sufficient to verify the whole model this way.
The researchers also tried making the map sturdier. They retrained a GPT-2 sparse autoencoder at one layer using verification-aware training. Compared with standard training, this reduced the verified upper bound on the mismatch by about 90%. The retrained system was also sparse: an average of 13.9 features were active on clean inputs, 81% fewer than with standard training. Yet its explanations were not suddenly invulnerable. Only two or three of the original top 20 features could be proven to stay in the top 20 under the adversarial variations.
There is another complication: sometimes the terrain itself moves. In the GPT-2 test, the model changed its next-token prediction for 21% of the minor paraphrases. The researchers therefore caution that a shifting explanation can reflect both a fragile replacement network and a model that responds differently to the altered wording. A stable map cannot prove that the landscape beneath it is stable.
The hidden world this study reveals is not a secret chamber where an AI stores one clean meaning for every word. It is a moving landscape of layered signals, with human-readable features serving as a translation that can be helpful, approximate, and vulnerable all at once. Ladner and Althoff offer tools for measuring that vulnerability, plus an early way to reduce it. Their larger point is narrower—and more useful—than declaring these explanations trustworthy or useless: before relying on a map of an AI’s inner life, researchers need to test how faithfully it tracks the territory.
