Imagine reading a page of ordinary prose while an invisible marker quietly changes behind every word. It might linger for a while, then move to one of two neighboring positions on a hidden track. Nothing in the text announces the switch. Yet a language model trained on enough pages might learn not only to guess where the marker is, but to sense the shape of the track.
That is the unusual experiment at the heart of a paper posted to arXiv on August 27, 2026. Researcher Alexandru-Iulius Jerpelea set out to test whether an AI’s internal representations can reflect the hidden patterns that generate text. The result offers a glimpse of a world beneath the words: a model can pick up a deliberately planted sequence of hidden states, and the way those states tend to follow one another can leave a geometric imprint inside the model.
The key was to make the text feel less like a toy puzzle. Earlier experiments could give a small transformer sequences produced by a carefully specified mathematical process. Those studies made it easier to know what hidden pattern a model should learn, but the sequences were not ordinary language. Here, a larger language model wrote natural-looking text, while the researcher quietly nudged its internal activity in one of eight selected directions.
Those directions were extracted from the teacher model’s internal signals using a tool called a sparse autoencoder. They were chosen to be largely unrelated and not to activate together on ordinary text. Their individual meanings were not the point. Instead, the researcher imposed a rule for how the active direction changed: it could stay put, or move to a neighboring direction on a ring of eight states. It stayed in place with probability 0.95, lingering for about 20 tokens before typically hopping to a neighbor.
The teacher generated 400,000 documents, each 256 tokens long, for a corpus of about 102 million tokens. The state path was set before each document was written, and the steering was applied as the teacher produced the text. The result was prose with a hidden pattern threaded through it, not a visible label attached to each sentence. Then a smaller, 110-million-parameter transformer was trained from scratch on the text. It had no direct access to the teacher’s internal directions or to the secret state labels.
Could the student reconstruct where it was on the ring? To check, the researcher compared its internal activations with an idealized observer: a mathematical reader that knew the hidden transition rules and the teacher’s token probabilities, then updated its estimate after each token. This observer’s running estimate is called a belief state. It does not mean the model believes something in a human sense; it is a probability distribution over which hidden state is currently active.
A simple probe—a statistical tool for reading information from a model’s internal activity—found that the student’s activations carried a partial version of that estimate. At the best layer, the probe reached an R2 of 0.49 and identified the actual state with 0.577 accuracy. The ideal observer’s accuracy ceiling was 0.762. The student therefore captured a substantial, but incomplete, part of the hidden-state information, despite learning only from the generated text.
The deeper surprise appeared when the researcher asked not just whether the model could estimate the state, but how it arranged the eight states internally. At first glance, the answer seemed obvious: take a two-dimensional view of the model’s activations, and the states appeared in ring order. But that apparent circle had a trap inside it. When the hidden state switches, the model may briefly retain traces of the previous state. Because each state’s neighbors are also its possible predecessors, that short-term memory alone could pull the states into a ring-shaped pattern.
So the researcher narrowed the test to each document’s first stretch, before the hidden state had switched even once. The clean circle vanished from the most prominent two-dimensional view. But that did not settle the question: a real pattern might be present in a less obvious plane. A targeted analysis searched for a plane in which the eight state representations formed a circle in the exact order dictated by the hidden transitions.
In that more focused test, the circular pattern accounted for 38% of the state-centroid variance at one layer. The researchers also tested every possible ordering of eight states around a ring: 2,520 arrangements in all. The planted order ranked first for the student trained on ring-structured text. It did not stand out in either comparison model, including one trained on text where states could jump randomly rather than move to neighbors.
The finding suggests that an AI’s internal geometry may record not only what different states are like, but how they tend to unfold over time. The ring was not a picture of eight meanings. The directions had been selected to be unrelated; what linked them was the transition rule imposed on the text. In this experiment, statistical neighborhood—not semantic similarity—helped shape the neighborhood inside the model.
That is a revealing result, not a map of how every language model understands language. The hidden signal was deliberately inserted, the text came from one teacher model, and the circular geometry was not the dominant structure in the student’s internal activity. The planted variable supplied only about 0.05–0.1 nats of evidence per token, and the evidence for its geometry was correlational, not proof that the transition rule caused the pattern. The author also notes that the eight directions may overlap with other features the teacher model uses.
Still, the experiment opens a useful door. If researchers can plant a known hidden pattern in realistic text and then find its faint trace in a model, they can study how learning turns sequences into structure. The next challenge is to look for patterns that were not planted: perhaps the hidden rhythms of syntax or other real features of language. For now, the ring is a controlled clue—a small, carefully placed signal that reveals how much of a model’s unseen world can be shaped by the way information arrives.
