Imagine a language model that never gets to finish its sentence. It offers a next-word prediction, then receives a blended version of its own possibilities as input and tries again. The result is not an ordinary chat. It is a loop: each turn nudges the next, and the question is whether those nudges settle into a pattern—or spiral into something unpredictable.
In a recent arXiv study, researcher Timothy D. Jones built just such a loop. Instead of choosing one next token, the model fed back an average of the embedding vectors for all its possible next tokens, weighted by their probabilities. That averaged vector is not a word a person would read. It is a point in the model’s internal numerical landscape. The prompt and the place where this soft token entered the model stayed fixed, turning each model into a system that could be studied step by step.
That change of viewpoint matters. In ordinary text generation, the model adds each chosen token to a growing conversation. Here, the prompt stays put, and one continuous state is repeatedly replaced. For the detailed experiments, the state lived in 896 dimensions inside Qwen2.5-0.5B-Instruct. Jones also scanned 15 released models, from 0.5 billion to 72 billion parameters, to see whether the phenomenon appeared beyond that one small system.
Two controls helped reveal the hidden terrain. Temperature adjusts how broadly the model spreads probability across possible next tokens. Activation steering adds a chosen direction to the model’s internal activity—in this case, a direction built by contrasting philosophical sentences with neutral ones. Adjust those controls, and the loop’s behavior can change. In maps of the settings, regions of repeating text sat beside regions where nearby versions of the loop diverged rapidly.
Some regions settled into short verbal orbits. One loop repeatedly produced “is it”; another cycled through “The existence.” These phrases were not evidence that the model had settled on a meaningful thought. They were the readable traces of a deeper pattern: the internal state had returned to the same points in the same order.
Elsewhere, the loop behaved chaotically. In the language of dynamical systems, that means tiny changes in the starting point can grow over time, making later states hard to predict. The team’s test did not label a region chaotic just because its text looked strange: it used a mathematical measure of how quickly nearby trajectories separated. In one scanned region of Qwen2.5-0.5B-Instruct, 30% of the tested settings met the study’s finite-horizon chaos criterion; 74% of a random sample of those settings remained chaotic for 100,000 steps. Those figures describe a particular experimental map, not language models in everyday use.
The most revealing surprise was how familiar the map looked to researchers who study much simpler systems. The loop’s chaotic areas contained narrow windows of orderly repetition. As conditions changed, some repeating cycles doubled in length, then doubled again on the path toward chaos. The ratios between successive changes approached Feigenbaum’s constant, a hallmark of period-doubling in nonlinear systems. The study also found “shrimp-shaped” regions—named for their outlines in parameter maps—where repeating cycles held on amid surrounding chaos.
That resemblance was more than a visual analogy. The model’s own derivatives—measurements of how its internal state responds to a small nudge—let the researchers predict how long a loop would linger near a repeating pattern after that pattern had disappeared. Across two stages of prospective tests, their predictions came within one iteration of the measured transition time in 42 of 48 cases. Interval-arithmetic calculations also proved that selected repeating orbits exist and are locally attracting in the full transformer map. The researchers did not prove that the chaotic trajectories persist forever; those findings rely on finite-duration measurements.
Why would a word-prediction system produce such a landscape? A clue lies at the moment when several next tokens are plausible at once. The feedback vector blends their embeddings. If a small change shifts the balance among competing tokens, it can move the next input; the transformer processes that movement, and the next blend may shift again. In the detailed Qwen experiment, the analysis found that steering increased growth mainly through the output stage where next-token options compete. The model’s hidden dynamics mattered too, but the readout could magnify the effect.
This sensitivity has a practical cousin. In a separate test using a continuous-feedback reasoning method called Soft Thinking, Qwen2.5-7B gave different answers to 21% of 256 arithmetic problems when prompts were processed in batches of 16 rather than 8. When both runs were supplied with identical feedback vectors, the share of different answers fell to 6%. Ordinary greedy chain-of-thought decoding changed 2.3% of its answers in the comparison. But the Soft Thinking pipeline in this test answered only 22% to 30% of the problems correctly, compared with 89% for ordinary greedy decoding. The result is a warning about stability in this particular setup—not evidence that ordinary chat is chaotic, or that instability alone explains accuracy.
The deeper shift is one of scale. A language model’s visible answer is a string, but continuous feedback reveals a moving landscape underneath: routes that repeat, boundaries where patterns vanish, and regions where a rounding-sized nudge can grow. Studying that landscape does not turn text generation into a simple physics problem. It gives researchers a new set of instruments for asking when a model’s internal loop settles down, when it becomes sensitive, and how those hidden dynamics might matter in systems designed to reason through repeated feedback.
