Imagine slipping a word into a writer’s instructions, then asking for a story while forbidding any mention of it. The word is violin. The writer can’t name it, refer to it, or even hint at it. Yet the story might still fill with warm wood, strings, and a bow drawn across them. The word has vanished from the page. Its associations have not.
That is the hidden channel explored in a recent arXiv study. Researchers gave language models secret words and asked them to write stories, jokes, or essays. They checked that the forbidden word never appeared literally in the text. Then they asked another model to look for the secret—not by searching for the word, but by noticing what the writing seemed to be about.
The test was deliberately simple. A model saw two stories and was asked which one had been written with a particular secret in mind. If it had no useful clue, it should choose correctly half the time. This paired comparison can catch a faint thematic trace that a more demanding challenge might miss: naming the exact secret from an enormous vocabulary.
For several models, the trace was hard to miss. When instructed not to reveal a word, Llama 4 identified the matching story 79.3% of the time; Claude Opus 4.6 reached 78.1%, DeepSeek V3.2 reached 77.6%, and Claude Sonnet 4.6 reached 70.5%. Those scores don’t mean the models openly disclosed their secrets. They mean the secret helped steer choices—what scene to set, which images to use, what kind of story to tell.
The result held up when the reader and writer were different models. In one test, Opus read DeepSeek’s stories and matched their secret at 86.7%. The signal, then, was not just a writer recognizing its own habits. Something about the text itself carried information that another model could pick up.
But the strangest finding came when researchers told the models to try harder to hide the word. Most did not become neutral. Their writing veered away from the secret, and that detour could be detected too. Given the secret patience, one model wrote about a furious day trader smashing a plate and speeding down the road. A reader who knows the model is avoiding patience can treat that conspicuous distance as a clue.
In this test, below-chance accuracy wasn’t automatically evidence of successful concealment. A model choosing the wrong story more often than random could mean its writing had been pushed away from the secret. Reverse the reader’s choice, and much of the signal reappears. The model may have hidden the word, but its effort to avoid the idea left a different kind of footprint.
Whether a footprint appears depends on how much freedom the task gives the writer. Short jokes—averaging about 12 words—carried almost no detectable signal. Several models repeated the same familiar joke regardless of the secret. A story, by contrast, averaged about 450 words and asked the model to make many unprescribed decisions. Long-form jokes, with 3–4 paragraphs of setup and a similar average length, also leaked for some models. Five-paragraph essays were mixed: only Llama showed a clear signal in the reported comparison.
Model size mattered in the two families the researchers tested. Llama models at 3B and 8B parameters showed no detectable leakage in this experiment, while the 70B model reached 84%. In the Gemma 3 family, the 4B model was marginal at 59%, while the 12B and 27B models reached 83–84%. These results describe those tested models and tasks, not a law that every larger model will leak more.
The researchers also tried giving the writer a decoy word to focus on instead. For Llama, this nearly redirected the trace: readers detected the real secret at 51%—about chance—but detected the decoy at 71%. DeepSeek and Opus showed partial redirection; Sonnet was the exception to the general pattern. The trick suggests that attention can shift what colors the output, but it is not proof that a decoy can protect realistic confidential information.
Why would a word the model is trying to suppress still shape its writing? The authors propose a possible explanation, not a demonstrated account of the model’s inner workings: to avoid a concept, the system must attend to it. Meanwhile, open-ended writing offers a hundred small choices. If the secret is among the strongest ideas in the prompt, it may tip those choices toward related settings and images. A command to avoid a thought can keep that thought active in the process of avoiding it.
There are important boundaries around the finding. The secrets were single English words, the tasks were creative writing, and detection was tested model-to-model. The researchers did not formally measure whether human readers could spot the same traces. Real confidential information is often longer, more contextual, or used in more constrained tasks, which could change the result. The study shows a vulnerability in a controlled setup—not that every hidden instruction or private document will surface in a story.
Still, the experiment shifts the question of secrecy. It is not only whether a system prints the forbidden word. Information can travel in the choices around it: a setting selected, an image repeated, a topic avoided too emphatically. A model may keep the word off the page and leave its outline there instead.
