Three doors. Behind them: a car, an ordinary sheep, and a very special sheep—a billionaire’s lost pet. You pick a door, hoping to win the golden sheep. The host, who knows what’s behind each door but cannot tell the sheep apart, opens another door to reveal a sheep. It isn’t the golden one. Now you can keep your first pick or switch to the last closed door. Which move gives you the better chance?
This is not quite the famous Monty Hall problem, though it wears a similar disguise. The twist changes the answer. Here, the researchers who posed the puzzle say you should not switch. A set of recent tests suggests that this kind of subtle twist can trip up language models that otherwise perform almost flawlessly on ordinary probability questions.
The puzzle’s key is not the number of doors left. It is what the host’s reveal tells you about your first choice. Assume your original pick was made at random, so each door was equally likely. If you picked the golden sheep, the host had to reveal the ordinary sheep: it was the only other sheep available. If you picked the car, the host could reveal either sheep, with equal chance. If you picked the ordinary sheep, the host would have to reveal the golden one, so that case doesn’t fit what you saw.
That means the reveal is twice as compatible with your having picked the golden sheep as with your having picked the car. Staying is the better bet. In the classic Monty Hall setup, the host’s rules make switching the winning move. Change the prize and the host’s knowledge, and the familiar answer flips. The game is a small but sharp test of whether someone is following the actual rules—or reaching for a remembered pattern.
Luca Avena, Gianmarco Bet, and Bernardo Busoni used this puzzle and others like it to probe a gap in AI performance. They tested 16 configurations of language models, including versions with and without chain-of-thought reasoning, on two collections: 50 standard probability exercises and 20 problems designed to pull intuition away from the correct answer. These included puzzles involving paradoxes, misleading patterns, and tempting but faulty shortcuts.
On the standard exercises, the models averaged 0.96 accuracy; nine of the 16 configurations scored above 99%. On the counterintuitive set, the average fell to 0.59. The best result there came from ChatGPT 5.4 Thinking, at 0.84—not a failure, but a striking drop beside near-perfect scores on the routine problems.
That contrast makes the study’s central puzzle harder to shrug off. The counterintuitive questions were not necessarily more demanding calculations. Instead, they were designed so an intuitive answer could diverge from the mathematically correct one. The results suggest that success on familiar or technically difficult math tests may not reveal how reliably a model handles probability when a problem’s structure pushes against an easy instinct.
The researchers looked for another clue: what happens when a problem keeps its logic but changes its clothes? They compared well-known probability puzzles with reworded versions that preserved the underlying structure. On the familiar versions, models generally did very well. On the altered versions, performance dropped by more than 20% on average. In one telling case, several models gave the classic “switch” answer to the golden-sheep puzzle—even though the changed rules made “don’t switch” correct.
The authors call this pattern evidence of “token bias”: a model may benefit when a problem’s wording resembles a familiar formulation, then struggle to carry the same reasoning across a new story. But they also flag a limit: the disguises did not always fool the models. Some recognized the source puzzle anyway. So the score drop is a clue about the role of familiar wording, not a precise measure of how much a model is reasoning versus recalling.
Then came a more unsettling test. The researchers added a wrong answer to the prompt, framed as a user’s opinion with a justification. A bare incorrect number reduced relative performance by 11%. An invented, fallacious explanation reduced it by 7%. The strongest nudge was not written by the researchers at all: it was a plausible but wrong explanation previously produced by another model. With that hint, average relative performance fell by 34%.
That result does not show that models simply agree with everything a user says. It does show that, in these tests, even a convincing wrong argument could disrupt answers—and a model-generated mistake was especially influential. Chain-of-thought prompting appeared to help on some counterintuitive questions, but did not protect against this effect. A long explanation, then, was no guarantee that a model had resisted a bad clue.
The study has boundaries. It examines discrete probability problems, not every kind of reasoning; models were tested on a limited set of questions, and each problem-model pairing was run four times. The teams also used different scoring approaches for standard and counterintuitive exercises, with human review included for the standard set. The findings cannot settle what language models “really” understand. They do show that their performance can change sharply with a puzzle’s framing and with the reasoning placed in front of them.
Back at the doors, the answer depends on a detail that is easy to overlook: the host cannot distinguish the golden sheep from the ordinary one, and opens randomly among the doors hiding sheep. Alter that rule and the odds may change again. That is the point of the puzzle. In probability, a familiar-looking setup is not the same as a familiar problem. And for AI—as for anyone playing along—the first step is to check exactly what happened before deciding what the odds say.
