Give a model a task with two ways to get the answer: a real rule that takes work to learn, and a simple clue that usually points in the right direction. The obvious worry is that the model will seize on the clue and never learn the rule. That is exactly what happened in one set of experiments—but the researchers also found a twist: for some models, making the clue more reliable helped them eventually move beyond it.
The task was deliberately simple. Each example was a sequence of five integers, and the model had to report whether their sum was even or odd. But the training data offered a shortcut: the parity of the largest number in the sequence. In some training sets, that shortcut matched the correct answer half the time. In others, it matched 90 percent of the time.
To see whether a model had learned the rule or merely memorized the clue, the researchers tested it on a split where the shortcut always pointed to the wrong answer. A model relying on the largest number would fail every time. One that had learned to combine the numbers and calculate the sum could succeed.
The outcome depended on the model’s capacity. In the main comparison, a two-layer transformer with two attention heads reached perfect accuracy on that adversarial test in 23 of 30 training runs when the shortcut matched the answer 90 percent of the time. At a 50 percent match rate, none of those 30 runs did. The same increase in bias did not rescue the smaller models: a one-layer model could become more firmly stuck on the shortcut.
That is the unexpected result—not that bias is good, but that the training signal can change as a model learns. At first, the shortcut works. The model finds the largest number, uses it to predict the label, and gets most common examples right. But once it is confidently correct on those examples, they have less influence on its next updates. The rare cases where the shortcut fails remain mistakes, so they continue to push the model to change.
In the 90-percent setting, those counterexamples are outnumbered nine to one. Yet the researchers found that, in the larger models, their measured gradient signal could outweigh the signal from shortcut-consistent examples. The paper’s mechanistic analysis suggests a sequence: the model first builds a shortcut-based circuit, then, in some runs, reorganizes its internal processing so it can represent the parity of the full sum. That explanation fits the evidence, but the authors describe the proposed pathway as a mechanism consistent with their observations—not a settled, experimentally proven cause.
The researchers tracked this shift by examining what the transformer’s attention heads appeared to do. Early on, one pattern favored the maximum-valued number. In successful runs, that pattern weakened as a representation associated with parity strengthened. The change was not simply “more training fixes it”: models trained for as many as 15,000 epochs could remain trapped. Nor did every run that showed signs of a more robust internal circuit reach perfect adversarial accuracy.
The result also depended on what counted as a shortcut. Across four synthetic tasks, the researchers tested shortcuts based on either the largest number or the first number, with answers that could be divided into two or three categories. In the three-category tasks, the striking pattern was not that one particular kind of imbalance always helped. Instead, two-layer models often did better when the shortcut was unusually likely to agree with the answer or unusually likely to disagree. Near the chance-level match rate, generalization was often weakest. The size of the effect varied, however, and on one task the deeper models did worse than a one-layer version.
That variation matters. It blocks an easy but misleading takeaway: researchers should not deliberately make training data more biased. The experiments used small transformers and invented number puzzles, not photographs, medical records, or language. The team also compared different data ratios and model architectures; they did not test whether specific debiasing methods, such as reweighting or subsampling examples, would help or hurt. Whether this dynamic appears in practical AI systems remains an open question.
Still, the work complicates a familiar prescription. When a model learns the wrong cue, balancing the training examples can seem like an obvious remedy. These experiments suggest that, in a narrow set of conditions, removing the imbalance may also remove a useful pressure: the model first becomes so good at the shortcut that the exceptions stand out in its remaining learning signal. But the shortcut can just as easily become a trap, especially when the model lacks the capacity to build a better solution.
The deeper lesson is about how models learn, not how to engineer a lopsided dataset. A training set is more than a pile of examples; its structure shapes which signals keep influencing a model as learning unfolds. Sometimes the cases that matter most are not the majority. Here, a few stubborn counterexamples could become the strongest voices in the room—but only when the model was capable of listening.
