Imagine being handed a handful of unfamiliar words, a few translations, and a blank space where the next answer belongs. There is no dictionary to consult and no grammar book to open. The only way forward is to work out the rules from the examples: which sounds change, how a sentence is assembled, what a verb ending means.
That is the sort of puzzle set for contestants at the International Linguistics Olympiad. It also makes an unusually sharp test for artificial intelligence. A model might seem to solve a problem because it recognizes facts it encountered before. But what happens when the puzzle itself has never been published?
A study posted to arXiv in August 2026 took up that question using five problems from that year’s Olympiad. They covered sound patterns in Central Alaskan Yup’ik, color terms in Yélî Dnye, syntax in Iquito, kinship words in Sakurabiat, and verb structure in Komnzo. The tasks included translating sentences, filling blanks, and matching phrases. The questions were kept from participants until the contest, so the models could not have memorized their exact solutions.
The researchers invited teams to submit systems under a tight budget: one T4 GPU and 30 minutes. The open challenge drew 731 submissions from 46 teams. At first glance, one result seemed to settle the puzzle in favor of clever engineering. The winning entry used a 14-billion-parameter model and scored 19.79 on the automatic measure, more than twice the strongest baseline using that same model, which scored 9.40. Nine of the top ten submissions used a quantized 14B Qwen model.
So what had changed? Not the size of the model. The strongest teams adjusted how it generated and formatted answers. In within-team comparisons, self-consistency voting—sampling different answers and selecting a common one—was associated with a 2.08-point gain, while retrying or reformatting outputs added 1.40 points. Chain-of-thought prompting, which asks a model to spell out intermediate reasoning, was associated with a 1.55-point drop; none of the top ten used it. These comparisons are clues, not controlled proof that any one technique caused the difference. Still, they unsettle a simple assumption: asking a model to show more reasoning does not automatically help it solve a rule-finding problem.
Then came the harder clue. The automatic leaderboard compared answers with reference answers using text-based measures. The researchers also asked members of the official Olympiad jury to grade a selection of AI solutions using rubrics similar to those applied to human contestants. That changed the picture dramatically. The automatic system ranked the five evaluated submissions in the same order as the jury, but the scores told different stories.
The challenge winner’s automatic score was 19.8; the jury awarded it 5.9. The entry landed in the bottom 5% of contestants by the human assessment. Meanwhile, the frontier model Claude Opus 4.8 earned a jury score of 79.5, above that year’s gold-medal cutoff of 70.0 and equivalent to fourth place among 255 contestants. Gemini 3.6 Flash reached a silver-medal-equivalent score of 60.5. The gap between the two kinds of grading was not just a technical nuisance: matching strings could reward an answer that missed the underlying rule, while a correct explanation written in prose could earn human credit that the automatic metric missed.
But perhaps the frontier models had simply recognized the languages. Some had public documentation, however limited, and a model could carry scraps of linguistic knowledge from its training. The researchers probed models on language facts and vocabulary, then removed language names and background information from some prompts. Knowledge-probe accuracy had a positive but modest association with overall task scores: Pearson’s correlation was 0.38 and Spearman’s was 0.43. That does not show that prior knowledge caused better performance. When context was removed, the average score change was not statistically significant across the tested models and problems. One exception offered a useful clue: on Yélî Dnye, six of seven models lost an average of 5.8 percentage points. The language name itself gave a hint for one item, because “dnye” means “language.”
The evidence does not say that familiarity was irrelevant. It says familiarity alone did not explain who solved the puzzles. The problems were designed so solvers had to infer patterns from examples and apply them to new forms. Models could know something about a language and still struggle to generalize its rules; the researchers also found no reliable link between a language’s public presence and how easy the corresponding puzzle was for models.
The jury’s written comments revealed another divide. Strong models could identify linguistic structure, but their explanations often worked assignment by assignment instead of building one compact account that covered the whole dataset. They sometimes invented details, missed broad patterns, or offered elaborate analyses that did not fit the examples. In one Komnzo problem, none of the models identified a particular marker used in non-recent-past sentences. The issue was not simply whether a model could produce a plausible translation; it was whether it could assemble a theory that held together.
That distinction matters because the experiment was small: it used one year’s five problems, and only a subset of AI systems received expert grading. The authors caution that the results should not be treated as a final verdict on reasoning in general. The proprietary systems that reached medal level also remain difficult for outside researchers to inspect, while the resource-constrained open submissions fell far behind.
So, can AI discover a language’s rules without seeing the puzzle? In this test, some frontier systems did well enough to earn human judges’ medal-level scores. But the clearest answer is narrower and more interesting: an unreleased puzzle can expose a gap between producing plausible answers and building a reliable explanation. The next clue is not just whether AI gets the words right. It is whether it can make the rule that links them unmistakable.
