The final whistle brought a surprise before the celebrations: Spain had beaten Argentina 1–0, and only one of ten AI assistants had picked Spain to win the tournament. That forecast belonged to GPT-5.5 Thinking, which finished at the top of a World Cup prediction leaderboard. But the model that scored highest across the tournament was not the one that most often predicted the result of an individual match. That mismatch is the story—and a clue to how difficult it is to measure what an AI system can actually do.
In a preprint posted to arXiv, Jonaid Shianifar and Iias Faiud audit a benchmark in which ten assistants submitted a single forecast before the 2026 FIFA World Cup. Each received the same tournament snapshot, prompt, and response format. They had to predict group-stage scores and standings, then fill in a complete knockout bracket and name the eventual champion. The tournament offered plenty of room for a forecast to go wrong: 48 teams, 104 fixtures, and a knockout phase where one unexpected result can rearrange the route to the final.
The leaderboard’s winner earned 744 points. GPT-5.5 Thinking had picked Spain as champion and mapped a strong path through the knockout rounds. Yet its group-stage outcome predictions were correct in 42 of 72 matches, or 58.33%. Claude Sonnet 4.6 got 46 right, the best result on that measure, but finished sixth overall. Qwen 3.7 and Perplexity Pro each had the highest group-stage match score and the most exact score predictions. Ask which assistant was “best,” then, and the answer changes with the question.
The benchmark combined several kinds of success in one total. It awarded points for match outcomes and exact scores, for predicting group standings, and for placing teams in the right knockout rounds. Those knockout rewards accumulated: a team forecast to reach the final could earn points for its earlier rounds too. That makes sense if the aim is to reward a coherent forecast of the whole tournament. But it also means an early bracket mistake can throw off later predictions, while a strong run through the bracket can create a large advantage.
That is what happened here. Group-stage match scores ran from 246 to 270 points, and group-standing scores from 221 to 264. Knockout scores ranged from zero to 242. The researchers found that the knockout component was closely associated with the final ranking, while the group-stage components had little relationship to it. GPT-5.5 Thinking moved from eighth place on match points to first once knockout performance counted. The leaderboard did not simply discover the best match picker; it favored the assistant whose forecast worked best as a tournament-long path.
There is a real capability worth testing in that longer view. A forecast of a whole tournament asks a system to keep its predictions consistent across many dependent events. It is different from asking who will win one match at a time. In other settings, long-range forecasts could help people compare possible outcomes or plan around uncertainty. A useful AI forecaster would need to do more than produce a confident-sounding answer: it would need to say how likely different outcomes are, and show how its view changes as new evidence arrives.
But this experiment cannot tell us whether the winning bracket reflects a general forecasting skill. There were only ten submissions, and each assistant submitted one forecast. The authors note that a different response from the same system could have changed the result. They also had no bookmaker, statistical model, or human forecaster in the original leaderboard for comparison. The contest therefore ranks the submitted assistants against one another—not against the strongest alternatives, and not across repeated trials.
Confidence was another revealing weak spot. The models reported average confidence between 60.64% and 73.27%, but those self-reported figures showed almost no relationship to either match accuracy or total points. That does not prove the systems are incapable of estimating uncertainty: each model gave a confidence value for its chosen scoreline, not probabilities across the possible outcomes. Still, it shows why a system’s statement that it is “sure” should not be mistaken for a calibrated forecast.
The next step is less cinematic than asking an AI to name a champion, but more informative. The authors recommend collecting repeated forecasts, asking for probabilities across outcomes, and scoring those probabilities with methods designed to reward reliable uncertainty estimates. They also argue that researchers should report match predictions, group standings, and knockout forecasts separately before combining them. A single score can be convenient; when its components vary so much, it can quietly decide which skill appears to matter.
That is the price of a simple leaderboard: clarity can come at the cost of precision. A tournament-wide score may capture the challenge of keeping a long forecast coherent, but it can obscure whether a system is good at predicting matches, groups, or the bracket’s final shape. The authors’ own conclusion is appropriately narrow: under this scoring system, one submission produced the strongest realised tournament path. That is an intriguing result, not proof that it understood football better than the others.
For AI forecasting to become useful beyond a sporting contest, the evaluation has to be as carefully designed as the forecast itself. The World Cup audit offers a small but vivid demonstration: change what the scoreboard rewards, and the apparent winner can change too. Before asking an AI to guide decisions under uncertainty, we need to know whether it can forecast well—or merely excel at the particular test we built.
