Give a vision-language model a photograph and it may identify a dog, describe a street, or read a sign. But can it put a picture back together after it has been cut into pieces? That sounds like a child’s puzzle, not a serious test of technology. Yet assembling a jigsaw asks for something image description does not: every fragment must find its precise place, and each local fit must agree with the larger picture.
A team of researchers turned that deceptively simple challenge into a probe of machine vision. Their benchmark, called JigShape, asks models to arrange shuffled puzzle pieces across grids ranging from sixteen pieces to 256. The result is a mystery with an unexpected shape: some models can manage a small puzzle, but performance plunges as the pieces multiply. The question is why—and whether the models are seeing the same clues a human solver would.
The first complication was the test itself. Earlier benchmarks often cut images into rectangles. Imagine a landscape puzzle with several pieces of plain blue sky: more than one arrangement might look perfectly plausible. A model could assemble a visually reasonable picture and still be marked wrong because it did not reproduce the one arrangement chosen as the answer key. That makes it hard to tell whether the model failed or the test was ambiguous.
JigShape replaces those rectangles with interlocking edges. Each piece has tabs, matching blanks, or flat edges along the puzzle’s outer border. A tab must meet a blank; flat sides belong at the boundary. The shapes supply clues independent of the image’s colors and objects. The researchers also scrambled piece labels and randomly varied edge patterns, so a number or familiar-looking shape could not simply give away a piece’s position.
There was a second safeguard: the team used a conventional constraint-based solver to check the puzzles. It found the intended arrangement for all 500 held-out test instances at each of the four grid sizes, supporting the claim that every puzzle had a unique solution. The benchmark contains 95,468 puzzle instances generated from high-resolution images. For model evaluations, the researchers used a separate set of 250 puzzles at each grid size.
Now the test had a clearer answer key. On the smallest, four-by-four puzzles, GPT-5.5 placed 69.65% of pieces correctly and solved every piece in 26.40% of puzzles. The random piece-accuracy baseline was 6.25%. The other zero-shot model configurations tested—including versions prompted to reason more extensively—scored near chance on piece placement. The clue seemed to favor one model, but it did not yet explain what that model was doing differently.
So the researchers removed the jigsaw edges, keeping the source images and piece-label arrangements the same. Without tabs and blanks, GPT-5.5’s piece accuracy fell from 69.65% to 23.90%; its rate of solving an entire puzzle fell from 26.40% to 0.40%. It still did better than random, suggesting that visual content helped. But the comparison pointed to a stronger clue: geometric fit was doing much of the work.
Fine-tuning offered a tempting rival explanation. Perhaps the models simply needed practice. Two fine-tuned systems, Qwen3-VL-8B and Gemma3-12B, both reached more than 97% piece accuracy on four-by-four puzzles. That is a striking turnaround. Yet when the researchers took away the shaped edges, Qwen’s accuracy dropped from 97.28% to 10.00%; Gemma’s fell from 97.82% to 13.78%. The systems had learned the task, but their success leaned heavily on the easiest geometric clues.
Then came the scaling cliff. GPT-5.5’s piece accuracy fell from 69.65% on four-by-four puzzles to 4.37% on eight-by-eight and 0.67% on twelve-by-twelve. The fine-tuned models did better at eight-by-eight, but reached only 27.34% and 33.58% there, then dropped to roughly 3–5% at twelve-by-twelve. At sixteen-by-sixteen, their scores were near the random baseline. More training did not make the difficulty disappear.
One clue exposed where the problem lay. The fine-tuned systems were much better at placing corners, whose two flat edges identify their boundary positions, than interior pieces, which have no flat sides. On twelve-by-twelve puzzles, Gemma’s accuracy was 39.70% for corners but just 1.78% for interior pieces. The models could use an obvious local cue; they struggled when success required maintaining a web of compatible relationships across the board.
A further test asked whether hints could break the deadlock. In one eight-by-eight setup, the model was given the correct locations of 60 of the 64 pieces and asked to place the four that remained. GPT-5.5 got 25% piece accuracy on those remaining pieces. The researchers report that it identified the right two-by-two region but swapped the exact positions. Knowing roughly where to look was not enough to resolve the fine spatial arrangement.
This contrast with the classical solver sharpens the puzzle. Using exact edge information and a backtracking search, that solver completed all 500 held-out instances at each grid size. So the generated puzzles were solvable by a method designed to apply explicit constraints. JigShape does not show that machines cannot solve jigsaws, or that every vision-language model fails at every spatial task. It shows that these tested models, asked to infer the arrangement from images, did not reliably keep many local constraints consistent as complexity grew.
The researchers’ benchmark is a controlled test, not a verdict on how well these systems would handle every real-world scene. Its pieces keep their original orientation, and the evaluation measures performance on a limited set of examples at each size. Still, the contrast between a vivid small-puzzle success and near-chance performance on larger grids makes the central question hard to ignore: recognizing what is in an image is not the same as reasoning about where every part must go. In this puzzle, the missing skill was not seeing the pieces. It was making them all fit.
