The banana in the picture is yellow. Beside it sit brown donuts on a white plate. But the text accompanying the image insists the banana is brown. Asked to name its color, a vision-and-language AI answers: brown.
That sounds like a simple case of a machine failing to see what is right in front of it. But when researchers traced the model’s answer as it formed, they found something stranger: the model appeared to pick up the correct color, then changed course.
In a study of five multimodal AI models, Xingming Li and colleagues describe this as a late-layer textual override. Their finding opens an intriguing possibility: perhaps some AI systems do not need to be taught to see better so much as helped to hold on to what they have seen. The obstacle is knowing when to step in—and when not to.
A multimodal model processes an image and a text prompt through many computational layers before producing a response. The researchers took snapshots of the model’s evolving word predictions at different depths. In the banana example, the correct answer, “yellow,” rose to the top of the candidate list around layers 19–21. After layer 21, “brown” took the lead, while “yellow” fell to around rank 10. The image had supplied a useful signal; it was the final answer that lost track of it.
To see whether this was more than an odd banana story, the team tested models on questions where written context deliberately contradicted image evidence. They included five systems, from InstructBLIP to Qwen3-VL, and used two conflict benchmarks: Conflict-VQA, with 5,969 samples, and PhD-icc, a set of 16,844 yes-or-no question pairs.
The researchers compared how each model’s preference shifted as information moved through its layers. A late shift, by itself, did not mean the model was going wrong. Sometimes deeper processing improved an answer. The telling clue was the shift’s direction: 85% of failures moved toward the text’s answer, while 89% of correct answers in conflict cases shifted toward the visual answer. In ordinary, non-conflict cases, 91% shifted toward vision or stayed there.
That pattern suggested a targeted fix. Rather than always favoring an earlier prediction, the team’s method, called Conflict-Aware Layer Reference Decoding, checks whether a confident intermediate prediction fades before the final answer. If it does, the method blends some of that earlier signal back into the output. If the prediction holds steady, it largely leaves the answer alone.
The distinction matters. A blunt rule such as “trust the earlier layers” could undo useful reasoning along with harmful overrides. In the researchers’ tests, other methods that apply corrections more uniformly sometimes made performance worse. Their selective approach, by contrast, improved conflict resolution across the five models and three decoding strategies. On a conflict benchmark, its gain reached 9.4 percentage points. The researchers also tested ordinary image-question and captioning tasks, where the method largely preserved or improved performance rather than trading away general accuracy.
There is a practical appeal to the approach: it does not require retraining the models or adding outside knowledge. In one reported efficiency test on a single H100 GPU, it increased decoding time from 18.3 ms/token to 19.8 ms/token and used 80 MB of additional memory. That modest overhead points toward a tool that might be easier to try than a full model rebuild, though a benchmark result is not the same as reliable performance in everyday use.
And there is the catch. The technique does not independently confirm that an intermediate answer is true. It treats a confident early prediction that later weakens as a warning sign, not as proof. If the picture is ambiguous, or the text supplies relevant information the image cannot show, an intervention could preserve the wrong clue or suppress helpful context. The study’s deliberately constructed conflicts reveal a mechanism; they do not establish that a correction will help in every real-world situation.
The authors also leave important questions open. Their experiments do not settle how model size or training-data composition affects these reversals, and they have not shown whether the same pattern appears when audio and text disagree. Those are next steps, not conclusions.
Still, the banana points to a useful shift in perspective. When an AI gives a confidently wrong answer, the failure may not begin with missing perception. It may happen later, as the system weighs competing signals and lets one drown out another. If researchers can reliably spot that moment, future multimodal systems might become better at resisting misleading captions, labels, or instructions without ignoring language altogether. The frontier is not simply making AI see more. It is helping it keep its evidence in view—and learning when a second look is wiser than a correction.
