The most unsettling moment in a tutoring session may come after the lesson: a quiz asking whether a learner can tell a fact from a very plausible lie. In a recent experiment, people learned unfamiliar science from chatbots that sounded sure of themselves. Some of those assistants were quietly taught to give false information. The students often absorbed it—and did not reliably notice when their learning had gone off course.
The researchers recruited 32 participants and asked them to learn about five subjects, including GPS, carbon dating, fireflies, electrocardiograms, and noise-cancelling headphones. Each lesson came from an assistant built on the same language model, GPT-5-mini. Behind the scenes, the assistants were assigned different reliability levels: 0%, 25%, 50%, 75%, or 100% of the facts in their briefings were replaced with plausible false versions.
The setup made confidence part of the test. Assistants were instructed to present their information confidently, avoid hedging, and not recommend checking other sources. Participants were asked to rely only on the assistant. After each lesson, they faced a surprise quiz. Each question offered the correct fact, the planted falsehood, and two other plausible answers, then asked how confident they were.
The lies were designed to sound like ordinary explanations, not absurd traps. A false briefing might say carbon-14 has a half-life of about 50,000 years rather than about 5,730. Or it might claim that firefly light comes from glowing bacteria living inside the insect. The point was to see what a novice might remember after hearing a fluent explanation—not whether people could spot nonsense they already knew was wrong.
As false information increased, quiz performance fell. Across 104 quizzes, the average score dropped from 7.4 out of 10 when the assistant’s briefing was accurate to 2.9 when every briefing fact had been falsified. But the sharper finding was about individual claims: when an assistant actually stated a planted falsehood, participants chose it on roughly two-thirds of the relevant questions. By contrast, when the assistant had taught the fact correctly, participants selected that false option only 11% of the time.
In other words, learners were not simply drawn to the wrong answer. Hearing it from the assistant made a substantial difference. And the assistant’s overall reliability did not appear to protect people much once a particular false claim had reached them: the chance of accepting a stated falsehood stayed broadly similar across the different reliability levels.
Confidence made the result more troubling. Participants were nearly as sure of absorbed falsehoods as they were of facts they had learned correctly. Their confidence in the false answers averaged 8.4 on a 10-point scale; confidence in correct answers to truthfully taught facts averaged 8.9. When the assistant’s briefing was accurate, confidence helped distinguish right answers from wrong ones. As falsification rose, that distinction weakened toward chance. A mistaken fact could feel, to the learner, like knowledge.
Self-reported trust barely tracked the damage. Ratings of the assistants’ accuracy shifted only modestly as their information got worse, and participants’ willingness to rely on them again did not fall. That gap matters because people cannot always judge an unfamiliar scientific claim against knowledge they already possess. A student learning how GPS works may have no ready way to spot a confident error about satellite signals.
The experiment also exposed a trap in “checking.” About 30% of user turns involved pasting or retyping quiz questions into the chat. But the question was being sent back to the same source whose accuracy was in doubt. When participants challenged an assistant, it did not reverse its position in any of the 16 recorded challenges. Asking the same confident source again could therefore repeat the claim rather than independently verify it. Crucially, the study had instructed participants not to consult outside sources, so this pattern should not be mistaken for proof that learners would ignore other checks when those were available.
There is a real possibility on the other side of this problem. An AI tutor could make unfamiliar subjects easier to explore, offering explanations whenever a question arises. But access to an explanation is not the same as access to dependable knowledge. The researchers argue that a useful learning interface should make reliability visible claim by claim, help learners check information independently, and measure what they actually learn—not just whether they say they trust the assistant. Those design ideas still need to be tested.
The limits are important. This was a small, highly educated sample dominated by frequent AI users. The assistants were deliberately extreme: they delivered systematic falsehoods and were told never to hedge. The quiz came immediately after each lesson and used multiple-choice recognition, not delayed recall. The paper is a recent arXiv preprint, and its authors note that transcript coding also needs validation with human double-coding. The results do not tell us how often everyday AI tutors mislead students, or how other learners would respond.
Still, the study points to a larger design challenge: trust cannot be calibrated if errors are hard to see. A next-generation tutor may need to do more than sound helpful. It may need to show where each claim comes from, make uncertainty legible, and leave room for an independent check. Otherwise, the very fluency that makes an AI easy to learn from could make its mistakes unusually easy to carry away.
