Imagine an experiment that finds no effect. The new treatment did not help; the predicted link did not appear. That result can feel like a dead end, but in science it narrows the map: one plausible explanation has been ruled out. Now ask a machine to suggest the next study. Will it think to test whether anything happens at all?
That question sits at a crossroads between artificial intelligence and the study of science itself. AI systems are increasingly asked to propose research ideas, but their fluency is easy to mistake for insight. To test the difference, researchers turned to the people best placed to judge an idea’s fit: scientists whose own papers supplied the research puzzle.
The team assembled 121,640 recent preprints from biology, medicine, chemistry, and social science, then asked 26 AI systems to suggest follow-up hypotheses based on each paper’s context and question. The researchers kept the paper authors’ original hypotheses out of the prompts to reduce the chance that a model would simply echo the answer it was meant to extend. In the end, 6,749 scientists returned 25,139 sets of ratings, judging the suggestions for novelty, feasibility, likelihood of being true, and whether they would pursue them.
The most revealing gap was not between a brilliant idea and a foolish one. It was between a positive claim and its quieter counterpart: no effect. A null hypothesis asks whether two things are unrelated, whether a treatment makes no difference, or whether an expected pattern is absent. Such questions are essential to testing claims, yet every AI system in the study proposed null hypotheses less often than human scientists did. Even systems that could search the web rarely offered them.
That does not mean the machines never produce a null hypothesis. It means that, in this study’s prompts and across the tested systems, they reached for one less often than people did. Meanwhile, the models did differ in how widely they explored. Non-reasoning systems tended to generate ideas closer to one another; reasoning systems ranged across a broader set of possibilities. But broader exploration did not reliably include the possibility that the expected effect was absent.
The researchers point to a mismatch between what science needs and what its published record contains. Papers are more likely to report a positive association or a successful result than a failed attempt or an undetected effect. Researchers may have tried an experiment and found nothing, then never written it up. That “file drawer” leaves fewer examples of negative results for AI to learn from. The missing knowledge may be missing not because it is unimportant, but because it was never made visible in the text the systems learn from.
This is where the crossover matters: the tools of AI evaluation meet questions about how scientific communities decide what counts as knowledge. The study did not just compare machine-generated ideas with an abstract score. It asked scientists to judge concrete proposals in the context of their own work—and found that human judgment has its own patterns. Researchers tended to favor ideas that resembled their existing research. Those ideas seemed more feasible and more likely to be true, even as scientists rated them as less novel. Across the sample, likelihood of being true mattered more to adoption than novelty did. Scientists in social science were more receptive to risk than those in biology and medicine.
That portrait complicates any simple story in which humans are adventurous and machines are dull. People also lean toward familiar, likely-to-work ideas. But scientists can draw on laboratory experience and unrecorded failures; a model trained mostly on published text has a different kind of blind spot. In fields where evidence is contested and context matters, the researchers found, AI-generated ideas received lower ratings for novelty, feasibility, and likelihood than in the other fields studied.
The researchers also tested whether automated judges could reliably rate scientific ideas in place of experts. General-purpose AI judges tracked the scientists’ assessments only weakly; no model exceeded a correlation of 0.35 on any rating dimension. Popular automated measures of novelty performed near chance on comparisons between pairs of ideas. In other words, a system’s confidence that a proposal is original did not reliably tell the researchers whether the relevant scientists would see it that way.
There was a practical turn in the results. The team trained a reward model on scientists’ own ratings. Its performance beat leading comparison systems by up to 27%, with an average improvement of 14 percentage points across novelty, feasibility, and likelihood. It also exceeded a 61.0 ± 0.1% benchmark for agreement between independent human reviewers. That does not make the trained model a universal arbiter of good science. It shows that expert judgments can teach an AI some of the field-specific distinctions that generic metrics miss.
A broader analysis gave the researchers reason to look beyond individual prompts. They examined 38,863,847 papers published from 2010 to 2025 and found that, after ChatGPT’s release in November 2022, the share of abstracts containing no detected null claims rose in medicine and social science. Their analysis also suggested that scientific ideas initially spread out and later contracted in medicine. These are patterns in a large observational dataset, not proof that ChatGPT caused the changes. The authors’ simulations offer a possible mechanism: if many studies inherit the same AI-shaped blind spots, adding more studies may not cancel those shared errors out.
The finding is less a verdict on machines than a question about what we feed them and what we ask them to value. A system trained on published successes can help researchers search the known landscape, but it may struggle to notice the paths science has quietly abandoned. The researchers suggest that records of replications, registered reports, and failed experiments could help make those absences visible. The deeper challenge is to build tools that seek not only the most plausible next claim, but the result that could prove a favored claim wrong. In science, sometimes the most informative answer begins with no.
