A scientific paper can be surrounded by readers and still be short of one crucial thing: useful criticism. Feedback helps researchers spot weak explanations, sharpen an argument, or notice what a reader cannot follow. But the chance to get that feedback is uneven. Some scientists have dense networks of experienced colleagues; others, especially early in their careers or working outside English-dominant research settings, may have fewer people to ask.
Could an AI system help fill that gap? To find out, researchers ran an unusually broad field experiment: instead of testing whether an AI could write a convincing review in a lab, they asked whether sending AI-generated comments to working scientists would change what those scientists did.
The team collected first-version preprints posted to arXiv between January and June 2024, then randomly assigned papers to receive feedback or to a control group. The analysis covered 31,020 papers and 45,466 authors across 150 fields and 133 geographic regions. Authors in the treatment group received an email with a private link to tailored comments. The control group received no intervention.
The comments were produced under a fixed protocol. A multi-agent system analyzed the paper, suggested possible titles, and checked grammar, using the manuscript’s content to shape its feedback. This was not a human peer review, and it did not test whether AI could certify that a scientific claim was true. The experiment asked a more immediate question: would the comments prompt authors to revisit their work?
Often enough to show up in the records. Within a month of receiving feedback, papers in the treatment group had an average of 0.044 revisions, compared with 0.039 in the control group. The researchers report this as a 12.55% relative increase over the baseline revision rate. The absolute difference was small: 0.005 revisions per paper. But across thousands of manuscripts, a small nudge can reveal a meaningful change in behavior.
The pattern was not evenly spread. The strongest revision effects appeared among researchers who might have had less access to established feedback networks. For papers from non-English-dominant environments, the relative increase was 19.9%. For papers with fewer references, a measure the researchers used to estimate how deeply a manuscript was connected to existing scholarship, it was 26.4%. The reported increases were also larger for authors with lower h-indexes and those earlier in their careers.
That result suggests a possible new role for AI in science: not replacing a colleague’s judgment, but making some form of critique easier to reach. A researcher does not need to know the right person, find space in a busy colleague’s schedule, or already belong to a well-connected group to receive comments on a draft. If the feedback is useful, a system that can deliver it at scale might widen the circle of people who get a chance to improve their work.
But a revision is not the same as a better paper. The study tracked whether authors posted a new version, not whether the revised manuscript was more accurate, more original, or more likely to lead to a discovery. An exploratory analysis suggested that the changes were not limited to spelling and formatting, and that substantive revisions may have increased. Still, the experiment does not establish that the AI’s suggestions were correct or that every change helped.
There is another wrinkle: authors who received feedback also showed a small increase in later use of language-model tools. Among authors with little detectable prior use, the study found a 5.29% relative increase in estimated adoption over the following year. But adoption was inferred using an AI-text detection model, not measured by directly watching which tools researchers used. The authors note that such detectors are noisy, so this result is evidence of a possible shift, not a precise count of tool use.
The experiment also compared substantive feedback with no contact. That makes it hard to completely separate the value of the comments from the simple attention of receiving an email about one’s paper. And because the intervention was a single, light-touch event, some recipients may never have opened the feedback. The study took place in early 2024, when AI tools were widely available but had not yet become routine in many research workflows. Its effects may not transfer unchanged to today’s scientific landscape.
Then comes the price of making critique abundant. When many researchers turn to similar systems, their feedback may start to sound alike. The authors warn that standardized recommendations could narrow the range of criticism, or displace some informal exchanges between human collaborators. In science, disagreement is not merely friction to be removed; it can expose assumptions that a polished, widely shared template might leave untouched.
The next frontier, then, is not simply to give every researcher an AI reviewer. It is to learn how to distribute helpful criticism without making scientific judgment more uniform. These results offer a starting point: in a randomized experiment, AI feedback was followed by more revisions, with larger effects among researchers who may have had fewer established sources of critique. Whether that turns into more reliable science depends on what happens next: who checks the comments, whose perspectives the systems reflect, and whether researchers keep room for the human feedback that challenges rather than merely smooths an idea.
