An A/B test is supposed to settle a question: Which message works better? But the answer often stops there. A winner gets picked, the next campaign begins, and the evidence gathered along the way rarely becomes a reliable guide to what should be tried next.
A new study suggests AI could help turn those isolated tests into a learning loop. Instead of asking a language model to invent persuasive messages from general knowledge, researchers gave an AI system the results of a large healthcare messaging experiment and asked it to find patterns, propose explanations, and design a second round of messages. In a trial involving 693,139 prescription-message invitations, the best message from that second round drew more clicks than the original baseline. The intriguing part is not simply that AI wrote a more successful text. It is that the system used one experiment to decide what to test in the next.
The work, by Junjie Luo, Ritu Agarwal, and Gordon Gao, began with a practical problem. When a prescription is written, a patient may receive a text from a healthcare provider’s office with a link to review the details. Will the patient open it? In the first experiment, researchers tested 13 message variations across 444,691 invitations. Behavioral experts worked with a chatbot to create the options, drawing on familiar ideas about persuasion, such as social proof, urgency, and authority.
Some of those ideas did not travel well into the clinic. In the first round, messages invoking social norms or gain framing performed worse than the standard message. The results offered more than a ranking: they gave the researchers a record of how actual recipients responded. But turning a huge experiment into useful design rules is hard. A spreadsheet can show which message got more clicks; it cannot, by itself, tell a team which pattern is robust, why it might have worked, or how to build a fresh message around it.
That is where the researchers’ system came in. It divided the job into stages, moving from data to patterns, from patterns to behavioral interpretations, and from those interpretations to new message designs. One agent checked the data; another used code to calculate statistical patterns; a third proposed explanations; and a fourth translated the findings into candidate texts. An orchestrator organized the work, while a human researcher reviewed and approved the analysis plan before it ran. The system was built to keep a trail from each proposed design back to the observations that inspired it.
In the second experiment, the AI produced 17 new message variants. Researchers tested them alongside three messages carried over from the first round, across 248,448 more invitations. The top performer reached a 69.8% click-through rate, 6.5 percentage points above the standard message tested in the same round. Four AI-generated messages significantly outperformed that baseline after the researchers corrected for multiple comparisons. The strongest also exceeded the best message from the earlier human-and-chatbot round.
But perhaps the study’s sharpest result came from a comparison that did not involve the experimental pipeline. Two language models, given the candidate messages but not the first experiment’s results, were asked to predict which ones people would choose. They failed to identify the eventual winner: one ranked it 15th, the other 14th. The models favored approaches such as direct commands and gentle urgency, while the data-backed system had learned that efficiency-focused wording and professional authority did better in this particular setting.
That contrast points to a different vision for AI. The promise is not that a general-purpose model can intuit what people will do in every situation. It is that an AI system equipped with local evidence and analytical tools might help organizations learn from their own past tests. A message that works for patients receiving prescription notices may not work for customers, voters, or patients facing a different decision. The study’s findings suggest that broad behavioral theories can miss these local differences—and that accumulated evidence may be more useful than confident-sounding guesses.
The researchers also found that principles commonly used in behavioral messaging did not all help here. Efficiency framing, completion framing, and professional authority emerged as promising; social proof did not. That does not establish a universal rule about how patients respond. It shows how conclusions that sound persuasive in the abstract can be tested against behavior in a specific context, then used to generate new ideas for another test.
There is an important price tag on the promise. The main outcome was whether someone clicked a link within 24 hours. The team also tracked whether people completed identity verification, a further step in the process. Neither measure shows whether a patient picked up a prescription, refilled it, took it, or experienced a health benefit. The authors call for future studies to follow those outcomes, and to examine possible costs such as stress, decision fatigue, or eroded trust. They also note that the work used SMS messages in one healthcare setting; the principles may not transfer even to other kinds of digital messaging.
The experiment also does not prove that an AI pipeline is better than a skilled team of human analysts given the same data. The authors say a direct comparison remains to be done. And while the system’s evidence trail makes its reasoning easier to inspect, human review still matters: people must decide which questions to ask, whether a message is appropriate, and what outcomes count as success.
The next frontier, then, may not be AI that replaces experimentation. It may be AI that helps experiments remember. If each test can shape the next one, organizations could spend less time rediscovering what their own evidence already shows—and more time checking whether those lessons hold. But in healthcare, a more clickable message is only the beginning of the question. The harder test is whether the added attention leads to better care, without steering patients in ways they did not choose.
