A program can produce code that runs, generate a tidy result, and still miss the point of the scientific task it was given. That gap is the puzzle at the heart of a new test of AI agents: when does an assistant stop merely completing steps and start doing something like science?
To investigate, researchers built SciAgentArena, a benchmark of about 200 tasks across five biomedical fields: drug discovery, single-cell and spatial biology, electronic health records, and genetics. They tested 18 AI agents, from general-purpose assistants to specialized systems. The tasks ranged from routine data analysis to optimization, discovery, and a deceptively difficult challenge: deciding whether a request is scientifically valid in the first place.
The first clue seems encouraging. Agents often handled well-defined workflows—especially when the method, tools, and success criteria were clear. Some could load data, run familiar analyses, and produce outputs that matched expert reference solutions. In one single-cell biology task, Claude Code, ToolUniverse, and a version of STELLA could complete all the tested steps in a preprocessing pipeline.
But the same work exposed a trap: a successful run is not necessarily a successful analysis. In one example, an agent produced executable code for modifying biological labels, yet simply reused labels it had been asked to change. The code did not crash, so the agent never triggered its own debugging. The software had passed a basic test—does it run?—while failing the more important one: did it do what the scientific question required?
That distinction sharpened when researchers stretched tasks into longer pipelines. Steps that agents could handle separately became harder when each result had to feed the next. In spatial biology, agents had to construct a neighborhood graph that respected a specific symmetry requirement. Only three systems—GPT-5.2, STELLA with memory, and Biomni—got that requirement right. Others either encountered runtime errors or overlooked it despite hints in the instructions.
Perhaps, then, the obstacle was simply coding: give agents better tools and the science would follow. But another clue points beyond programming. When asked to select methods, agents often converged on well-known choices—such as Leiden clustering or Harmony for batch-effect correction—even when a task called for more tailored judgment. Familiarity produced plausible answers, not necessarily the best fit for the data. The researchers suggest that frequently documented methods may be easier for agents to reach than less common or newer approaches.
The most revealing clue came from a test where the right answer sometimes meant refusing the assignment. In statistical genetics, researchers posed ten questions. Five were well-formed; the other five contained subtly false methodological premises and should have been rejected with a scientific explanation. For example, a request might ask an agent to draw a conclusion that the supplied evidence could not support.
Only Claude Code got all ten right. Gemini 3 Pro scored eight out of ten, correctly rejecting the false-premise questions but making mistakes on two valid ones. GPT-5.2 and Claude Sonnet 4.6 each scored six, but each failed to reject two of the deliberately flawed questions. Other agents showed different mixes of errors: Biomni rejected four false questions but struggled on valid ones, while STELLA failed to reject three false questions.
This was not a contest where confident-sounding prose could substitute for checking the premise. A system had to carry out the sound requests and recognize when the others could not be answered responsibly. The benchmark exposed two different failure routes: an agent could make a quiet technical mistake while analyzing valid data, or accept a false premise and build a polished response on top of it.
That helps rule out one easy explanation: that agents simply need to become better at writing code. Coding ability mattered, but so did keeping track of data across steps, checking whether a tool was being used correctly, choosing a method suited to the task, and asking whether the evidence could support the conclusion. The benchmark’s results suggest that competence in one part of a workflow does not automatically carry through to the whole investigation.
There are limits to what this puzzle can establish. SciAgentArena focuses on five domains, mostly in biomedicine, and its tasks were selected by domain experts. Its authors note that this may introduce bias, and that fields such as physics and materials science remain outside this version of the benchmark. The results are a map of performance on particular tasks—not a final score for every AI system, or a definition of what science must be.
Still, the pattern offers a useful answer to the opening question. Current agents can be capable collaborators when the path is specified and the checks are concrete. They become less dependable as the work demands adaptation, sustained planning, or skepticism about the request itself. A scientific assistant must do more than get an answer to run. It must also notice when the answer rests on a broken assumption. That may be the hardest step to automate—and the one that separates a completed workflow from a credible result.
