In one virtual workshop, a robot has to pour a measured glass of wine without spilling. In another, it must fold a towel, weld a bracket, or retrieve a block from a gift box. The scenes are more than digital backdrops: liquids slosh, cloth bends, and objects respond to contact. But making a convincing virtual world is not the same as making a robot that can work in it.
That gap drives a new study from researchers at Carnegie Mellon University. They built a system called Gym-Anything for Robotics, in which coding agents create robot simulations and other agents try to find flaws. The team’s first surprise was how much careful checking those virtual worlds needed. Their second was that, even after building and auditing them, the most capable robot-controlling models could not complete a single task in the held-out test set.
The researchers wanted simulations to cover physical work beyond the familiar pick-and-place tasks of many robot benchmarks. They used U.S. employment and economic data, along with estimates of time spent on everyday activities, to identify valuable physical skills. Pouring, folding, mixing, and handling tools can all require different combinations of robot control and physics. From a larger selection of 120 potential simulations, the team built 25, ranging from food service and manufacturing to laboratories and logistics. Together, those simulations contain 200 environments: 25 for testing and 175 for training.
Each simulation had to pass checks for realistic physics, useful tasks, and reliable scoring. A pouring scene, for example, should not merely look like a bar: the liquid must behave plausibly, and the robot must be able to move the bottle and glass through physical contact. The builder produced evidence about how the world behaved; auditors then examined the code, repeated tests, and tried legal robot actions that might break the scene. The creator repaired problems and reran the checks.
The audits found plenty to fix. In the wine simulation, an auditor discovered liquid particles passing through the bottom of a glass bottle; after a repair, the same drop test produced 0.0 mL of liquid outside it. Across the benchmark, adversarial audits found a real defect in about three quarters of the simulations. A football’s turf could launch the ball without a robot touching it. In an ingredient-mixing scene, a bowl lowered past the counter’s edge left sauce floating in midair. These were not reasons to abandon the simulations. They were evidence that building a useful virtual world meant actively searching for ways it could lie.
The team also compared its scenes with existing simulators for similar physical interactions. In a blind comparison, an evaluator model judged the new simulations ahead in 80% of visual comparisons and 76% of physics comparisons. That is a relative assessment by a model, not proof that every virtual detail matches the physical world. Still, the checks gave the researchers grounds to ask a tougher question: if the simulation itself was not the obvious weak link, how would current robot models fare?
They gave three frontier models up to three hours per attempt to perform each test task. The models could use camera images and robot feedback, but they did not see the task’s reward while working. None completed any of the 25 test tasks. Their average rewards, which also credited partial progress, ranged from 0.002 to 0.06 on a scale from 0 to 1. Even the strongest model received no reward at all on 16 tasks.
The result overturns an easy assumption about robot learning: that a capable model mainly needs more room to practice. These tasks demanded precise, sustained physical interaction, and the test showed how far the models still had to go. But the simulations offered another resource unavailable on a real robot: the exact state of the virtual world. The system could reveal where an object was, whether two surfaces were touching, or what forces were acting—details a camera view might not make clear.
To test the value of that hidden information, the researchers let one model act as a coach. The student saw only the ordinary robot observations; the coach could inspect the full simulator state and give qualitative hints. It could not simply issue commands or pass along numerical state readings. On its first attempt, the coached student earned an average reward of 0.29, compared with 0.14 for the student working alone. In a towel-folding task, for example, the student had struggled to hold the cloth with both grippers. The coach suggested folding in stages with one gripper, a change of strategy that led to success.
The team tried a second route from hidden information to visible skill. A model wrote code-based robot policies that could use the full state to complete two gift-box tasks across varied starting conditions. Researchers used 1,000 successful simulated demonstrations to train a smaller vision-and-action model, π0.5. The trained model, which relied on camera views and robot feedback rather than the simulator’s hidden state, succeeded on 53% of attempts in those tasks, up from 0% before training. On variations it had not seen during training, it succeeded 55% of the time.
That is a striking jump, but it happened inside the simulation. The study does not show that the trained model can yet do the same work with a physical robot. Nor does it show that all important real-world physics can be recreated in code. Instead, it points to a useful possibility: a virtual world can teach by exposing information that a deployed robot will never receive directly, then use that experience to train a system that acts from cameras and its own movement feedback.
The unexpected lesson is not that virtual practice has solved physical work. The new benchmark found the opposite: powerful models failed every held-out task. Yet when the simulated world could reveal its hidden mechanics, those failures became more informative. The challenge now is to make that bridge—from carefully audited virtual practice to reliable behavior in the physical world—hold up outside the screen.
