A robot reaches for a towel, misses a corner, and starts to drag the fabric into a useless heap. In a conventional training setup, fixing that mistake may mean stopping the physical robot, resetting the scene, and asking a person to demonstrate what to do instead. Researchers have now tested a different route: let the robot fail inside a learned simulation, rewind to the trouble spot, and practice several human-guided recoveries there.
The approach, called Human-in-the-World-Model, or Hi-WM, turns a learned model of the robot’s surroundings into more than a place to preview what might happen. It becomes a practice room. The robot’s existing policy acts in the model; when its behavior starts going wrong, a human operator steps in and supplies corrective actions. Those lessons are then added to the data used to train the policy further.
In this setup, the world model predicts how the scene will change when the robot moves. It is not simply a conventional hand-built animation of a robot in a virtual room. The researchers trained a system with a visual encoder, an action-conditioned dynamics model, and a visual decoder: in plain terms, it takes in what the robot sees and does, then predicts the next view. The policy acts on those predicted views in a closed loop, much as it would respond to new camera images during a physical task.
The key move comes when the rollout goes off course. Hi-WM can cache intermediate states, restore an earlier one, and try a different corrective action from that same point. One simulated near-miss can therefore become several alternative recovery examples. An operator can intervene through different devices, including a keyboard, a robot arm, or virtual-reality controllers. Once a correction gets the robot back on track, control can return to the policy.
That rewind button is only useful if the simulated world responds realistically enough. A model that makes a poorly aligned grasp look successful could teach a robot the wrong lesson. So the researchers deliberately included difficult examples in training the world model: failed grasps, unintended contact, incomplete tasks, and poses that miss their target. They also added data from the edges of the robot’s workspace, where movements can be harder to represent accurately. The model was trained to respond to a 14-dimensional continuous action space for a dual-arm robot.
The team tested the method on a dual-arm robot in three tabletop tasks: folding a towel, pushing and rotating a T-shaped object into position, and routing a rope into two slots. These tasks span firm objects and flexible materials, which behave quite differently when touched. The researchers compared two robot policies, known as DP and π0, under matched post-training budgets. They also compared Hi-WM with a baseline that collected successful simulated rollouts but did not ask a human to correct failures.
In all six combinations of policy and task, Hi-WM produced a higher real-world success rate than either baseline. For the DP policy, towel-folding success rose from 42.1% for the base policy to 92.1% after Hi-WM training; for π0, rope-routing success rose from 64.7% to 100.0%. Averaged across the six comparisons, the authors report gains of 37.9 percentage points over the base policies and 19.0 points over the simulation-only baseline.
That comparison matters because simply generating more practice is not the same as practicing the hard part. A policy can replay successful behavior it already knows, but those extra examples may leave its weak spots untouched. Hi-WM directs human attention toward the moments when the policy begins to fail, then lets the operator explore alternatives from a reusable starting state. The results suggest that this targeted supervision—not just more simulated data—helped the robots recover more reliably in the physical tasks.
The researchers also checked how well the model’s predictions tracked the real robot. Across their evaluations, success rates in the world model and in the physical world had a Pearson correlation of r = 0.953. They further tested the pushing task under changes to appearance, background, and distracting objects, and found that policies trained with virtual corrections did better in those tested conditions. These are encouraging signs that the simulated practice transferred—but they are evidence from a specific setup, not a guarantee that any generated robot world will be trustworthy.
That boundary is important. Hi-WM was evaluated on three manipulation tasks and one dual-arm platform, and the world model depended on data that covered both ordinary actions and tricky edge cases. Its promising results do not establish that the approach will work for every robot, object, or unfamiliar environment. The model must first be faithful enough where the robot is likely to struggle, and its lessons still need to be checked on real hardware.
Even with those limits, the study reframes what a robot simulator can do. Instead of merely showing a possible future or grading a finished policy, it can give people a place to interrupt failure, rewind, and teach a better next move. The physical robot still has to prove that the lesson holds. But if the practice room is accurate where it counts, one costly mistake on hardware may become several useful experiments in a world the researchers can rewind.
