A robot has a simple job: pick up a cube, put it on a target, do that three times, then press a button. In a simulated tabletop experiment, researchers slipped two five-second pauses into the routine. The cube stayed put. No extra pick happened. Yet the robot pressed the button after only two repetitions. What had it counted—its work, or the time that passed while it was waiting?
The robot’s memory offered a clue. Rather than keeping every moment of an episode, the tested policy saved 32 frames spaced evenly across it. Each frame also carried a timestamp. That arrangement gives a memory a compact view of the whole task, like a small set of pages sampled from a long diary. But a pause gets pages, too. The record can show that time passed without showing any new progress.
At first, the mistake could have come from the interruption itself. Maybe the robot’s arm ended up in an odd position; maybe the pause unsettled its movements. The researchers tested this by running the same pause while keeping its frames out of memory. The robot still waited the same five seconds, but its memory did not record the interval or advance its timestamps across it. Success returned to 45 out of 50 trials on the pick-and-place task, close to the 47 out of 50 without pauses. On a second task, moving a cube back and forth, hiding the pauses restored success to 46 out of 50, near the 44 out of 50 baseline.
That was the first strong clue: the physical wait alone was not enough to explain the failure. When the pause appeared in memory, success fell to 16 out of 50 on the pick-and-place task and 10 out of 50 on the back-and-forth task. The mistakes were tellingly specific. The robot usually did not drop the cube or make too many trips. It stopped early, as if the uneventful interval had counted toward the required repetitions.
Perhaps the policy was reading the timestamps as a stopwatch. The researchers changed the timestamps while leaving the remembered frames alone. A faster-running clock pushed the robot toward premature stops; a slower one encouraged extra repetitions. This fit the suspicion that elapsed time had become a shortcut for progress. In the demonstrations used to teach the policy, actions tended to unfold at a steady pace. If a certain amount of time usually meant a certain number of repetitions, timing could seem like a reliable answer.
But removing the timestamps should have settled the question. It did not. A version trained without explicit time stamps still failed under pauses: success dropped to 21 out of 50 on one task and 14 out of 50 on the other. Hiding the pauses from memory again brought performance back. The clock, it turned out, had more than one way to sneak into the record.
Even without time labels, the memory still sampled frames by time. So the pause occupied some of its limited slots with scenes that showed no progress. The researchers tested whether those slots, rather than their specific pictures, were the problem. Replacing the pause frames with copies of the last pre-pause frame did not remove the damage. But filling the same number of slots with repeated copies of real task frames, without inserting a pause, preserved performance. The empty interval mattered because it took up room in a memory organized around time.
There was a broader reason the shortcut might seem attractive to a learner. In the study’s scripted demonstrations, elapsed time alone predicted how many repetitions had been completed 89% of the time for one task and 80% for the other. In some regular real-robot teleoperation demonstrations they examined, the relationship was also strong. The robot was not inventing a useless clue from nowhere; it was leaning on a clue that often worked in its examples. The trouble came when time and progress parted company.
The researchers tried two remedies. The first was to teach with pauses inserted into the training examples, hoping the policy would learn not to count waiting as work. That did not reliably solve the problem, and counting without pauses could get worse. The second remedy changed how memory advanced: instead of giving every stretch of time a share of its slots, it collapsed intervals in which the scene returned to a prior state, or remained still. In this event-indexed memory, the record moved forward when something relevant happened, not simply because the clock ticked.
In these tests, event indexing at least doubled success under pauses, fidgets, and air-grasps—robot motions that returned to the starting pose without moving the cube. On the back-and-forth task, for example, success after an air-grasp rose from 13 to 43 out of 50. The change was made at test time, without retraining the policy. That makes the result a useful clue about the source of the error, not a finished memory system ready for any robot: the detector relied on simulator signals and was designed around the scripted events in the experiment.
And the proposed fix has its own puzzle. A real repetition can also bring the robot and cube back to a familiar arrangement. If the memory treats every return to an earlier state as no event, it could erase genuine progress. Meanwhile, tasks that depend on duration still need a clock; ignoring a pause would be wrong if the job were to wait for a set time. The experiment points toward a distinction, not a universal rule: memory of what happened and awareness of how long it took are different kinds of information.
The study tested simulated tasks and a limited set of memory policies, so it does not show how often this failure occurs in real deployments. But its central question reaches beyond one cube and one button. A timeline can tell a robot how long it has been acting; it cannot, by itself, tell whether anything useful happened. The mystery was not that the robot forgot its task. It was that its memory had confused the passage of time with the making of progress.
