A robot tidying a table faces a deceptively hard question: where, exactly, should its hand go next? It may need to pick up a marker, pull open a drawer, or place something inside. A camera captures the whole scene, but the next movement usually depends on just a small part of it: the marker and the nearby space, or the drawer handle and the surfaces around it. Researchers have built a robot-learning system around that observation. Instead of asking a model to learn every movement from a complete snapshot, they teach it to pay attention to the local details most relevant to the action.
That shift matters because robot demonstrations are expensive. A person must guide the machine through a task, and the robot must record what it saw and did. In language and image research, enormous collections of examples can be gathered from the internet. A robot cannot learn to open a drawer by reading about it; it needs demonstrations tied to its own view of a real or simulated workspace. The fewer examples a system needs, the easier it becomes to teach it new tasks.
Earlier approaches have tried to give robots a useful map of the scene. One, called PerAct, converts camera images and depth measurements into a three-dimensional grid, then predicts a promising location in that grid for the robot’s next action. The method showed that carefully representing space can help a single model learn a range of tasks from relatively few demonstrations. But a dense scene map still leaves a difficult learning problem: connect everything the robot sees to the movement it should make.
The newer system, SGRv2, takes a more local route. It starts with a three-dimensional point cloud—a collection of points representing the visible surfaces in the workspace. Rather than compressing all those points into one summary before predicting an action, the model builds features for points throughout the scene. Each point can contribute a proposed movement or decision, while learned weights let the system favor the parts that matter more for the current task.
Imagine a robot approaching a drawer. The table, cabinet front, and surrounding objects all appear in the point cloud, but the handle is especially informative when the immediate goal is to pull. SGRv2 does not depend on a person drawing a box around the handle in advance. It learns which local regions are useful from the demonstrations. The same idea can apply when the robot must grasp a cube, press a button, or move an object into place: the useful clue is often close to the object or surface involved in the action.
The system also predicts movement relative to points in the scene rather than relying only on a fixed absolute location. That gives it a practical advantage: if an object shifts across the table, the action can shift with it. The approach combines this relative positioning with a point-by-point prediction process and extra training feedback for those local predictions. In effect, the model receives clues from many places in the scene about what the action should be, then learns which clues deserve the most influence.
In simulations, the researchers tested SGRv2 on 26 tasks using five demonstrations per task. Those tasks included opening doors and drawers, moving objects, and stacking blocks. The model achieved an average success rate of 53.2% and outperformed its strongest comparison system, RVT, on 23 of the 26 tasks. With 100 demonstrations, SGRv2’s average success rate was 63.3%; with five, it was 53.2%. The comparison suggests that its performance held up better as the training examples became scarce—not that five demonstrations were enough to master every task.
The researchers also tested the system on other simulated benchmarks, including tasks that require a continuous stream of movements rather than a small number of planned poses. SGRv2 performed better than comparison methods on average in those settings too, though results varied considerably by task. That variation is important: a strong average does not mean the robot has acquired a dependable, general skill for every kind of manipulation.
Some of the model’s learned attention offered a visual clue to why the approach might work. When researchers displayed which points received the strongest weights, the highlighted areas often matched functional parts of objects—for example, regions useful for opening a microwave or lifting a toilet seat. In a separate test, they added unfamiliar objects as visual distractions in simulated scenes. SGRv2 handled those distractions better than the comparison system. The result suggests that local focus can help a model avoid treating every visible object as equally important.
The approach was also tried on a physical Franka robot arm in tasks such as tidying a table and making coffee. With eight demonstrations per task, SGRv2 averaged a 63% success rate across the tested subtasks, compared with 41% for RVT and 30% for PerAct. But the coffee sequence included a stubborn weak point: pouring powder into a funnel succeeded in only 10% of trials for SGRv2. Real-world tests were also small, with each subtask evaluated over ten episodes, so the results are an early demonstration rather than a broad measure of reliability.
Generalization remains an open question too. In one color-based test, the robot learned from examples involving four cup colors and was then tested on familiar and unfamiliar color scenarios. SGRv2 succeeded on the unseen scenarios in 70% of trials, but the researchers noted that their tests did not cover broad changes in object shape, camera position, or background. A robot that learns to focus on the right points may need fewer examples, yet it can still be surprised by a new situation.
The deeper lesson is about what a learning system is asked to notice. More data and larger models are not the only ways to improve a robot. A useful design can also build in a sensible expectation: for many physical actions, nearby surfaces and objects carry more relevant information than the rest of the room. SGRv2 turns that intuition into a trainable method, and its results suggest that a robot can learn more from each demonstration when its attention is organized around the action at hand.
