From a Single Demo to a Reusable Robot Program: Closing the Agentic Coding Loop
Coding agents like Codex and GitHub Copilot have demonstrated remarkable ability to generate and iteratively refine software programs. The same iterative cycle -- generate, execute, verify, revise -- is what makes these agents powerful. But bringing this capability to robotics has faced a fundamental obstacle: a robot programming agent needs a testable task specification, a library of manipulation primitives, and an interactive simulation environment to verify its code. Existing systems assume all three are provided in advance. What happens when they are not?
A new paper from MIT, the National University of Singapore, and NVIDIA introduces RAPID (Robot Agentic Programming from Demonstrations), a framework that infers all three ingredients from a single visual human demonstration. Given one video of a human manipulating objects and a brief language description, RAPID automatically generates, verifies, and refines a reusable robot program -- without any pre-specified task rewards, human-engineered primitive libraries, or provided simulation environments.
The Problem with Current Robotic Coding Agents
Code-as-Policies and similar approaches use language models to generate robot control code directly from natural language instructions. CaP-X and related systems add iterative refinement: the agent generates a program, executes it in simulation, observes the result, and revises the code. But these approaches assume the coding agent already has access to success signals, manipulation primitives, and a resettable simulation environment -- the very ingredients that are hard to provide for novel tasks.
The challenge is particularly acute in nonprehensile manipulation, where a robot repositions objects using environmental contact rather than grasping. Tasks like pushing, flipping, pivoting, and toppling involve complex, geometry-dependent interactions that cannot be captured by a fixed set of reusable primitives. Each new object, pose, or fixture requires different contact mechanics, making manual primitive design a substantial engineering burden.
RAPID targets exactly this gap. The key insight is that a single demonstration already contains everything needed to bootstrap the agentic coding loop: it shows the task specification implicitly, it demonstrates the manipulation strategy that can be decomposed into primitives, and it provides the initial scene geometry from which a simulation environment can be reconstructed.
How RAPID Works
RAPID operates in three phases: reconstructing an interactive simulation environment from the demonstration, constructing manipulation primitives through an agentic coding loop, and composing those primitives into a task-level strategy.
Reconstructing the Environment
From the initial RGB-D frame of the demonstration, RAPID uses a vision-language model to identify objects, assign semantic labels, and estimate physical properties such as mass, friction, and elasticity. It then uses SAM 3 to obtain segmentation masks and SAM 3D to reconstruct complete 3D meshes. FoundationPose registers each mesh to the observed frame, assembling the reconstructed objects and robot model in MuJoCo. The resulting interactive simulation environment preserves the scene geometry and spatial relationships needed to execute and verify candidate programs.
To improve generalization, RAPID generates feasible scene variants by randomizing object appearances, sizes, poses, and physical parameters. However, not all randomly sampled variants are physically meaningful. The coding agent writes a filter for each primitive that discards variants violating the declared semantic roles and physical feasibility constraints.
Constructing Primitives Through the Agentic Loop
RAPID segments the demonstration into ordered interaction phases. For each phase, the coding agent infers a semantic goal and success predicate from the demonstration and language description. If the behavior is covered by an existing primitive, it retrieves that primitive. Otherwise, the agent generates a new one.
Each primitive is represented as a local trajectory-optimization program with five components: a semantic goal describing the intended object-level effect, a relational interface specifying which scene entities participate and what relations must hold, a success predicate that checks whether the effect was achieved, a geometry resolver that computes scene-dependent geometric features at each step, and an optimization cost that guides the trajectory search.
The agent then verifies the primitive across the demonstrated scene and its variants, iterating until the success predicate is satisfied. The verified primitive is frozen and added to the library. This construction takes on the order of tens of minutes per task, but the resulting primitive is reusable across all 50 test scenes.
Composing Strategies Through Relational Constraints
After primitives are constructed, the coding agent composes them into a task-level strategy. The strategy specifies the sequence of primitives to invoke, how task-level semantic roles map to primitive roles, and how the goal of each phase is determined from the outcome of the preceding phase.
The composition is governed by relational constraints. For example, consider a task where the robot must first flip an object to expose a graspable edge and then push it into a fixture. The connecting constraint takes the post-flip state of the object, computes the goal position based on the spatial relation between the object and the fixture, and determines the displacement parameter for the press-move primitive. This constraint is evaluated just in time at the beginning of each phase.
The critical design choice is that primitives are expressed in terms of object-level effects -- push by a displacement, flip by an angle -- rather than fixed trajectories or waypoints. The geometry and physical properties are resolved at execution time. This is what enables generalization: the same program can operate on objects of different sizes, shapes, poses, and materials because the trajectory optimizer adapts to each scene's specific geometry.
Experimental Results
RAPID was evaluated on eight challenging contact-rich nonprehensile manipulation tasks in simulation, on the LIBERO-Pro benchmark for prehensile manipulation, and on a real Franka Research 3 robot arm.
Nonprehensile Manipulation
For each of the eight tasks, RAPID receives a single visual demonstration and a language description. The test evaluates generalization across 50 novel scenes that differ in object pose, appearance, size, shape, material, and include irrelevant distracting objects. The results are dramatic.
Vanilla Code-as-Policies (CaP) achieves only 1.3% average success across the eight tasks. CaP generates a program tailored to each test scene but has no interactive environment to verify whether the proposed motions produce the intended object interactions. Adding the object-centric relational program representation (OReP) does not improve performance, confirming that the representation alone is insufficient without closed-loop verification.
CaP-Agent0, which receives ground-truth success signals and may revise its program at evaluation time, achieves 14.6% average success. Adding scene variants brings this to 12.6% -- actually slightly worse, because without the relational program representation, the agent cannot construct a generalizable program from diverse scene variants.
RAPID without scene variants achieves 53.2% average success. The full RAPID system, with both the object-centric relational representation and scene-variant verification, achieves 75.9% average success across all eight tasks. The individual task success rates range from 58.7% to 93.3%.
The ablation study isolates the contribution of each component. Removing scene variants causes substantial performance drops on seven of eight tasks, with increased variance indicating reduced stability. The scene-variant generation module is essential for verifying that programs generalize, not just that they work on the demonstrated scene.
LIBERO-Pro Benchmark
RAPID also demonstrates strong performance on the LIBERO-Pro benchmark, which features prehensile manipulation tasks including picking, placing, and repositioning objects. RAPID achieves 100% success on the object-centered subset, 95% on the goal-centered subset, and 83% on the spatial subset, substantially outperforming baselines including OpenVLA, pi-0, CaP-Agent0, and ASPIRE.
Real-World Deployment
The most compelling validation comes from the real Franka Research 3 robot arm equipped with a parallel gripper and an Intel RealSense L515 camera. RAPID was deployed on all eight nonprehensile tasks, with 10 test scenes per task. The contact-rich manipulation behaviors optimized in simulation transfer directly to the real robot and execute effectively.
CaP-Agent0, by contrast, struggles to generate reliable trajectories using procedural programs. The real-to-sim gap is managed through the VLM's prior knowledge of physical properties, SAM 3D's strong 3D reconstruction, and the compliant operational-space controller on the Franka arm.
Why This Approach Matters
RAPID addresses a genuine bottleneck in robotics. Every manipulation task requires custom programming: the specific sequence of interactions, the geometric constraints that connect them, and the success criteria that verify them. Traditional approaches require an engineer to design primitives, specify rewards, and build verification environments for each new task. RAPID compresses this into a single demonstration plus a language description, with the coding agent handling the rest.
The object-centric relational program representation is the key enabler of generalization. By expressing primitives as local trajectory-optimization programs that realize object-level effects, RAPID separates the task's invariant structure from the specific demonstration trajectory. The strategy and primitives remain frozen after construction. At deployment time, a lightweight semantic binding assigns scene objects to the strategy's semantic roles, and the trajectory optimizer adapts to the new scene's geometry. Semantic binding takes only a few seconds per scene.
The iterative agentic coding loop -- the same pattern that makes coding agents powerful for software development -- is what makes RAPID robust rather than brittle. A single-shot program generator cannot reason about whether a motion will produce the intended contact interaction. But an agent that can execute a candidate program in simulation, observe whether it succeeds, diagnose the failure, and revise the code can converge on a correct solution. The scene-variant verification further ensures that the solution generalizes rather than overfitting to the demonstration.
Limitations and Open Questions
RAPID's construction cost is significant: tens of minutes per task for the agentic loop to converge. This is acceptable for offline program synthesis but limits real-time adaptation. The system also depends on the accuracy of the vision-language model's physical property estimates and the quality of the 3D reconstruction.
The evaluation focuses on quasi-static manipulation with a single robot arm. Extending RAPID to dynamic manipulation, multi-arm coordination, or deformable objects would require additional primitives and verification strategies. The system also assumes a single demonstrator providing one video; handling demonstrations from multiple users or noisy demonstrations remains an open direction.
The paper also notes that the coding agent (Codex with GPT-5.6 Sol) is the primary cost driver. Using more capable or efficient coding agents could reduce construction time, while more efficient verification could reduce the number of iterations needed for convergence.
Nevertheless, RAPID demonstrates a compelling new paradigm for robotic programming. Rather than requiring engineers to manually design every primitive and specify every success criterion, the system lets a coding agent derive them from a single demonstration. The resulting programs generalize across substantial variations in object appearance, pose, geometry, physical properties, and scene configuration -- a capability that, until now, required extensive engineering effort per task.