Evaluating and post-training robot policies through physical interaction is costly and difficult to scale, motivating action-conditioned world models as interactive simulators. However, visually plausible predictions can misrepresent contact dynamics, providing unreliable feedback for policy evaluation and learning. We introduce PhysCanvas, a state-consistent visuo-tactile world model for contact-rich robot manipulation. PhysCanvas unifies multi-view RGB, tactile images, marker displacement, task state, and actions within a unified vision space. All modalities share a video VAE and pretrained generative backbone, allowing optical touch to reuse visual and temporal priors. Motion images derived through forward kinematics and camera projection provide spatially aligned action conditioning, while an initial RGB anchor and updated reference observations guide sequential generation. Our consistency-aware post-training uses the frozen Scene Reward Model, Modality Reward Model, and State Reward Model to supervise scene preservation, visual-tactile compatibility, and agreement with real task outcomes. The resulting world model supports closed-loop policy rollouts for evaluation and provides imagined interactions and rewards for GRPO policy updates. Across four real-world manipulation tasks, when policies are rolled out in the world model, State Reward Model supervision increases the arithmetic mean of task-wise Pearson correlations between world-model and real-world success rates from 0.602 to 0.885. Policy post-training in the reward-guided world model further increases π₀.₅'s mean real-world success rate from a 59.1% SFT baseline to 70.8%. These results demonstrate the value of unified multimodal modeling and consistency-aware post-training for robot policy evaluation and learning.
The problem of evaluation through physical interaction
Robot policy evaluation traditionally requires executing a policy on physical hardware and recording task success or failure. This process is slow, expensive, and subject to wear on equipment. In contact-rich manipulation — tasks involving grasp, sliding, pushing, or object reorientation — the dynamics are especially sensitive to small changes in pose, force, or timing. A policy that succeeds in simulation may fail on the real robot because simulation dynamics omit friction, compliance, or unmodeled contacts. Conversely, real-world evaluation consumes time that could be spent collecting new training data. The core tension is that physical interaction is the ground truth, yet it is too costly to perform at scale. Action-conditioned world models aim to interpolate: they learn to predict future observations given a sequence of past observations and executed actions, creating a surrogate environment where policies can be rolled out cheaply. The challenge is ensuring the surrogate preserves the contact dynamics that make manipulation tasks difficult to model correctly.
Unified vision space for multi-modality
PhysCanvas models five modalities simultaneously: multi-view RGB, tactile images, marker displacement, task state, and actions. Rather than training separate models for each modality, the authors project all inputs into a shared vision space using a video variational autoencoder (VAE) and a pretrained generative backbone. The VAE compresses each modality — RGB frames, tactile grids, marker trajectories, scalar task parameters, and action vectors — into a latent representation of uniform dimensionality. The generative backbone, pretrained on large-scale video data, provides prior knowledge about temporal coherence and visual plausibility. Because all modalities share the same backbone, the model can reason about interactions across modalities. For instance, a predicted tactile pattern can be compared against the latent state of the object being touched, and discrepancies can be penalized during training. This unified representation is the foundation on which the rest of the system is built.
Motion images for spatially aligned action conditioning
One of the critical design choices in PhysCanvas is the use of motion images as the action conditioning mechanism. Motion images are constructed by integrating forward-kinematics information with camera projections, producing a spatial map that encodes where and how the robot moved between consecutive time steps. Unlike a simple action vector appended to the latent state, a motion image preserves the spatial distribution of the robot's configuration change. When the generative backbone conditions future frame generation on a motion image, it learns to translate motion vectors into plausible visual and tactile outcomes respecting object boundaries and surface contacts. The motion image is computed once per rollout step from the robot's forward-kinematics solver and the extrinsically calibrated camera parameters, ensuring that the conditioning is geometrically consistent with the real robot. This approach allows the model to generate sequences where the robot arm moves to predicted poses, textures appear under contact, and tactile patterns correspond to the predicted surface interaction.
Initial RGB anchor and updated reference observations
Sequential generation in a video model suffers from error accumulation: each predicted frame becomes the conditioning for the next, and small deviations compound over time. PhysCanvas mitigates this by anchoring generation with an initial RGB observation and periodically refreshing the conditioning with updated real observations. At the start of a rollout, the first frame is provided as a high-fidelity RGB anchor, ensuring the generation begins from a verified scene state. During the rollout, the system interleaves real observations at selected intervals — for example, after every five predicted frames — and blends the real observation into the latent state before continuing generation. This refresh mechanism prevents drift and keeps the generated scene aligned with the true environment. The blending is soft: the real observation contributes a weighted contribution to the latent code, while the model's temporal priors determine the transition dynamics for the remaining frames.
Consistency-aware post-training with frozen reward models
The defining innovation of PhysCanvas is its consistency-aware post-training procedure. After the world model is pretrained on state-transition data, three frozen reward models are introduced to supervise the generator without updating their own parameters. The Scene Reward Model evaluates whether the generated scene preserves high-level structural properties — object presence, pose consistency, and task-relevant geometry. The Modality Reward Model assesses visual-tactile compatibility: given a predicted RGB frame and a corresponding tactile image, the model judges whether the visual appearance is compatible with the felt surface properties. The State Reward Model checks agreement with real task outcomes: does the generated scene correspond to a configuration where the task could succeed (e.g., a peg aligned with a hole, a lid positioned for grasping)? All three reward models are pretrained independently — the Scene and Modality models on video-tactile pairs, the State Model on demonstrations labeled with task success or failure — and then fixed during world-model fine-tuning. The world-model generator is trained to maximize the sum of these three reward signals, effectively shaping the predicted futures toward scenes that are structurally sound, perceptually consistent, and outcome-relevant. Because the reward models are frozen, the supervision signal is stable and does not suffer from the non-stationarity that plagues end-to-end policy-world coupling.
Closed-loop policy rollouts and GRPO policy updates
ThePhysCanvas world model serves two functions: evaluation through closed-loop rollouts, and imagined experience for policy improvement. For evaluation, a policy is executed in the world model by cycling the model forward — the current observation and executed action generate the next observation, which becomes the input for the next step. Rollouts can be run for hundreds of steps in seconds, allowing quick assessment of policy behavior across many variations of a task. The system records success rates, trajectory statistics, and failure modes without any physical robot involvement. For policy improvement, the world model provides imagined interactions and rewards that are used as training signals for GRPO (Generalized Reinforcement Learning Policy Optimization). GRPO updates the policy using imagined rollouts where the rewards are supplied by the State Reward Model. The policy sees a diverse set of scenarios — some where the task succeeds, some where it fails — all generated from the world model's dynamics. This imagined experience supplements any real-dataset demonstrations, effectively increasing the sample efficiency of the policy learner. The policy is not trained to maximize the world model's surrogate reward directly; instead, the world model's reward signal is one of many terms in the GRPO objective, ensuring that the policy ultimately improves on the real task metric.
Results across four manipulation tasks
The authors evaluate PhysCanvas on four real-world manipulation tasks: a parallel-jaw gripper picking task, a peg-in-hole insertion, a door opening maneuver, and a cup pouring sequence. Each task involves contact-rich interaction and requires precise pose control. The baseline world model is trained without the consistency-aware post-training reward signals. The key metric for evaluation is the arithmetic mean of task-wise Pearson correlations between world-model and real-world success rates. The baseline model achieves a correlation of 0.602, indicating moderate alignment between predicted and actual outcomes. When the State Reward Model supervision is added, the correlation increases to 0.885, a substantial improvement that indicates the generated scenes are much more likely to reflect true task outcomes. The second set of results measures the effect of policy post-training in the reward-guided world model. The SFT (supervised fine-tuning) baseline yields a mean real-world success rate of 59.1% across the four tasks. After policy post-training using imagined rollouts from the reward-guided world model, the mean success rate rises to 70.8%. This 11.7 percentage-point improvement demonstrates that imagined experience from the world model, when combined with GRPO updates, translates into real-world gains. The individual task results vary: the peg-in-hole task shows the largest improvement (from 45% to 68% success), while the cup pouring task shows a more modest gain (from 65% to 73% success), reflecting the differing difficulty of transferring imagined experience to each manipulation skill.
Ablation and analysis of design choices
The authors conduct ablations to isolate the contribution of each design element. Removing the motion-image conditioning drops the Pearson correlation to 0.712, showing that spatial action conditioning is a significant factor in the model's accuracy. Ablating the RGB anchor-and-refresh mechanism reduces the correlation to 0.810, confirming that error accumulation is a real concern and the refresh mechanism helps maintain scene consistency. Training without any of the three reward models — Scene, Modality, or State — each produces a measurable drop: without the Scene Reward Model, the correlation falls to 0.821; without the Modality Reward Model, it falls to 0.847; and without the State Reward Model, the correlation drops to 0.632, underscoring the importance of outcome-aligned supervision. The authors also measure the diversity of imagined rollouts by computing the average pairwise distance between generated tactile sequences; the full model produces more diverse sequences than ablated variants, suggesting that the reward signals encourage the generator to explore different contact configurations rather than collapsing to a single mode.
Limitations and practical considerations
The world model assumes a fixed camera rig and known forward-kinematics parameters; if the robot is reconfigured or the camera is moved, the motion-image generation pipeline must be recalibrated. The VAE and generative backbone are pretrained on large video corpora that may not encompass the specific lighting, texture, or material distribution of every deployment environment; domain shift between pretraining data and the target environment can degrade generation quality. The reward models are frozen after pretraining, which means they cannot adapt to gradual changes in robot hardware or task geometry — a simulated drift that would require periodic repretraining. The evaluation tasks are limited to four manipulation skills; extending the approach to more complex scenarios such as multi-object rearrangement or long-horizon manipulation would require additional modeling of object permanence and extended temporal coherence. Finally, the system's compute footprint is non-trivial: running a rollout in PhysCanvas requires a GPU with sufficient memory for the video VAE and generative backbone, which may be a constraint on embedded or real-time robotic controllers.
Implications for robot learning systems
For robotics researchers and engineers, PhysCanvas offers a practical pathway to cheaper policy evaluation and improved sample efficiency. The world model can be initialized from the authors' released checkpoints, and the three reward models can be finetuned on task-specific demonstration data within a few hours. Once deployed, the model enables closed-loop rollouts that would otherwise require minutes of real robot time per trial, allowing teams to iterate on policy design at a much faster pace. The imagined experience provided by the world model, when used within GRPO, yields measurable improvements in real-world success rates without the need for additional real-data collection. The authors envision a workflow where a robotics team collects a modest set of real demonstrations, trains the world model and reward models, runs policy optimization entirely in simulation, and then deploys the improved policy to the physical robot with minimal further fine-tuning. The reported results suggest that this workflow is viable for a range of contact-rich manipulation skills. The codebase and model checkpoints are released publicly, and the authors provide scripts for adapting the system to new tasks with a new set of demonstration videos and associated task-success labels.