Why Generated Video Alone Fails at Contact-Rich Manipulation
Video generation models have opened a new path for robot learning. Instead of teleoperating a robot hundreds of times to collect demonstrations, you can generate a video of the task from a language prompt, extract the end-effector trajectory, and replay it. Pipelines like Dream2Flow, NovaFlow, and ATM do exactly this: they segment objects, track 3D points, and convert the resulting motion into robot commands. For pick-and-place and simple positioning tasks, this works. The robot picks up an object and moves it to the right spot.
But contact-rich tasks are different. Wiping a whiteboard, peeling a carrot, pressing a spring-loaded button, or sliding a gripper under a box all require the robot to apply appropriate forces during contact. A kinematic trajectory that reproduces the right motion shape but ignores force will either hover above the surface, apply too little pressure to accomplish the task, or slam into the object and trigger a protective stop. The problem is that contact forces are not directly observable from video frames. A penetration depth of one millimeter versus two millimeters looks nearly identical in pixels but produces a large difference in contact force. Video alone does not and cannot encode this information.
Prior work has addressed force-aware manipulation through force-instrumented physical demonstrations (using F/T sensors during teleoperation), simulation-based training with engineered force supervision, or learned compliance policies from kinesthetic teaching. All of these require either physical interaction data or a simulation environment with accurate force models. The question Guanhua Ji, Tianyu Li, Dayoon Suh, Yuqian Zhang, Boyan Zhang, and Nadia Figueroa from the GRASP Lab at the University of Pennsylvania ask is whether you can construct a desired-force profile from generated data alone, without force-labeled demonstrations or simulation.
The Core Idea: Contact Makes Sound, and Sound Encodes Force
The key observation is that contact produces sound, and the loudness of that sound correlates with the intensity of the contact. When you wipe a surface hard, it sounds louder than when you wipe gently. When you strike a table with a hammer, a harder strike produces a louder sound. This is a psychoacoustic relationship: a 10 LU (loudness units) increase corresponds to a doubling of perceived loudness. The paper exploits this by using generated audio as a proxy for relative force variation, not as a prediction of the sound produced during real-world execution.
Modern video generation models like Seedance 2.0 jointly generate video and synchronized audio within a single model, temporally aligning the sound with the visual interaction. The system prompt instructs the generator to emphasize sounds produced by physical contact while suppressing ambient, background, and unrelated audio. The result is a video-audio pair where the audio track captures the temporal structure of contact events: when contact begins, when it intensifies, when it stops.
From this audio, the pipeline extracts per-frame loudness in LUFS (loudness units relative to full scale), discards low-energy frames below a threshold of -65 LUFS (treating them as noise and assigning zero force), converts remaining loudness values to a relative ratio using the psychoacoustic formula r_t = 2^(l_t / 10), and maps these ratios into a bounded force range through min-max normalization. The result is a time-varying desired-force profile F* that stays within [F_min, F_max], where F_min = 0.25 * F_max and F_max = 15 N for all experiments. The audio determines the shape of the force profile; the absolute level is set by the task-level bound.
This is a deliberately conservative design. The loudness-to-force mapping is a heuristic, not a physical model. But combined with geometric contact gating (force is only applied when the gripper and object point clouds are within 3 centimeters), closed-loop force regulation during execution, and the bounded force range, it provides a useful signal for tasks where the relative timing and intensity of contact matter more than the precise force value.
The Full Pipeline: From Prompt to Force-Aware Trajectory
The pipeline has five stages, each building on the previous.
Video and audio generation. Seedance 2.0 takes an initial RGB image and a structured task prompt, producing a video V = {I_t} and synchronized audio A. The system prompt fixes the camera as stationary and directs the generator to emphasize contact sounds. For whiteboard wiping, carrot peeling, and lamp button pressing, every generated video is used directly without manual screening. For chocolate box stacking, 20 videos are generated and 10 that deviate from the prompted grasp (gripper grasps from sides instead of sliding under) are discarded, leaving 10 usable trajectories.
Object segmentation and 3D reconstruction. The pipeline identifies the two entities involved in contact (gripper and object) using MolmoPoint for 2D localization and SAM 2 for mask propagation throughout the video. Depth and 3D point tracks are estimated using TAPIP3D. The predicted depth is calibrated from the first frame using ground-truth depth with a global scale and offset fitted by RANSAC linear regression: z_cal(u,v) = a * z_pred(u,v) + b. This calibration is applied to every frame of the generated video.
End-effector trajectory estimation. Using the gripper mask and TAPIP3D tracks, the pipeline collects 3D gripper points in the first frame and their correspondences in each subsequent frame. Since the visible end-effector is approximately rigid, a relative rigid transform is estimated by minimizing the sum of squared distances between corresponding points, yielding a rotation and translation in SE(3). This relative motion is then mapped from the camera frame to the robot base frame by conjugation with the calibrated camera pose, and applied as an incremental delta on top of the robot's measured initial end-effector pose. The camera-derived motion is anchored to the true robot state at execution time, which is critical for robustness to small trajectory errors.
Audio-shaped force profile. SAM-Audio processes the generated audio with a text prompt "The sound of contact between gripper and [object]" to isolate contact-related signals from background noise. Per-frame loudness is computed, low-energy frames are discarded, and the remaining loudness values are converted to a relative ratio and mapped into the force range [F_min, F_max] through min-max normalization. The result is a bounded, time-varying desired-force profile that captures when contact occurs, how its intensity changes over time, and when it stops.
Point cloud force direction. For frames with nonzero desired force, the pipeline forms nearest-neighbor pairs between gripper and object point clouds and averages the pairwise vectors to estimate the force direction. A nonzero force is only maintained when the minimum distance between gripper and object points is below 3 centimeters. This geometric gating ensures that force is only commanded during actual contact, not during approach or retraction.
The combined output is a force-aware trajectory tau = {(x_t, d_t, F*_t)} for t = 1 to T, where x_t is the end-effector pose, d_t is the contact direction, and F*_t is the desired contact-force magnitude.
Execution: Closed-Loop Force Regulation Without Audio
The trajectory is executed on a Franka Panda using a Cartesian impedance controller running at 1 kHz with fixed translational stiffness k = 2000 N/m and rotational stiffness k_r = 150 N*m/rad. The force regulator updates its commanded pose at 50 Hz using measured force feedback from the robot's joint torques. No audio is recorded or processed during execution; the regulator uses only force measurements to track the audio-shaped profile.
The regulation mechanism is a virtual-target displacement. Instead of modulating stiffness, the controller drives a displacement delta along the contact direction, so the commanded pose is x_cmd(t) = x_plan(t) + delta. At steady state, the contact force is approximately F = k * |delta|. The displacement is updated by integrating a normalized force error:
delta += K_I * d_t * clip((F* - F_hat) / F*, -1, 1)
where K_I = 0.0003 is the integrator gain, d_t is the contact direction, and F_hat is a short-horizon predicted force magnitude formed by extrapolating a low-pass filtered estimate with a fixed lookahead time. The clip operation bounds each update to +/- K_I along the contact direction, preventing force spikes or transient outliers from inducing large one-step jumps in the displacement.
The lookahead is important. Sensor filtering and impedance-controller response introduce lag between the commanded displacement and the resulting force. Without lookahead, the regulator consistently overshoots or undershoots, especially at contact onset. The predicted force allows the regulator to anticipate changes and respond more smoothly.
Zero-Shot Results: 90% Success Across Four Contact-Rich Tasks
The pipeline is evaluated on four tasks on a real Franka Panda: whiteboard wiping, carrot peeling, chocolate box stacking, and lamp button pressing. For each task, 10 videos are generated and one trajectory is derived from each, so every trial executes a distinct generation. The force-aware pipeline and a kinematic-only baseline (same trajectories, zero force throughout) are both evaluated on all 10 trajectories.
The kinematic-only baseline achieves 8/40 (20%) success, while the force-aware pipeline achieves 36/40 (90%). Wiping goes from 0/10 to 10/10, peeling from 5/10 to 9/10, stacking from 3/10 to 9/10, and pressing from 0/10 to 8/10. All executions remain below the 20 N force limit; reaching this limit triggers a protective stop and counts as failure.
Kinematic-only failures fall into two categories. Insufficient contact: the end effector passes a few millimeters above the surface in all wiping and pressing failures. Excessive contact: the peeler strikes the carrot in all peeling failures, or the gripper is not flush with the tabletop and crashes into the box instead of sliding under it in all stacking failures. The force-aware failures are more varied: peeling arm collision with the second gripper holding the carrot, insufficient lift height during box placement, and imprecise button pressing.
The key takeaway is that the audio-shaped force profile provides the right temporal structure for contact: it ramps up gradually at contact onset (avoiding the step-change overshoot that causes constant-force failures), varies intensity during sustained contact, and ramps down at release. This temporal structure matters more than the exact force value.
Force-Profile Ablations: Variable Beats Constant
A detailed ablation on whiteboard wiping and carrot peeling compares four conditions: variable force with F_max = 10 N, variable force with F_max = 15 N, constant 10 N, and constant 15 N. For variable profiles, the audio loudness determines the force variation within [F_min, F_max]; for constant profiles, the same extracted motion trajectory is used but the force is held fixed throughout the predicted contact interval.
On wiping, variable F_max = 15 N achieves 10/10 success with peak force 12.38 N and impulse 21.50 N*s, versus constant 10 N at 8/10 with peak 12.86 N and impulse 25.71 N*s. On peeling, both variable profiles achieve 9/10, constant 10 N achieves 8/10, and constant 15 N achieves only 6/10. The variable profiles achieve comparable or better success with lower peak force and lower impulse.
The critical difference is at contact onset. When the desired force steps abruptly from 0 N to 10 N or 15 N, the robot attempts to track this step command and the measured force overshoots to the 20 N safety limit, triggering a protective stop. The audio-shaped profile increases smoothly after contact, avoiding this overshoot entirely. This is visible in the force traces: the variable profile begins with a lower desired force and rises gradually, while the constant profile steps immediately to its setpoint.
The ablation also shows that both tasks tolerate a wide range of contact forces. Constant 10 N and both audio-shaped profiles succeed in most trials. The role of the profile is therefore to stay within the feasible range and to enter contact gradually, rather than to track a specific force trajectory. This suggests that the audio loudness provides useful timing information (when contact starts, when it intensifies) even if the exact force magnitude it encodes is approximate.
Prompt-Conditioned Force Ordering in Generated Audio
A controlled hammer-striking experiment tests whether the relative loudness of generated audio reflects the relative force magnitude specified in the task prompt. Starting from the same input image of a hand holding a hammer above a table, the generator is prompted with "Strike the table exactly twice with a hammer: first with F1 of force, then with F2," where (F1, F2) is either (5 N, 20 N) or (20 N, 5 N). Ten videos are generated for each ordering.
For each trial, the two contact events are detected and the peak loudness is computed over a 400 ms window for each. A trial is order-consistent when the louder strike corresponds to the higher prompted force, regardless of which strike occurs first. The generated audio preserves the specified force ordering in 18/20 trials (9/10 for each prompted order).
This result is significant because it establishes that the video-audio generator can follow a prompted force order and produce audio whose relative loudness reflects the intended force magnitude. The test is about the generator's ability to follow a force-ordering instruction in its audio output, not about the absolute physical accuracy of the loudness values. But it validates the core assumption of the pipeline: that generated audio carries useful force-structured information that can be extracted and used for control.
Force-Aware Data Generation for Policy Learning
Beyond single-trajectory execution, the pipeline serves as an automated data-generation engine for imitation learning. For each task, 50 force-aware rollouts that satisfy the success criterion are generated as demonstrations, with randomized object positions. A CNN-based Diffusion Policy is trained from wrist and external RGB images plus end-effector pose, without force input, predicting 16 absolute end-effector poses at 10 Hz and executing the first eight before replanning. A force-input variant additionally observes 3D contact force from the robot's joint torques, encoded by a small MLP and concatenated with the end-effector pose.
Policies are evaluated on 10 trials at random, previously unseen positions within the trained range, with novel marker strokes for wiping and different-sized carrots for peeling. The policy without force input achieves 29/40 overall. Adding force input raises this to 34/40, with the largest gains on peeling (8/10 to 10/10) and pressing (5/10 to 7/10), and no change in stacking.
The force input variant has an interesting effect on the measured force profiles. On wiping and pressing, it lowers the impulse (89.94 to 22.03 N*s and 8.71 to 5.33 N*s) at similar peak force, indicating shorter contact duration for the same outcome. On peeling, it increases peak force from 6.76 to 10.20 N, pressing harder to remove skin more reliably. On stacking, where success depends on sliding under the box and aligning it rather than force application, force input makes no measurable difference in success, peak force, or impulse.
Both policy variants maintain sustained contact that follows the demonstration profiles within roughly 2 N on wiping, pressing, and stacking. Peeling is the exception: both policies press harder than the demonstrations, with the force-input variant pressing hardest. This suggests that the pipeline's generated demonstrations provide a useful but not perfect force baseline, and that explicit force input allows the policy to deviate from the demonstrated force when needed.
Limitations and What Comes Next
The pipeline has several limitations the authors acknowledge. SAM-Audio may retain unrelated sounds, making the loudness profile an imperfect force proxy. The audio-shaped force profile is neither unique nor necessarily optimal, and a tuned constant force can perform comparably on sustained-contact tasks. The pipeline requires time to generate video and extract trajectories, so the scene must remain unchanged during this process, and the system does not yet support real-time adaptation to environmental changes.
The current pipeline also does not use real-time audio feedback during execution. The generated audio is used only offline to construct the force profile before the robot starts moving. Incorporating real-time audio feedback during execution could enable closed-loop force adaptation, where the robot adjusts its force based on the sounds it actually produces rather than the sounds predicted by the generator.
For future work, the authors suggest combining the pipeline with an AI agent that proposes its own task prompts, executes the generated videos, feeds the outcomes and measured forces back into the next round of generation, and iterates. This would make force-aware task execution and data collection self-iterating: the system generates, executes, evaluates, and refines without human intervention. Better depth grounding in the generated video would also reduce trajectory error, which is currently the primary source of failure in the kinematic-only baseline.
The pipeline and dataset are released at dreamingcontactsound.github.io. The Dreaming Contact Dataset comprises force-aware data across four tasks, including end-effector poses, force profiles, and two RGB views (external and wrist-mounted). This is the first publicly available dataset of force-aware trajectories generated from video and audio, and it provides a starting point for other researchers to build on the approach.
The broader contribution is conceptual. Audio from generated video is a cheap, scalable source of force-structured information that requires no force sensors, no simulation, and no physical demonstrations. The mapping from loudness to force is approximate, but combined with closed-loop regulation and geometric gating, it is sufficient for tasks where the timing and relative intensity of contact matter more than the exact force value. For contact-rich manipulation, the sound of contact may be more informative than the image of contact.
Read the paper on arXiv