Recent years have seen video generation models produce frames with striking visual fidelity and smooth temporal transitions. Still, producing videos that respect exact spatial layouts, maintain coherent motion over time, and permit scene-level control remains difficult. Diffusion-based systems operate directly on pixel sequences, learning motion statistics from data but without an explicit representation of 3D geometry or object relationships. When a prompt specifies a particular configuration — for instance, a car changing lanes from left to right — the resulting video may look plausible frame by frame, yet the relative positioning can drift or the maneuver may violate physical expectations. Correcting this without a persistent scene representation requires elaborate prompt engineering that is both fragile and costly.
Video-Coder introduces a different approach. Instead of generating pixels alone, the system trains an agent to write and execute code that constructs a dynamic 3D scene — referred to as a Draft Video — which then provides structure and motion signals to a downstream video generator. This separation means spatial constraints are encoded before any pixel generation occurs. If the produced code defines a valid scene graph, the subsequent video output inherits that organization. The agent learns through two complementary signals: whether the code executes without error, and how closely the rendered video matches the intended output.
Why pixel-level video generation struggles with structure and motion
Traditional diffusion models for video take as input a sequence of frames and predict intermediate states in pixel space. They capture low-level regularities such as color continuity and basic motion patterns, but they do not maintain an explicit model of object positions, camera trajectories, or logical constraints like "the ball must remain above the ground." When fine-grained control is required — for example, ensuring two characters maintain a specific distance throughout a clip — the model may produce plausible-looking results that nevertheless fail the spatial logic test. Retraining or prompt-tuning can shift these behaviors, but the approach lacks a systematic way to guarantee structural correctness. Video-Coder addresses this gap by inserting a code-mediated layer between intent and visual output.
Code-driven scene building as an alternative pathway
The agent constructs a scene by generating code that defines object poses, camera paths, and motion parameters. After execution, the resulting 3D configuration becomes the conditioning signal for a video generator that operates on the Draft Video representation. This two-step process — code execution first, rendering second — ensures that spatial constraints are established before any visual details are synthesized. If the code produces a valid scene description, the downstream generator receives a consistent foundation, and the resulting video reflects that consistency. The agent receives feedback from both the execution environment and a video-level reward that compares the generated clip to reference footage, enabling learning that spans syntactic validity and visual fidelity.
Curated datasets: Video-Coder-SFT-6k and Video-Coder-RL-2k
Training an agent of this capacity requires high-quality annotated data. The paper introduces two datasets. Video-Coder-SFT-6k contains six thousand prompt-video pairs used for supervised fine-tuning. Each entry pairs a natural-language description with the code that generates a matching 3D scene, allowing the model to learn the mapping from language to executable instructions. Video-Coder-RL-2k provides two thousand examples paired with reference videos, reserved for the reinforcement phase. Both datasets span a range of categories — human actions, object interactions, camera motions — and include ground-truth video for reward computation. The data pipeline extracts code from reference videos, verifies syntactic validity, and records execution outcomes such as rendering errors or successful scene construction.
The interactive environment that couples code execution with 3D rendering
Video-Coder Harness serves as the training and inference backbone. It exposes three interfaces: a code executor that validates Python-like instructions, a 3D renderer that outputs a scene description compatible with video generators, and a feedback module that reports execution errors or rendering anomalies. The agent operates in a loop: propose code, execute, observe result, update policy. During reinforcement, the feedback signal combines an execution reward — nonzero when code runs without error — and a video reward that measures similarity between the generated clip and the target footage. An annealing schedule starts with a high execution weight and gradually shifts emphasis toward the video reward, letting the agent transition from memorizing correct syntax to producing visually faithful content.
Supervised fine-tuning followed by annealing-scheduled reinforcement
Training proceeds in two stages. The first stage applies standard supervised fine-tuning on the SFT dataset, teaching the model to map prompts to executable code. Accuracy on the execution test set rises quickly because the code distribution is narrow and the mapping from prompt to instruction is relatively deterministic. The second stage switches to harness-native reinforcement learning. Here, the policy receives a combined reward signal. Early in this stage, the execution reward dominates, ensuring that generated code remains syntactically valid and scenes render without crash. As training progresses, the annealing coefficient reduces the execution weight, allowing the video reward to have increasing influence. This gradual shift prevents the agent from collapsing into a narrow set of safe but visually impoverished code patterns, and it is essential for the benchmark gains reported in the experiments.
Minimax-H3 gains on PhyGenBench and VBench-2.0
Evaluation on PhyGenBench reports an improvement of approximately seven points in the Minimax-H3 metric, which evaluates physical consistency across generated clips. On VBench-2.0, the gain is roughly twenty-four points, reflecting better alignment between generated motion and human-judged quality criteria. Both datasets contain a mix of motion types — translational, rotational, and interaction-based — and Video-Coder shows the largest improvements on categories that require precise object positioning over time. Ablation results indicate that removing the annealing schedule causes a drop in both metrics, confirming that the reward-shifting mechanism is a contributing factor to the observed performance.
Constraints the authors identify and directions for future work
The paper acknowledges several limitations. The 3D scene representation is abstracted into a compact code form, which may lose fine-grained texture details that a purely pixel-based model could capture. The approach relies on the availability of a compatible video generator that accepts the Draft Video format; without such a generator, the intermediate scene provides no direct benefit. The datasets, while diverse, do not cover every possible camera trajectory or lighting condition, so performance on extreme or novel scenarios may be reduced. Additionally, the annealing schedule is hand-tuned for the current model architecture; different backbone choices may require re-scheduling. These constraints suggest that code-driven video generation is a complementary rather than universal replacement for existing approaches, and they point to several directions for extension: supporting higher-resolution renderers, expanding the range of valid code constructs, and automating the annealing schedule search.
What this enables for developers working in video generation
For engineers building video content, the framework offers a programmable middle layer. Instead of iteratively refining text prompts, a developer can write or generate code that defines object placement and motion, run it through the Harness environment, and pipe the resulting scene to any compatible generator. This separation means that changes to spatial layout can be made by editing code rather than re-prompting an entire diffusion model. The open-sourced datasets and code also lower the barrier to entry: researchers can extract prompt-code pairs, train custom agents, or experiment with alternative reward functions. In domains where physical correctness matters — robotics simulation, architectural walkthroughs, or visual effects pre-visualization — the approach provides a path to more predictable output without abandoning deep generative models entirely. Read the paper on arXiv