When Video Generation Needs to Listen to More Than Words

Text-to-video models have gotten good at translating prompts into motion. But professional creative work rarely starts from a blank slate. A designer hands you a character sheet and says "animate this." A director shows you a reference clip and says "match this camera move." A storyboard artist gives you a rough layout and says "fill in the details." The reference is not a caption. It is a constraint that the model must understand, preserve, and route to the right part of the output.

This is reference-to-video (R2V) generation, and it is moving fast. Models now accept images, videos, and compositional combinations as control signals. But the benchmarks keeping up with this shift are lagging. Existing evaluations focus on content references, the easy case of "put this object in this scene," and measure overall similarity with a single holistic score. They miss the harder question: did the model preserve the right factor, route it to the right target, and realize the instruction on top of it?

OmniVBench, from Tencent's Online Video BU and HKUST Guangzhou, addresses this gap with both a benchmark and a training dataset designed for the full R2V task space. The benchmark covers 7 task families and 18 fine-grained tasks across content, motion, style, structure, narrative, and multi-reference settings. The evaluation protocol decomposes each case into case-specific checklists that assess whether intended reference factors are faithfully preserved, correctly disentangled and bound to their targets, and properly realized according to the instruction. 12,172 checklist items total.

The Reference Taxonomy

The benchmark organizes R2V tasks along five dimensions under single-reference settings, plus two multi-reference extensions.

Content reference covers object identity, material, pattern, and part-level reference. Character reference spans single-view, multi-view, customized character (edit specified attributes while preserving identity), and attribute-grounded (identify the target through a spatial description in a multi-character scene). Scene reference evaluates environmental identity and layout while allowing instructed changes to foreground content.

Motion reference uses video references to transfer temporal dynamics while changing appearance and context. Action reference tests whether subject motion can be disentangled from identity, background, and camera movement. Camera-motion reference transfers only the viewpoint trajectory, covering pan, tilt, dolly, truck, pedestal, orbit, whip-pan, dolly-zoom, and compound movements. These tasks are harder than they sound because the model must figure out which source of motion to extract.

Style reference pairs a reference image with an instruction that specifies target content while leaving the style undescribed. The references span hand-drawn, painterly, animation, craft, digital, graphic, and cinematic styles. The model must disentangle visual style from semantic content and transfer the intended style.

Structure reference provides temporally ordered guidance with incomplete appearance details. Greybox, line art, and rough storyboards serve as references, requesting 2D animation, 3D animation, or live-action outputs. The model must recover the structure while completing missing appearance and surface details.

Narrative reference tests whether a model can extract story structure and follow, transform, or continue it. Multi-panel storyboard reference requires inferring character relations, event order, and transitions between panels. Story reference uses an existing video as a narrative template, requiring the model to re-imagine characters, setting, and appearance while preserving the storyline. Preceding-shot reference provides context and asks for a coherent next shot.

Multi-reference settings extend these to compositional scenarios. Multi-content reference uses separate references for characters, objects, and scenes, or introduces references containing multiple candidate entities that the model must locate and selectively transfer. Cross-aspect reference combines references controlling different aspects, such as content with motion, style, structure, or narrative, testing factor disentanglement and cross-reference integration.

Factor-Grounded Evaluation: Checklists, Not Single Scores

The evaluation protocol is the paper's most novel contribution. Existing R2V benchmarks rely on fixed similarity metrics or holistic VLM judgments of reference consistency. OmniVBench decomposes each case into fine-grained checklists grounded in its references and instruction.

Each case is evaluated along three dimensions. Reference Fidelity measures how faithfully designated reference information is preserved or transferred, covering content, structure, motion, style, and narrative fidelity. Only relevant sub-dimensions are evaluated per case. Changes explicitly specified by the instruction are not counted as fidelity errors.

Instruction Realization measures whether operations specified by the instruction are correctly realized on their intended targets, split into Reference-Factor Disentanglement and Routing (whether the intended factor is correctly extracted and assigned to the correct target) and Target Compliance (whether the instruction's requirements are correctly realized).

Video Quality evaluates output independently of reference fidelity and instruction compliance, using Technical Quality, Aesthetic Quality, and Physical Plausibility scores from DOVER++, Aesthetic Predictor V2.5, and UnifiedReward 2.0.

For Reference Fidelity and Instruction Realization, each sub-dimension has a fixed set of evaluation criteria. Given a sample, only the relevant criteria are instantiated as atomic checklists. A VLM (Gemini-3.1-Pro) evaluates each checklist separately, and scores are aggregated hierarchically so that each active sub-dimension receives equal weight. All checklists and reference-target mappings are manually verified to remove ambiguity and are fixed across all model outputs for the same case.

The protocol avoids the problem of holistic scores masking specific failures. A model might achieve high overall similarity while completely failing to route a motion factor to the correct target. Factor-grounded checklists catch this because each reference factor gets its own evaluation question.

The Omni-R2V Dataset: 340K Samples from Professional Footage

Training data for general R2V generation is scarce. Existing datasets are designed for specific tasks or individual reference types, with fragmented coverage across the broader R2V landscape. Some do not provide processed reference-video pairs, requiring users to download and process raw source video themselves.

The Omni-R2V Dataset draws primarily from professional video footage and comprises 339,570 processed training samples across 7 task families, covering 2D animation, 3D animation, and live-action. Clip durations extend to 20 seconds, resolutions range from 480p to 2160p+, with 1080p and 2160p+ accounting for the majority.

The construction pipeline has three stages. Source video collection segments professionally curated footage into temporally coherent clips and filters for visual quality, event completeness, and suitability. Task-specific pair construction determines required reference roles and reference-target relationships for each target video, using two strategies: cross-pair matching (selecting reference clips or frames from existing videos according to the relationship required by the task) and inverse construction (deriving references from the target through extraction or generation, preserving task-relevant information while modifying other aspects). Instruction generation uses Gemini-3.1-Pro to caption references and targets separately, then DeepSeek-V4-Pro to convert descriptions into training instructions using task-specific rules.

The dataset provides processed samples directly, unlike OpenS2V-5M which requires downloading and processing raw source video. This makes it immediately usable for R2V training without additional preprocessing.

What the Evaluation Reveals

The authors evaluate 11 models, 5 closed-source (Kling 3.0 Omni, Seedance 2.0, Seedance 2.5, Gemini Omni, Happy Horse 1.0, Vidu-Q2-Pro) and 5 open-source (UniVideo, OmniWeaving, LoomVideo, Bernini, MiniMax H3), across all 7 task families.

The performance gap between open- and closed-source models has narrowed. MiniMax H3 achieves 72.41 overall, competitive with the closed-source leaders (Seedance 2.5 at 72.68, Gemini Omni at 70.76). But no model consistently performs strongly across all task families. Content reference scores are generally higher, while motion, style, structure, narrative, and multi-reference settings show larger performance differences.

The three evaluation dimensions reveal distinct profiles. Strong Reference Fidelity does not guarantee strong Instruction Realization. A model might faithfully preserve reference factors but fail to route them correctly to their intended targets. A model might follow instructions well while poorly preserving reference information. Video Quality correlates weakly with the other two dimensions, confirming that visual quality alone is insufficient for R2V evaluation.

Motion reference tasks show the largest performance gaps. Action disentanglement, separating subject motion from identity and background, remains challenging. Camera-motion transfer, extracting only the viewpoint trajectory without the original subject's motion, is even harder. These tasks require the model to understand which source of motion to extract and which to discard, a capability that current models handle inconsistently.

Structure reference tasks, where the model must recover appearance from incomplete structural guidance (greybox, line art, storyboards), also show significant gaps. The model must infer missing details while preserving spatial composition, poses, and action timing. Narrative reference tasks, requiring story understanding and continuation, show the lowest scores across most models.

What This Means for Video Generation

The paper identifies a clear trajectory for R2V research. Content references are relatively well-handled. The frontier is in motion, style, structure, and narrative references, and especially in multi-reference compositional settings where the model must extract, disentangle, and route multiple factors simultaneously.

The factor-grounded evaluation protocol is a methodological contribution beyond video generation. Any domain where the model must selectively preserve and route information from multiple references can benefit from checklist-based evaluation that decomposes the task into atomic assessment items. The alternative, holistic similarity scores, consistently masks specific failures while rewarding superficial alignment.

The 340K training dataset and the task-specific construction pipelines lower the barrier for R2V research. The pipeline is designed to be scalable: cross-pair matching and inverse construction can be applied to new video corpora, and the instruction generation step uses existing LLMs. For researchers who cannot access professional video footage at Tencent's scale, the pipeline provides a practical recipe for building similar datasets from smaller collections.

The benchmark and dataset are released, with the evaluation protocol, checklists, and scoring code. For developers building video generation systems, the immediate takeaway is that R2V evaluation needs to go beyond "does the output look like the reference." The question is whether the model understood which factors to preserve, where to route them, and how to compose them with the instruction. OmniVBench gives the community a way to measure that.

Read the paper on arXiv