Joint audio-video generation models can produce scenes with multi-event composition, multi-layer audio mixture, and asymmetric cross-modal relations. However, existing evaluation protocols largely reuse metrics from earlier simple and unidirectional tasks. These approaches typically assess an audio-video clip as a whole and rely on simple predefined relations, which makes them unsuitable for the complex scenes modern models generate.
We propose an event-centric evaluation framework that disentangles audio and visual events at the event level. The framework analyzes each event independently, cross-validates relations across modalities, and aggregates the judgments into a coarse-to-fine, three-level consistency metric. This metric assesses global narrative harmony, event-level temporal synchronization, and fine-grained attribute alignment. The formulation is general and applicable across diverse generation settings and cross-modal relations.
Metric design and three-level aggregation
The coarse level evaluates whether the overall narrative holds together across audio and visual streams. The middle level checks temporal synchronization between individual events, such as whether a visual action aligns with a corresponding sound. The fine level examines attribute alignment, including matching object identities, spatial positions, and audio-visual sound properties. By separating these three layers, the framework can pinpoint where a generated clip deviates from a consistent audio-visual scene.
We instantiate this framework as AVC-Reward, a trainable reward model. The model is trained to predict the three-level consistency scores, allowing it to serve as an automated signal for audio-video quality assessment.
Benchmark construction and evaluation
< p>AVC-Benchmark is constructed from diverse positive and negative samples generated by various audio-video models. This benchmark enables assessment of how well automatic judgments agree with human perception across different generation paradigms and relation complexities. The benchmark includes samples that exercise the three forms of complexity mentioned: multi-event composition, multi-layer audio mixture, and asymmetric cross-modal relations.Results show that AVC-Reward achieves state-of-the-art alignment with human judgment among automatic methods. The model also provides interpretable rationales that localize failures, indicating which event or attribute contributed to a low consistency score. When used as an online reinforcement-learning signal, AVC-Reward improves the audio-video consistency of a joint generation model.
Limitations and practical considerations
The framework relies on accurate event disentanglement, which can be challenging when audio and visual events overlap or are loosely correlated. The three-level metric, while granular, adds computational overhead compared to whole-clip evaluation. The benchmark construction depends on the diversity of generated samples; coverage of edge cases may be limited without continued expansion. Additionally, the reward model’s performance is contingent on the quality of the event detection and representation pipeline.
Practical takeaways
< p>Developers working on joint audio-video generation can integrate AVC-Reward as a reward model in training loops to shape consistency directly. The coarse-to-fine metric structure also offers a diagnostic tool: by examining which level of the metric degrades, practitioners can identify whether the issue lies in global narrative, temporal synchronization, or fine-grained attribute matching. The AVC-Benchmark provides a testbed for evaluating new evaluation metrics against human judgment. For systems that need to operate under compute constraints, the event-level analysis can be selectively applied to the most critical segments of a clip.Read the paper on arXiv