Redefining audio-visual caption evaluation as a deep-structured diagnostic



Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio-visual reasoning, creating an urgent need for evaluations that expose their capability limits. Audio-visual captioning is an ideal diagnostic task, yet current benchmarks face a coupled trade-off: whole-caption scores provide coverage without localization, local probes provide localization without coverage, and unconstrained LLM judges introduce instability.



From free-form text to atomic evaluation units



We introduce OmniCapBench (Omni-Video Caption Benchmark), a benchmark that reframes audio-visual caption evaluation as a deep-structured diagnostic framework. OmniCapBench shifts the prediction target from free-form text to sets of atomic, verifiable evaluation units across three tracks: entity references, visual shots, and audio events, enabling reliable scoring with deterministic constraint checks and localized LLM-based semantic comparisons.



Three tracks of fine-grained assessment



The entity references track identifies and evaluates whether the model correctly names objects, actions, or attributes present in the audio-visual input. The visual shots track assesses whether the model grounds its descriptions to specific time intervals or segments within the video. The audio events track evaluates whether the model accurately transcribes or describes sounds that occur synchronized with visual events.



Each track produces evaluation units that are simple enough for deterministic checking—such as exact string match for entity names, overlap interval checks for visual shots, or presence/absence tests for audio events—while also supporting localized LLM-based semantic comparisons when exact matching is too strict. This dual approach eliminates the instability that arises from unconstrained LLM judges while preserving the nuance of semantic evaluation.



786 densely annotated videos



OmniCapBench includes 786 densely annotated videos, each labeled with ground-truth entity references, visual shot boundaries, and audio event labels. This scale allows the benchmark to distinguish subtle perception errors that smaller collections would miss. The annotated errors include temporal grounding failures, where the model misplaces an event in time; identity drift, where the model consistently misidentifies an entity across shots; cross-modal misalignment, where the described audio does not match the visualized event; and hallucinated descriptions, where the model claims the presence of entities or events not present in the input.



Empirical findings from frontier MLLMs



Evaluating frontier MLLMs on OmniCapBench reveals a consistent pattern. These models show strong local perception—correctly identifying entities and grounding to visual shots—but weak long-horizon audio-visual reasoning. The greatest weaknesses appear in identity drift and cross-modal misalignment. Models frequently fail to maintain entity identity across multiple visual shots, and they often describe audio events that have no visual counterpart or vice versa. These results provide a fine-grained roadmap for omnimodal development, indicating where current architectures and training objectives need improvement.



Limitations and future directions



The benchmark currently focuses on short-form video clips with synchronized audio. Extending OmniCapBench to longer video sequences, multi-modal inputs without audio, or interactive audio-visual tasks would broaden its applicability. The atomic evaluation units can be adapted to other modalities, and the framework's deterministic scoring pipeline could be extended to evaluate other reasoning tasks beyond captioning.



Conclusion



OmniCapBench reframes audio-visual caption evaluation as a deep-structured diagnostic framework. By shifting the prediction target to atomic, verifiable units across entity references, visual shots, and audio events, the benchmark enables reliable scoring with deterministic constraint checks and localized LLM-based semantic comparisons. With 786 densely annotated videos, OmniCapBench effectively distinguishes MLLM perception errors including temporal grounding failures, identity drift, cross-modal misalignment, and hallucinated descriptions. Evaluating frontier MLLMs reveals strong local perception but weak long-horizon audio-visual reasoning, particularly in identity drift and cross-modal misalignment, providing a fine-grained roadmap for omnimodal development. Read the paper on arXiv