Long video understanding on edge devices runs into a fundamental tradeoff. You can subsample visual tokens and hope the model still catches what matters, or you can convert the video into text captions and lose the fine-grained visual details that make answers verifiable. Neither approach works well for hours of content under tight compute budgets. Weitong Cai, Hang Zhang, Yukai Huang, Yiqiao Xie, Shan Gao, Jiankang Deng, Songcen Xu, Jifei Song, and Zhensong Zhang at Queen Mary University of London, Huawei, Imperial College London, and Durham University propose a third option: build the narrative structure once in language, and retrieve raw pixels only when the question specifically demands them.
The visual-textual duality that motivates CFD
The authors start with an observation that is simple but underappreciated. For image-level questions like "what color was the mug?", captions fall short of raw pixels. Fine-grained attributes such as identity, color, and spatial binding get lost under linguistic compression. A caption that says "a woman in a red dress" cannot distinguish between two different reds or confirm whether the dress had patterns. But for long-horizon temporal questions like "what happened after the man left?", language memories can actually outperform dense visual input. A compact textual record of events and their order is easier to reason over than hundreds of frames, because the text already provides a discrete, indexed timeline. The authors demonstrate this with controlled experiments: on CaptionQA, raw images score 92.32% versus 77.17% for captions, confirming that pixels matter for attribute perception. On InfiniBench's chronological understanding, a 60-second-per-caption text pipeline scores 56.46% versus 48.44% for the raw video baseline, confirming that language carries temporal structure more efficiently.
This duality is not a bug in current captioning models. It reflects a genuine difference in what text and pixels are good at. Text compresses events into discrete, searchable records that scale to hours of video without running out of context. Pixels preserve the raw evidence needed to verify visual claims. The question is how to combine them without paying the full cost of both.
Three-tier memory architecture
CFD answers this question with a three-tier memory built offline in a single captioning pass on the edge device. The key constraint is that all textual memories are produced once per video and cached for reuse across queries. No re-captioning happens at query time, which keeps the online cost bounded.
Tier 1 is the event memory, a global story skeleton. The system segments the video into semantically coherent events using TransNetV2 shot-boundary detection, merging adjacent segments shorter than 60 seconds into the preceding event. A lightweight captioning MLLM processes each event with a structured prompt that produces a narrative profile: scene overview, entity listing, chronological event flow, and retrieval anchor tags, all at 1 frame per second. The full event memory is always presented to the reasoning agent, serving as the global timeline and starting point for all question answering.
Tier 2 is the clip memory, a local micro-log. Independent of event segmentation, the video is partitioned into fixed 30-second clips, each captioned with a micro-action-level prompt designed for high temporal granularity: one action per sentence. These clip captions capture fine-grained details (micro-actions, state changes, visible text) that may be too detailed for the event-level skeleton. The clip memory is not fed to the reasoning agent in full. Instead, when the reasoning loop localizes a specific event, all clip captions overlapping with that event are injected as nested sub-descriptions within the event's narrative text. This injection is cumulative: once clip captions are added for an explored event, they persist in the narrative memory for all subsequent reasoning steps within the same query. This mechanism replaces the online re-captioning used in prior work like VideoLucy, where the system calls the MLLM again to generate finer descriptions. CFD composes already-cached captions through pure text operations at zero additional visual cost.
Tier 3 is the visual working memory, a FIFO queue of raw image frames. This is the pixel-level evidence tier, and it is deliberately small. The queue has a fixed capacity of 16 frames (for a one-hour video), orders of magnitude smaller than the total frame count. When the reasoning loop determines that pixel-level inspection is necessary, keyframes are adaptively sampled from the relevant event and pushed into the queue, with the oldest frames evicted when full. This compact design prevents excessive visual tokens from diluting the LLM's attention and minimizes the volume of visual data transmitted from edge to cloud.
The Visual-Need Router: gating visual access on question type
The central innovation is the Visual-Need Router, a per-query gating module that classifies each question as perceptual or temporal-structural. When the reasoning loop cannot answer confidently from textual memory alone, the router decides whether pixel-level inspection would actually help. It accepts (triggers frame extraction) for questions targeting visual attributes: appearance, on-screen text, spatial layout, object disambiguation. It declines (skips frame extraction) for questions about temporal ordering, scene transitions, or long-range narrative structure, where the textual memory is already sufficient.
When the router declines, it clears the visual working memory and directs the loop to the next iteration without any visual cost. This is not a downstream perception module but an explicit cost-control gate. The router deliberately routes some questions away from frames to keep the per-query frame budget tight. Applications that prioritize attribute fidelity over visual cost can simply disable the router and always route to frames. The authors report both configurations in their ablations.
The router retrieves frames for 46.2% of questions and leaves 53.8% entirely in language space. Despite this selective gating, it captures 95.5% of the cases (127 out of 133) where frames empirically improve the answer. The 4.5% of missed cases represent the cost of the router's binary classification: questions that fall on the boundary between perceptual and temporal-structural, where a continuous confidence score might help.
The story-first reasoning loop
The reasoning loop orchestrates the four agent roles under a controlled visual budget with a maximum of 5 iterations. It starts with a story-first answer attempt: the Answering Agent receives the full event memory and the empty visual working memory, and tries to answer from the event skeleton alone. If confident, the answer is returned immediately, often requiring only a single LLM call. This early exit handles temporal and narrative questions efficiently.
When confidence is insufficient, the Localization Agent selects the most relevant event from the narrative, excluding previously explored events. The system then enriches that event by injecting overlapping clip captions as nested sub-descriptions, and the Answering Agent re-attempts with this enriched memory. If still not confident, the Visual-Need Router decides whether to retrieve raw frames. When it accepts, keyframes are sampled from the event into the fixed-capacity FIFO queue, and the Answering Agent performs multimodal re-reasoning over both the enriched textual memory and the visual evidence. The loop repeats until confidence is reached or the budget is exhausted.
The key design property is that each iteration makes the system's memory progressively richer without ever re-captioning the video. New events are explored for breadth, their clip details are permanently injected for depth, and visual evidence accumulates for grounding. All of this happens through text composition and bounded frame extraction, not through repeated MLLM calls to the video.
Results: competitive accuracy at 116x fewer frames
On Video-MME, CFD achieves 67.5% overall accuracy with an average of 5.8 frames per question. The dense baseline Qwen3-VL-32B reaches 75.9% but uses 768 frames per question, more than two orders of magnitude more. Among agent-based systems, VideoLucy reaches 64.7% with re-captioning at query time, MemVid reaches 64.0% with 1 fps uniform sampling, and VideoTree reaches 60.6%. CFD matches or exceeds all of them while using roughly an order of magnitude fewer frames per question. The gains are most pronounced on medium and long videos (66.6% and 63.6%), where re-captioning and dense sampling suffer from context saturation and redundant visual processing.
On InfiniBench, CFD achieves the best Chronological Understanding score at 55.10%, exceeding the 768-frame dense baselines Qwen3-VL (48.44%) and Qwen2-VL (48.41%), at only 14.8 frames per question. This is the task most aligned with the design principle: ordering events across a full video is a temporally structured task where the event-level narrative skeleton provides a compact discrete record that outperforms exhaustive visual sampling. On Scene Transitions, CFD remains competitive at 52.40% versus 53.97% for Qwen3-VL. On Global Appearance and Character Actions, CFD is ahead of agent-based baselines but below the dense Qwen3-VL, consistent with the visual-textual duality: fine-grained attribute binding benefits from dense pixel access that a tight on-demand budget does not fully substitute.
Ablation study: what each component contributes
The component analysis reveals how the pieces interact. Event memory alone scores 57.3% overall, too sparse for most questions. Dense clip memory alone recovers most of the gap (65.5%) but Global Appearance stays at 38.0%, confirming that text alone cannot substitute for visual evidence. Adding the reasoning loop over the dual-track memory without any frames scores 65.3%, nearly matching the clip-only baseline, which corroborates that language memories are a strong carrier for temporal structure. Enabling on-demand frames without the router brings sharp gains on appearance-centric tasks (Global Appearance jumps from 45.0% to 62.8%, Character Actions from 53.4% to 65.9%) at the cost of Chronological Understanding (drops from 52.4% to 51.7%) and long-video accuracy (63.0% to 62.6%). Indiscriminate frame injection dilutes the narrative context on temporally structured questions. The full model with the Visual-Need Router recovers Chronological Understanding to 55.1% and long-video accuracy to 63.6% while keeping the average frame count low, achieving a more balanced tradeoff than always-on frame retrieval.
The budget sensitivity analysis shows that overall accuracy varies by only 0.4% across an 8x range of per-event frame budgets. The middle setting (8 frames per event, 16-frame FIFO) is within 0.1% of the largest budget at one-quarter the visual cost. Long videos benefit most from increased budget (63.6% to 64.2%), consistent with sparser event memory leaving more room for visual verification. Even when the router is disabled and every localized event is routed to frames, the caption-once memory, temporal localization, iterative backtracking, bounded frame extraction, and fixed-capacity visual working memory reduce online frame usage from 768 to 6.63 frames per question, a 116x reduction. Within this already sparse pipeline, the Visual-Need Router further reduces frames from 6.63 to 5.76 while improving accuracy from 67.2% to 67.5%.
Captioner model size and the verbosity trap
Sweeping three Qwen3-VL captioner sizes reveals a counterintuitive finding. Scaling from 2B to 8B brings a substantial gain (62.6% to 67.5%), but scaling further to 32B marginally helps short and medium videos while degrading long-video accuracy (61.0% vs. 63.6%). The authors attribute this to verbosity accumulation: larger captioners produce longer, more detailed descriptions per segment, which compound across the many events of a long video and inflate the total narrative context beyond what the reasoning agent can attend to within a fixed inference budget. More information is not always better when the reasoning agent has a finite context window. The 8B captioner hits the sweet spot between indexing cost, caption conciseness, and downstream accuracy.
Limitations the authors acknowledge
The framework is fundamentally bottlenecked by offline captioning quality. If question-critical visual details are not captured in the question-agnostic memories, purely text-based backtracking cannot recover them. Sparse visual backtracking partially helps, but its effectiveness depends on correctly diagnosing why text-only reasoning failed and routing to the right event. Bounded keyframe extraction may also miss highly transient cues, small objects, or brief textual overlays that fall outside the sampled frames.
The event segmentation relies on shot-boundary detection, which works for edited video (TV shows, documentaries, lectures) but not for continuous egocentric or wearable recordings where hard cuts are rare. The Visual-Need Router operates as a binary zero-shot classifier, which mis-routes queries on the boundary between perceptual and temporal-structural categories. A continuous confidence score trained from routing data would likely improve per-skill balance. The edge-cloud setup is validated under simulated separation rather than physical deployment, so the practical impact of device-side compute constraints, communication overhead, and network latency variability remains to be characterized on representative edge hardware.
What this means for developers building video understanding systems
CFD demonstrates that competitive long-video understanding does not require saturating the context window with visual tokens. The design principle is allocation, not reduction: spend visual access where it matters and rely on language for the rest. A developer building a video assistant, content moderation system, or personal memory application could adopt the three-tier memory as a reusable infrastructure: caption once offline, serve queries cheaply from cached text, and retrieve pixels only when the question demands it.
The Visual-Need Router is the piece most immediately transferable to other systems. Any pipeline that combines text and vision for video understanding faces the same question: when do I actually need to look at the pixels? Making this decision explicit and query-conditioned, rather than an emergent property of pipeline depth, gives the developer a tunable knob for the accuracy-efficiency tradeoff. The 99.1% frame reduction from the caption-once architecture alone, before the router even activates, shows that the bottleneck in current systems is not insufficient visual understanding but redundant visual processing.
The verbosity trap with larger captioners is a practical warning: when building offline indexing pipelines, conciseness matters more than detail. A 32B captioner that produces richer descriptions per segment can actually hurt long-video performance because the accumulated text overwhelms the reasoning agent's context window. For long-form content, a smaller, more disciplined captioner is the better engineering choice.
Read the paper on arXiv