Imagine watching a person ride a skateboard past your field of view. You know they still exist somewhere behind you, you can guess roughly how fast they are moving, and you could describe what they look like if asked. This is visual memory, and it comes so naturally to humans that we barely notice it. But for 4D foundation models, the systems being built to reconstruct and generate dynamic 3D scenes, this seemingly trivial ability turns out to be an unsolved problem. Guangzhao He, Hadar Averbuch-Elor, and Wei-Chiu Ma at Cornell University introduce PersistBench, a benchmark that reveals a stark truth: current 4D models can see the world, but they cannot remember it.

Why Existing Benchmarks Miss the Point

The rapid development of 4D foundation models, which process video input and output dynamic 3D scene representations, has produced systems that can synthesize convincing novel views of moving objects. Benchmarks like VBench, WorldScore, and standard reconstruction metrics (PSNR, SSIM, LPIPS) evaluate these models on perceptual quality and pixel-level consistency. But there is a fundamental problem with all of these evaluations: they assume the target object stays visible throughout the test.

When a car drives out of frame in a standard video, its ongoing state is no longer captured by any camera. Without ground truth for what happens to that car after it exits, you cannot verify whether a model correctly maintains its existence, position, motion, or appearance. Existing benchmarks are reference-free in this respect: they judge visual plausibility without checking whether a specific entity was faithfully retained. This means they measure rendering fidelity, not memory.

The distinction matters. A model could produce beautiful novel views of an object that remains in frame while simultaneously having no internal representation of what that object looks like once it disappears. The ability to reconstruct observed content and the ability to remember unobserved content are different capabilities, and conflating them gives a misleading picture of model readiness for real-world deployment.

Using 360-Degree Video as Omniscient Ground Truth

The core technical challenge PersistBench solves is sourcing ground truth for what happens outside a camera's field of view. The authors' insight is to use 360-degree videos, which capture the full surrounding environment at every timestep. From a single equirectangular recording, you can crop perspective views at any camera angle after capture. This means you can construct an input video where a target object is initially visible and then leaves the frame, while simultaneously extracting a reference video where a virtual camera follows that same object, keeping it visible throughout.

The pipeline works as follows. Given a 360-degree video with estimated camera poses, the system identifies salient objects using a vision-language model (Qwen3-VL-Plus), then tracks them across the entire video using SAM3 with a workaround for equirectangular boundary crossings (the canvas is horizontally duplicated so objects crossing one edge remain trackable near the center). Camera trajectories for both the input and reference sequences are then optimized as B-splines using L-BFGS, balancing visibility constraints with kinematic smoothness by minimizing velocity, acceleration, and jerk.

From an initial pool of 24,000 raw 360-degree clips drawn from the 360-1M dataset, this process yields 2,000 high-quality paired evaluation sequences, split evenly between dynamic and static objects. The clips are categorized into 10 object types including human, vehicle, animal, structure, and furniture. Each clip is approximately 10 seconds long, with a median visible segment of 2.62 seconds and an invisible segment of 2.50 to 2.62 seconds.

Three Metrics That Decompose Visual Memory

PersistBench evaluates visual memory along three complementary axes, each grounded in cognitive science research on how humans represent the physical world.

Object permanence asks whether an object still exists in the model's representation after it leaves the field of view. The metric uses a dual-verification mechanism: SAM2 tracks the target object across the generated frames, and a vision-language model (Qwen3.8-27B) confirms the target's semantic presence. A frame counts as successful only if both the tracker produces a valid mask and the VLM verifies the object. The final score is the percentage of valid frames across the sequence.

Motion continuity measures whether the model maintains consistent spatial trajectories for moving objects. For each frame where the object is successfully tracked, the L2 distance between the predicted and ground-truth centroids is computed, normalized by the reference object's average area, and transformed through exponential decay with a temperature parameter of 1.0. The result captures how well the model preserves the object's motion state across time.

Appearance preservation evaluates fine-grained visual identity using DINOv2 features rather than raw pixels. For each tracked frame, feature maps are extracted from both the generated and reference images, masked to the target object, spatially averaged into a single vector, and compared via cosine similarity. This approach is robust to minor pose shifts and partial occlusions that would confound pixel-level metrics.

A human study with 864 comparisons across all 66 method pairs confirmed the metrics' validity. Object permanence and appearance preservation showed strong agreement with human judgments (Spearman correlations of 0.944 and 0.986, respectively), while motion continuity showed moderate agreement (0.566), reflecting the inherent ambiguity in evaluating motion predictions.

Twelve Models, One Unmistakable Pattern

PersistBench evaluates 12 publicly available 4D foundation models across three categories. The 4D reconstruction models (CUT3R, 4DGT, CogNVS, NeoVerse) process input video and reconstruct the underlying dynamic 3D scene. The camera-controlled video models (ReCamMaster, TrajectoryCrafter, GEN3C, HyDRA) generate novel views given target camera trajectories. The video-to-360-degree models (ViewPoint, Imagine360, Argus, CubeComposer) predict panoramic video from perspective inputs, which can then be cropped to perspective views for comparison.

The results tell a consistent story across every model and every metric: performance collapses once objects leave the input field of view. The invisible-segment scores are dramatically lower than visible-segment scores in every case. Consider the object permanence drops for dynamic objects: CUT3R falls from 43.99% visible to 0.99% invisible, 4DGT from 96.21% to 3.89%, Imagine360 from 98.14% to 25.84%. Even the strongest model on this metric, GEN3C, drops from 96.13% to 87.06%.

The pattern holds for motion continuity and appearance preservation as well. For dynamic objects, TrajectoryCrafter's motion continuity drops from 97.01% visible to 56.45% invisible. Argus's appearance preservation drops from 85.20% to 49.31%. These are not marginal differences. The models are substantially worse at remembering what they have seen than at rendering what they can currently see.

What Separates Good Memory from Bad Memory

Despite the universal degradation, the magnitude of the gap varies significantly across models, and the differences correlate with architectural choices. The authors identify that models using explicit conditioning, where input observations are projected onto target views before inpainting, consistently outperform models relying on implicit memory representations.

GEN3C achieves the highest dynamic object permanence at 87.06% (invisible), followed by TrajectoryCrafter at 77.80% and NeoVerse at 54.31%. These three models all use explicit geometric conditioning. By contrast, CUT3R, which maintains a persistent state internally but without explicit geometric projection, scores only 0.99% on dynamic object permanence for invisible segments. The gap between GEN3C and CUT3R on this metric is 86 percentage points.

The authors attribute this to a training data bias. Most 4D models are trained on videos where target objects remain continuously visible throughout. In such data, the model never needs to reconstruct an entity it cannot see, so it learns to interpolate from observed inputs rather than building a persistent internal representation. Explicit conditioning sidesteps this limitation by encoding object persistence directly into the input, reducing the generation task to a simpler refinement problem.

Memory is also multi-faceted. While object permanence and appearance preservation are strongly correlated (r = 0.93), motion continuity is only weakly correlated with either (r = 0.34 with permanence, r = 0.54 with appearance). This means a model can be good at keeping objects alive while being poor at tracking their motion, or vice versa. GEN3C and 4DGT both excel at motion continuity but differ sharply on permanence. NeoVerse and Imagine360 track dynamics similarly but diverge on appearance preservation. No single metric captures the full picture.

Static Objects Are Easier, But the Lessons Transfer

Every model performs better on static objects than on dynamic ones. This is intuitive: a stationary object does not change position or appearance, making it easier to maintain in memory. But the authors find that performance on static and dynamic subsets is strongly correlated across all metrics. A model that remembers static objects well tends to remember dynamic objects well too.

This correlation suggests that models may rely on shared underlying mechanisms for both object types. Improvements in static memory are likely to translate to dynamic memory, which simplifies the path forward for data curation and model refinement. Training on videos where static objects frequently leave and reappear the frame would provide a useful stepping stone before tackling the harder case of dynamic objects.

360-Degree Models: A Cautionary Tale

One might expect that models trained to predict full panoramas would inherently learn to remember, since 360-degree video captures the entire environment. The results partially support this. Argus and CubeComposer perform well on static subsets. Argus achieves the highest static object permanence (84.06%) and motion continuity (80.60%) among all evaluated models, despite being built on an older pretrained backbone.

But these same models suffer large performance drops on dynamic subsets. Argus's dynamic object permanence falls to 62.42%, and CubeComposer's drops to 45.29%. The authors attribute this to the training data distribution: these models are predominantly trained on scenes with few dynamic objects or simple camera trajectories, making complex dynamic evaluation scenarios out-of-distribution. Argus performs better than the others in this category because it was explicitly trained on more dynamic 360-degree scenes and models camera noise during training.

The lesson is that 360-degree data is a powerful resource for both evaluation and training, but simply having panoramic input does not guarantee memory. What matters is whether the training data includes frequent occlusion, reappearance, and dynamic entity tracking.

Limitations and What Comes Next

The evaluation pipeline relies on off-the-shelf models for pose estimation (ViPE), object tracking (SAM2, SAM3), VLM-based verification (Qwen3.8-27B), and feature extraction (DINOv2). Errors in any of these components can affect measured memory scores independently of the evaluated model's actual capabilities. The authors show through ablations that method rankings are largely preserved when individual scoring components are replaced, suggesting limited sensitivity to these choices, but the dependency remains.

The dataset itself is drawn from YouTube 360-degree videos, so the scene distribution reflects what is available on that platform. Extending to other sources and less represented environments could broaden coverage and reveal different failure modes.

For practitioners building with 4D foundation models, the implications are clear. If your application requires maintaining awareness of entities that leave the camera's view, such as autonomous driving systems tracking vehicles that pass out of the sensor range, or robotics systems maintaining world models during occlusion, current models are not reliable. The gap between seeing and remembering is large, and explicit geometric conditioning is the strongest available mitigation.

The authors propose two directions for closing this gap: curating training data with frequent occlusion and reappearance events, and developing architectures that combine explicit geometric conditioning with the generative priors of video models. PersistBench provides the measurement tool to track progress on both fronts.

Read the paper on arXiv