Late-interaction retrieval has become the dominant paradigm for searching visual documents. Models like ColPali embed every image patch on a page as a separate vector and score queries against them using MaxSim, a per-token maximum inner product summed across all query tokens. This design preserves local evidence like individual table cells or figure captions, and it is provably more expressive than collapsing a page into a single vector of comparable size. The catch is storage: ColPali produces 1,031 patch vectors of dimension 128 per page, roughly 258 KB in bfloat16. Scale that to a million pages and you are looking at a quarter terabyte of embeddings before any index structure is added. The community has named index footprint as one of the central open problems for the paradigm.
Existing compression methods attack this cost by retaining a subset or local average of the encoder's vectors. Token pooling merges nearby vectors by hierarchical clustering, pruning keeps only salient patches, and merging schemes combine both. These approaches work well at moderate budgets, retaining 97% or more of quality when storing roughly 100 vectors per page. But they all share a floor: none reports operating below roughly 16 vectors per page without retraining the encoder, which invalidates every embedding already computed and demands substantial GPU time. At aggressive storage budgets, the quality drops sharply and there is no path forward without giving up the frozen checkpoint.
Two geometric properties that change the compression game
A team at KAUST and Edge Hill University started from a different question: what is the stored object, geometrically? They measured page embeddings from ColPali across all ten ViDoRe v1 evaluation corpora and found two consistent properties. First, the 1,031 vectors per page lie exactly on the unit sphere in 128-dimensional space, because ColPali L2-normalizes its outputs. Second, these vectors concentrate near a manifold of intrinsic dimension 5 to 6. The TwoNN estimator, a standard tool for measuring intrinsic dimensionality from local neighborhood structure, gives a median of 4.9 across all pages, with per-corpus medians ranging from 4.7 to 5.1.
This is a striking result. A page with over a thousand vectors in a 128-dimensional space actually lives on a surface with only five or six degrees of freedom. The researchers confirmed the finding holds across three different encoders: ColPali (intrinsic dimension 4.9, ambient 128), ColQwen2 (5.1, ambient 128), and Nemotron v2 (6.1, ambient 3,072). Despite the ambient dimension varying by 24x, the intrinsic dimension varies by only about one. This is not an artifact of the estimator. For comparison, a Gaussian distribution fitted to each page's own covariance matrix reads 32.2, and uniform noise of the same size reads 6.14. The low dimensionality is a genuine property of how these encoders represent document pages.
The two properties together point to a clear implication: a compressed representation should describe those five or six degrees of freedom rather than subsample the patches. Existing methods all store points on or inside the sphere, but the geometry says there is a much smaller structure to describe.
Spherical anchoring: a free 0.093 nDCG@5 improvement
The first insight from the geometry is that standard k-means clustering has a systematic bias. When you cluster unit vectors with k-means, the centroids are Euclidean means of points on the sphere, which means they fall strictly inside the sphere. Their norm is less than one. Since MaxSim computes dot products, this smaller norm systematically underestimates the true inner product for every query direction. The authors prove this formally: the average squared distance from cluster members to their centroid equals one minus the squared centroid norm. Tighter clusters push the centroid closer to the surface, but it never reaches it.
The fix is trivial: project the centroids back onto the unit sphere by dividing each by its norm. This costs nothing, requires no training, and improves nDCG@5 by up to 0.093 over raw k-means centroids. The improvement is largest at small k (where clusters are looser and centroids are further from the surface) and shrinks as k grows, exactly as the geometry predicts. At k=4, the jump is 0.093; at k=64, it is 0.030. The authors call this spherical anchoring, and it becomes the training-free stage of their system. As a one-line practical recommendation: any dot-product late-interaction system that clusters should normalize its centroids.
GLIE: regenerating evidence on demand
Normalized k-means is a strong baseline, but it has two blind spots. Each centroid summarizes its cluster by an average, so scoring-relevant structure inside the cluster is unrecoverable from the code no matter how the centroids are post-processed. And the code is a sample: it stores k points and can only ever score with those k points, so a query token pointing at a region the sample misses loses its evidence entirely.
GLIE addresses both problems with a codec that stores k vectors per page and regenerates the full set of N vectors on demand. The system has three components, each provably starting at and improving on normalized k-means:
Spherical anchoring. Per-page k-means centroids are computed and projected onto the unit sphere. This is the free starting point.
Zero-initialized refinement. A shared cross-attention module refines the anchors against the full token set. The anchors serve as queries, the page's tokens as keys and values. Crucially, the output projection is initialized at zero, so at initialization the refined code is exactly the normalized centroids. Training can only move it where the objective improves. The refiner reads the full token set, so it can encode scoring-relevant structure that centroids average away.
Anchored generative read-out. A shared decoder expands the k stored vectors back into N unit vectors for reranking. The decoder has three structural guarantees. First, it respects the page's cluster structure: cluster j owns exactly n_j output slots proportional to its size. Second, slot 0 of each cluster emits the refined vector verbatim, so the decoded set contains the code set. This means MaxSim over the regenerated vectors is always at least as large as MaxSim over the code, per query token. Regeneration can add evidence but structurally cannot destroy it. Third, each generated child moves along the surface of the sphere by a learned step of at most alpha=0.75, then renormalizes. Children fill a patch around their anchor rather than scattering across the sphere.
The entire codec, refiner and decoder combined, holds 415K parameters against the 3B backbone. It runs once per page at indexing time, on cached embeddings from a frozen encoder.
Two-stage inference: cheap everywhere, expensive on almost nothing
At query time, GLIE runs an asymmetric pipeline. Stage one scores every page against the query by MaxSim over its k stored vectors, at the cost of a pooled baseline. Stage two expands only the top-L candidates (L=20 by default) back to N vectors with the decoder and rescores them exactly. Non-candidates keep their first-stage order. The asymmetry is the point: search is cheap over everything, expensive over almost nothing.
The decoder is shared across all pages and clusters, so the expansion cost is negligible for the shortlist. What changes is which pages get expanded. The authors decompose the remaining error at k=4 into two components: decode fidelity (the gap between GLIE and a hypothetical perfect decoder operating on the same code) and shortlist recall (the gap between the shortlist oracle and the uncompressed ceiling). On ViDoRe v1, the decode gap is larger, meaning better decoders are the immediate lever. Widening the shortlist from 5 to 100 helps the oracle significantly but barely helps GLIE, because the candidates are already present and the decoder does not yet exploit them.
Results across benchmarks and encoders
On ViDoRe v1, with 4 vectors per page (1.0 KB per page against 258 KB uncompressed, a 256x reduction), GLIE retains 79% of the uncompressed system's nDCG@5. The best prior post-hoc method, token pooling, retains 70% at the same budget. At 8 vectors per page, GLIE reaches 86%. At 16 vectors, 91%. At 64 vectors, 97%. The margin over the strongest training-free baseline is positive at every budget, forming a plateau of roughly 0.04 below k=8 and tapering toward noise at k=64, where normalized clustering already sits 0.027 from the ceiling.
On ViDoRe v2, which is a harder benchmark that saturates nowhere, the margin never decays. GLIE retains 64% at k=4 and 95% at k=64, and the generative read-out contributes meaningfully at every budget. This suggests that how sharply the gains concentrate at aggressive budgets is a property of the benchmark's difficulty rather than a constant of the method.
The recipe transfers to a second encoder. On ColQwen2, GLIE retains 82% of uncompressed quality at k=4 and 98% at k=64. Because ColQwen2 is the stronger encoder, clustering already sits closer to its ceiling at every budget, so the room a learned code has is correspondingly smaller. But GLIE still wins at five of six budgets and improves all ten ViDoRe v1 subsets at every budget above k=2.
A small codec beats encoder fine-tuning at matched budgets
One of the paper's most striking results compares GLIE against Light-ColPali's fine-tuning stage under identical training budgets. The authors reproduced Light-ColPali using 4,000 training pages and 13.3M LoRA parameters over 1.5 GPU-hours per budget. This fine-tuning does not reach even free normalized k-means at any budget. Meanwhile, GLIE, using 415K parameters trained in under three GPU-minutes on 1,000 pages, beats the fine-tuned system at every budget by margins of 0.074 to 0.132.
The explanation is straightforward. LoRA fine-tuning with a small budget perturbs a 3B backbone too little to help and enough to hurt, while a small dedicated codec reshapes the stored code directly. The training data requirement is also telling: GLIE saturates on roughly 1,000 training pages, a hundred times smaller a slice of the collection than the encoder trains on. What the codec learns is a property of the encoder's geometry, which Table 2 of the paper showed is corpus-independent.
What each component actually contributes
An ablation study assigns gains cleanly. Spherical anchoring is the largest single factor at every budget, from +0.093 at k=4 to +0.030 at k=64. The learned code adds +0.044 to +0.016 up to k=16 and nothing beyond it, because the code itself is bounded by what k vectors can hold. The generative read-out is the aggressive-budget specialist, peaking at +0.016 at k=4 on ViDoRe v1 and not decaying at all on ViDoRe v2.
Decoder capacity is not where the result comes from. Sweeping decoder width from 128 to 1,024 and depth from one to three blocks, a seventy-fold range from 184K to 13M parameters, moves nDCG@5 by at most 0.009 with no monotone trend. The smallest decoder tested (184K params) is actually the best at k=4. The 415K configuration reported is an arbitrary point inside a flat band rather than a tuned optimum.
Limitations and open questions
The paper identifies three areas where the approach has room to grow. First, the decoder is the main design surface and the largest source of remaining error: at k=4 on ViDoRe v1, a perfect decoder operating on the same code would reach 0.782 nDCG@5 against GLIE's 0.657. Better decoders are the immediate next step. Second, the authors tested only two encoders; extending the sweep to further late-interaction models would test how far the geometric properties generalize. Third, GLIE composes with quantized storage since the two act on different axes of the footprint, but this composition has not been explored.
There is also a practical constraint. Because the code is learned post-hoc from cached embeddings, changing the storage budget of a deployed index touches only cached embeddings rather than requiring re-encoding. But the decoder itself is fixed after training, so if you want to expand to a different k, you need a new decoder fit. The authors report this takes under three GPU-minutes, so it is cheap but not free.
What this means for practitioners
If you are running a late-interaction retrieval system and your bottleneck is index size, there is a one-line improvement available today: normalize your cluster centroids to the unit sphere. This requires no training and gives up to 0.093 nDCG@5. Beyond that, GLIE provides a path to extreme compression without retraining the encoder. At 4 vectors per page, you get 80% of uncompressed quality at 0.4% of the storage. The system trains in minutes on a thousand pages and serves from a frozen backbone. The decoder is where future gains will come from, making this an active area rather than a solved problem. For video late interaction, where token sets are largest and most redundant, the gains should be even larger. Read the paper on arXiv