Register and outlier tokens in self-supervised Vision Transformers



Self-supervised Vision Transformers such as DINOv2 learn rich visual representations, yet the functional role of their internal tokens remains poorly understood. Architectures that introduce dedicated register tokens aim to reduce high-norm outlier patch tokens that appear in background regions, but the semantic and functional status of both token types has not been fully established.



Sparse autoencoders and automated interpretability on DINOv2 activations



The authors train sparse autoencoders (SAEs) on register-token and outlier-token activations within DINOv2. Using an automated interpretability pipeline, UMAP clustering, and CLIP-space cross-checks, they analyze the semantic content encoded by each token type. The pipeline extracts features from activation states, clusters them into meaningful groups, and projects them into CLIP space to verify semantic correspondence.



Register tokens encode high-level semantic concepts



Register-token features show stronger association with high-level semantic concepts. The SAE features extracted from register tokens cluster around object categories, scene types, and other conceptual abstractions that persist across viewpoints and lighting conditions.



Outlier tokens encode lower-level structural and texture patterns



Outlier-token features are more often associated with lower-level structural, background, and texture-dominant patterns. These features capture fine-grained surface details, regional textures, and background geometry that are less tied to object-level semantics.



Causal ablation reveals functional asymmetry between token types



Causal ablations demonstrate a substantial functional asymmetry. Disrupting top-activating register-derived features produces a 48.17% drop in representation cosine similarity, whereas disrupting outlier-derived features produces only a 0.31% drop. This large difference indicates that register tokens carry disproportionately more task-relevant information despite being far fewer in number.



Experimental setup and SAE training details



SAEs are trained on activations from DINOv2's final transformer block. The sparsity penalty is annealed during training to promote interpretable features. L0 targets are set to maintain a balance between reconstruction error and feature sparsity. Features are evaluated by their reconstruction quality, sparsity, and downstream CLIP-space alignment.



UMAP clustering and semantic grouping



UMAP projections of SAE encoder activations separate into distinct clusters corresponding to semantic categories. Register-token clusters map to object classes, object parts, and scene-level concepts. Outlier-token clusters align with texture categories, material types, and background regions. Cross-validation confirms that clusters are stable across different input images and augmentations.



CLIP-space cross-checks validate semantic content



Projected SAE decoder directions are compared against CLIP text embeddings for object categories and material adjectives. High cosine similarity between decoder directions and CLIP text embeddings confirms that register-token features encode recognizable semantic concepts, while outlier-token features align with low-level visual descriptors.



Limitations and practical trade-offs



The analysis is confined to DINOv2 and may not generalize to other self-supervised ViT families or supervised pre-training regimes. The SAE sparsity constraint may discard rare but functionally important features. The causal ablations use targeted unit removal and do not capture distributed rewiring effects that could occur in deeper network contexts.



Practical implications for ViT analysis and design



For researchers analyzing Vision Transformers, the register token provides a concise window into high-level semantic representation. Model designers seeking to improve semantic efficiency can consider regularizing outlier token norms or augmenting the register token pathway. The observed asymmetry suggests that architectures emphasizing register token functionality may achieve better semantic performance per parameter.



Future directions in token interpretability



Extending this pipeline to mid-layer activations, cross-dataset generalization, and multimodal ViTs would broaden the scope of token specialization findings. Investigating how register token functions evolve during fine-tuning could reveal mechanisms of semantic drift in adapted models.



Read the paper on arXiv