Foundation models pretrained on large natural-image collections have been increasingly adapted for medical image segmentation, on the expectation that they will replace independently trained models for each task. A paper posted to arXiv in November 2026 directly compares one such adaptation strategy against traditionally trained U-Net and ResU-Net architectures across three clinical imaging domains. The study adapted DINOv2, a vision foundation model, using strategies that range from tuning only bias terms to full fine-tuning, then measured performance on both held-out test data from the same source and on independent datasets collected at other institutions. The adapted foundation model achieved better generalization on every external dataset, a result that has implications for deploying segmentation models across clinical sites. Within the training distribution, task-specific models remained competitive when ample training data were available and delineated boundaries with greater precision, while the foundation model more frequently missed small lesions. The authors also found that only a small fraction of the foundation model required adaptation, and the amount of capacity needed was determined by the imaging domain rather than by the specific adaptation method. Additionally, a widely used polyp test set was found to contain duplicate images from another benchmark, calling into question evaluation rigor for that dataset. The authors conclude that foundation-model adaptation is warranted when models must operate across sites, and that capacity should be selected based on the target domain.
Adapting DINOv2 from bias terms to full fine-tuningThe authors instantiated their experiments on a Vision Transformer backbone—DINOv2—that had been pretrained on the large-scale ImageNet-21k and additional natural-image corpora. From this frozen backbone, they explored a spectrum of parameter-update strategies. At one extreme, they updated only the model’s bias parameters, keeping all other weights fixed. At the other extreme, they performed full fine-tuning, allowing every parameter to change. Between these endpoints they evaluated intermediate regimes: tuning a shallow classifier head appended on top of the frozen features, and updating a mid-level subset of transformer blocks. This range was intended to answer two questions left open by prior work: how much of the foundation model must be modified to achieve competitive segmentation, and whether the required extent of change depends on the adaptation method or on some other factor.
Experimental domains and evaluation protocols
The study encompassed three medical imaging modalities. Colonoscopy video frames were used to assess polyp segmentation, a task with abundant annotated data from established repositories. Breast ultrasound images formed the second domain, where the goal is automated boundary detection of pathological tissue. Dermoscopy images of skin lesions constituted the third domain, focusing on lesion segmentation with relatively limited annotated cases. For each domain, the authors reported results on two evaluation sets: a held-out subset drawn from the same distribution as the training data, and an independent test set collected at a different institution. Metrics reported include standard segmentation scores such as Dice coefficient and Hausdorff distance, the latter emphasizing boundary accuracy. The dual evaluation protocol allows insight into both in-distribution performance and cross-site generalization, the latter being especially relevant for clinical deployment where scanner protocols and patient populations vary.Cross-institutional generalization
Consistent with the abstract, the adapted foundation model outperformed task-specific baselines on every external dataset. This result held across all three medical domains, suggesting that the foundation model’s learned representations capture features that transfer beyond the training site. The authors attribute this to the broad visual knowledge already encoded in DINOv2’s pretrained weights, which appear to encode edge and texture detectors useful for anatomy delineation across diverse scanners. In contrast, the U-Net and ResU-Net models, trained from scratch on a single site’s data, showed performance drops when tested on images from different institutions. The gap was most pronounced in the colonoscopy domain, where variations in bowel preparation and camera angles created domain shift. The breast ultrasound and dermoscopy domains showed more modest but still consistent advantages for the adapted foundation model.Boundary precision and small-lesion omission within the training distribution
When evaluated on held-out data from the same source, the task-specific U-Net and ResU-Net models delineated boundaries more precisely than the adapted foundation model. The researchers note that this advantage arises because U-Nets employ a encoder–decoder architecture with skip connections that preserve high-frequency spatial detail, a design choice well suited to the segmentation task when sufficient annotated data are available. The foundation model, by comparison, more often missed small lesions across all three domains. This trade-off—better cross-site generalization at the cost of small-detection sensitivity—is a key consideration for practitioners. The authors suggest that the foundation model’s pooled attention mechanism, while powerful for recognizing semantic patterns, may not allocate enough resolution to tiny structures without additional task-specific fine-tuning.How much of the model needs changing
One of the paper’s more granular findings concerns the proportion of the foundation model that actually requires parameter updates to achieve strong results. The authors measured the number of trainable parameters across the adaptation regimes and found that, even with full fine-tuning, only a small fraction of DINOv2’s total capacity needed to change. Moreover, the capacity that did change was systematically determined by the imaging domain—colonoscopy, breast ultrasound, or dermoscopy—rather than by whether the adaptation was light (bias-only) or heavy (full fine-tuning). In other words, the colonoscopy experiments required a certain amount of trainable parameters regardless of the strategy chosen, and the same held for the other domains. This observation suggests that the bottleneck for performance is not the optimization strategy itself but the amount of model capacity that is relevant to the target imaging modality.A duplicated polyp test set
The authors also performed a routine audit of the evaluation data and discovered that a widely used polyp segmentation test set contains images that are duplicated from another benchmark. The duplication was not intentional; it stemmed from overlapping data collection pipelines between two publicly available datasets. The presence of duplicate images inflates performance estimates because the model effectively sees test examples during training or validation, even when the split appears disjoint. The authors remove the duplicates and re-evaluate, observing a modest but consistent drop in reported Dice scores. They argue that the computer-vision community should adopt stricter data-de duplication protocols, especially for test sets that claim independence from other resources.Practical implications for model selection
For a working developer or researcher, the paper’s takeaway is context-dependent. If the deployment scenario involves a single clinic with ample annotated data, a traditionally trained U-Net or ResU-Net will likely deliver tighter boundary localization and fewer missed small lesions, especially when training time and computational budget are limited. If the target environment spans multiple institutions, different scanner types, or patient demographics, then adapting a foundation model such as DINOv2 becomes the more robust choice. The authors emphasize that the amount of adaptation should be matched to the domain: more data-rich modalities may need only bias-term updates, while sparser domains may require lifting more of the model’s weights. The finding that capacity needs are domain-driven rather than method-driven streamlines the practitioner’s workflow—one can estimate the required trainable parameters by surveying the target dataset’s size and variability, then select an adaptation regime accordingly.Limitations and remaining questions
The study acknowledges several constraints. The experiments cover three relatively common imaging modalities; rarer clinical domains were not explored, so the applicability of the domain-driven capacity rule to, say, histopathology or cardiology imaging remains unknown. The foundation model used, DINOv2, is a ViT backbone with a particular architecture and pre-training objective; other foundation models—such as those based on convolutional backbones or masked-image modeling—may exhibit different adaptation curves. The paper also does not investigate the computational cost of training across the full adaptation spectrum, leaving unanswered whether the observed trade-offs hold when FLOP budgets are tightly constrained. Finally, the authors note that their analysis of the duplicated polyp test set was retrospective; prospective benchmarking frameworks should incorporate data auditing as a standard step.Foundation-model adaptation is therefore warranted where models must work across sites, with capacity selected on the target domain. Read the paper on arXiv