Representations of supervised datasets serve as a foundation for higher-level machine-learning tasks such as dataset retrieval and distribution-shift monitoring. In practice, practitioners have needed reliable ways to characterize what a given supervised dataset encodes, beyond the raw feature matrix and target labels. These characterizations enable queries like "find datasets similar to this one" or "detect when a new data stream diverges from the training distribution." The utility of dataset-level representations has grown alongside the scale of available data repositories, where manual inspection becomes infeasible.
Why Dataset Representations Matter for Supervised Learning
Traditional approaches to dataset characterization have centered on hand-crafted meta-features. These include statistics such as the number of rows and columns, missing-value ratios, variable types, and simple correlation summaries. While interpretable, such features capture only a shallow slice of a dataset's structure. They may miss subtle distributional properties, higher-order feature interactions, or the implicit inductive biases encoded by the model architecture that will eventually process the data.
A second line of work has focused on learning-based dataset encoders. These models are trained from scratch or fine-tuned on specific representation objectives, often requiring labeled dataset pairs or carefully curated meta-tasks. The dependence on task-specific supervision limits generality: an encoder optimized for one meta-task may not transfer well to another, and collecting sufficient supervision across diverse domains remains a bottleneck.
Repurposing Tabular Foundation Models as Dataset Encoders
The paper introduces TabMF, a framework that repurposes pretrained tabular foundation models (TFMs) as encoders of supervised datasets. Tabular foundation models have become increasingly common, trained on large and varied collections of tabular data using self-supervised or semi-supervised objectives. Because these models already learn rich feature representations from raw tables, the authors explore whether their internal knowledge can be leveraged to produce dataset-level embeddings without discarding the predictive components that make the models useful.
TabMF equips a predictive TFM with an architecture-aware dataset-level readout. This readout module aggregates the model's per-sample or per-feature outputs into a single vector that summarizes the entire supervised dataset. The "architecture-aware" qualifier indicates that the readout design respects the specific inductive biases of the underlying TFM—whether it processes rows as sequences, exploits columnar patterns, or employs a mixture-of-experts routing scheme. By aligning the readout to the model's architecture, TabMF avoids a one-size-fits-all aggregation that could misrepresent the dataset's structure.
After attaching the readout, TabMF adapts the model end-to-end using contrastive post-training. The contrastive objective operates on stochastic views of synthetic supervised datasets. Concretely, multiple augmented versions of a given dataset are generated—through resampling, feature perturbation, or subsampling—and the model is trained to pull the readout vectors of augmented views closer while pushing apart those of distinct datasets. This post-training phase fine-tunes both the readout parameters and the TFM's internal weights, aligning the model's predictive power with the task of dataset discrimination.
Contrastive Post-Training on Stochastic Synthetic Views
The contrastive procedure in TabMF relies on the construction of synthetic dataset views. For each original supervised dataset, the framework generates augmented variants by applying stochastic transformations such as random feature masking, column subsampling, or label noise. These views preserve the core structure of the original dataset while providing enough variation for a contrastive loss to distinguish one dataset from another. The underlying intuition is that two augmented views derived from the same source dataset should elicit similar representations, whereas datasets drawn from different distributions should produce distinguishable readout vectors.
The contrastive loss used is a standard instance-pair discrimination objective, but applied at the dataset level rather than the sample level. During each training step, a batch contains pairs of dataset representations: positive pairs correspond to augmented views of the same original dataset, and negative pairs come from different datasets. The model learns to maximize agreement among positives and minimize it among negatives. Because the TFM was already pretrained on large-scale tabular data, the early layers of the model capture meaningful feature interactions, and the contrastive post-training primarily fine-tunes the readout and higher-level abstractions relevant to dataset identity.
An important design choice is that the synthetic views are generated on-the-fly, meaning the framework does not require a separate corpus of pre-augmented datasets. This makes TabMF applicable to any supervised dataset that can be fed to the underlying TFM, without the need for an external augmentation pipeline.
Experimental Evaluation Across Six Representation Tasks
The authors instantiate TabMF on two prominent tabular foundation models: TabICLv2 and TabPFN-2.5. TabICLv2 builds on in-context learning principles for tabular data, while TabPFN-2.5 is a "pure neural network" foundation model trained to predict across many tables. Both models already possess strong predictive performance on a wide range of tabular tasks, which makes them natural candidates for the TabMF repurposing effort.
Evaluations are conducted across six tasks involving unseen real datasets. The tasks span dataset retrieval, compact benchmark construction, and distribution-shift severity assessment, among others. For each task, the representations produced by different methods are compared against baselines that include hand-crafted meta-features and dataset encoders trained specifically for representation learning. The comparison surfaces how much of the representational power comes from the pretrained TFM itself, and how much additional benefit the TabMF post-training provides.
In every experiment, the evaluation metric reflects the model's ability to perform the target task using only the dataset embedding as input. This means the six tasks effectively probe different aspects of what the embedding captures—whether it's semantic similarity between datasets, the ability to construct minimal yet representative benchmark suites, or the sensitivity to detect when a new dataset exhibits shifted characteristics relative to the training distribution.
Frozen Readouts Already Induce Substantial Dataset Structure
A striking finding in the paper is that simple frozen TFM readouts—those without any post-training adaptation—yield competitive representations across the six tasks. This result suggests that the predictive pretraining objective, which already requires the model to fit numerous tabular tasks, induces substantial dataset-level structure as a byproduct. In other words, the model's internal architecture has learned to organize its knowledge in ways that are informative about the dataset it has processed, even before any explicit dataset-representation training.
The competitiveness of frozen readouts has practical implications: it means that practitioners can obtain useful dataset embeddings without additional compute cost or data collection. By simply attaching a readout head that aggregates the TFM's outputs and using those activations as the dataset representation, many higher-level tasks can be supported out of the box. This lowers the barrier to entry for teams that wish to explore dataset retrieval or shift detection without investing in custom encoder training.
Strongest Aggregate Performance in Retrieval and Shift Assessment
While frozen readouts are competitive, the two TabMF encoders—one based on TabICLv2 and the other on TabPFN-2.5—obtain the strongest aggregate performance among all evaluated learned and classical dataset representations. The TabMF framework consistently improves upon the frozen baseline, with the largest gains appearing in the three tasks highlighted by the authors: dataset retrieval, compact benchmark construction, and distribution-shift severity assessment.
In dataset retrieval, the TabMF embeddings enable more accurate ranking of datasets by similarity. This capability supports use cases such as locating previously encountered data that is relevant to a current project, or identifying duplicates and near-duplicates in a growing dataset repository. The contrastive post-training appears to sharpen the embedding space, making inter-dataset distances more discriminative.
For compact benchmark construction, the embeddings help identify minimal subsets of a dataset that preserve the essential predictive signals. This is useful when researchers or practitioners need to create quick-prototype benchmark suites without retaining the full original data. TabMF's representations appear to capture the "core" predictive structure of a dataset, allowing smaller, carefully selected subsets to maintain performance characteristics comparable to the full table.
In distribution-shift severity assessment, the embeddings provide a quantitative measure of how much a new dataset diverges from the distribution assumed by the TFM. This can inform decisions about whether a model's predictions remain reliable or whether re-training or adaptation is warranted. The TabMF encoders show particular sensitivity to shift-related distributional changes, making them a practical tool for monitoring model deployment environments.
Acknowledged Constraints and Trade-offs
The paper notes several limitations and trade-offs. First, the quality of the dataset embedding is ultimately bounded by the capabilities of the underlying TFM. If the foundation model was trained on a narrow subset of tabular domains, its dataset-level representations may not generalize well to datasets outside those domains. The authors acknowledge that TabMF's benefits are most pronounced when the base TFM has been exposed to diverse data distributions during pretraining.
Second, the contrastive post-training requires access to the TFM's architecture and internal representations, which may not be available for all tabular models. Models that lack open-weight releases or that encapsulate their parameters behind APIs cannot be directly adapted using the TabMF procedure. This restricts the framework's applicability to the open-release TFMs that the authors selected, namely TabICLv2 and TabPFN-2.5.
Third, the synthetic view generation strategy, while flexible, may not capture all forms of dataset variation. Certain subtle structural changes—such as the introduction of latent confounders or complex label dependency shifts—may require more elaborate augmentation schemes than the random subsampling and feature masking employed in the paper. The authors suggest that future work could explore more sophisticated view generators, but note that the current simple augmentations already yield strong results.
Fourth, the evaluation focuses on six representative tasks, and other downstream tasks involving dataset representations may present different strengths and weaknesses. The authors encourage the community to test TabMF embeddings on additional objectives, such as dataset clustering, version detection, or active learning query prioritization.
Practical Implications for Working Developers
For developers who maintain or consume tabular data repositories, TabMF offers a low-cost pathway to generate high-quality dataset embeddings. Because the framework builds on already-available pretrained TFMs, integrating TabMF into an existing pipeline typically involves attaching a readout head and running the contrastive post-training loop—a process that can be completed in a fraction of the time required to train a representation model from scratch.
The finding that frozen readouts already produce competitive embeddings means that the simplest possible integration—extracting activations from a pretrained TFM without any further training—may already suffice for many use cases. Teams can start with this baseline and incrementally add the contrastive post-training step if their application demands the additional discriminative power that TabMF provides.
The strong results in dataset retrieval and shift assessment suggest that TabMF embeddings could be incorporated into operational monitoring tools. For instance, a continuous integration pipeline for machine learning could periodically compute dataset embeddings for newly arriving data and compare them against stored embeddings from previous periods. A significant drop in similarity or a sudden shift in the embedding space could trigger alerts about potential distribution drift, prompting a review of model suitability.
Additionally, the compact benchmark construction capability opens the door to more efficient experiment workflows. Instead of version-controlling large tabular datasets, teams could store only the dataset embeddings and, when needed, reconstruct a minimal representative subset using the readout vector as a guide. This can reduce storage costs and improve data management overhead, particularly in organizations with extensive experiment histories.
A Natural Ending
TabMF demonstrates that pretrained tabular foundation models carry substantial dataset-level information that can be surfaced through a lightweight readout and optional contrastive adaptation. The framework bridges the gap between hand-crafted meta-features and fully learned dataset encoders, offering a path that is both broadly applicable and empirically effective. By showing that frozen readouts already capture meaningful dataset structure, the authors highlight the implicit representational benefits of predictive pretraining on tabular data. The two TabMF instantiations, TabICLv2 and TabPFN-2.5, achieve the strongest overall performance across the evaluated tasks, particularly in retrieval, benchmark condensation, and shift detection. As the ecosystem of tabular foundation models continues to expand, approaches like TabMF are likely to play a key role in enabling higher-level dataset analytics without the overhead of task-specific model development.
Read the paper on arXiv