While deep learning models for tabular data have recently received significant research attention, the choice of objective functions for regression tasks remains an understudied degree of freedom. Several lines of research offer alternatives to the standard Mean Squared Error (MSE), but differences in datasets, backbones, and evaluation protocols make their practical benefits difficult to compare. In this work, we address this gap by performing a systematic re-evaluation of regression objectives across a broad collection of established tabular datasets and multiple neural backbones. Our comparison reveals that many proposed objectives provide limited or inconsistent gains over a properly tuned MSE baseline, while quantile loss yields consistent aggregate improvements over MSE across the tested backbones. We then identify a limitation that remains even with this strong objective: regions with different levels of target noise can reach their best predictive performance at different training epochs. Consequently, a single global early-stopping checkpoint can impose a compromise between uncertainty regions, even when the objective can represent heteroskedastic target distributions. Motivated by this limitation, we introduce a heteroskedasticity-aware extension of quantile loss that uses input-dependent label smoothing as a potential remedy and improves over quantile loss on several backbones. Overall, our results provide a modern empirical picture of regression objectives for tabular deep learning and highlight how differences in optimal stopping times across uncertainty regions can limit the effectiveness of global early stopping. The code is available at this https URL.
Systematic Re-evaluation of Regression Objectives
Despite the growing adoption of deep learning models for tabular data, in practice, it is not entirely clear what objective function should be used to solve regression tasks with DNNs. While the design of advanced regression objectives is an active research direction, the traditional Mean Squared Error (MSE) remains widely used. Existing studies evaluate alternative objectives on different datasets, backbones, and tuning protocols, making it difficult to establish which alternatives provide reliable improvements in practice. The authors revisit traditional and modern regression objectives on 41 established tabular datasets with extensive hyperparameter tuning, focusing on conventional DNNs. They find that most objectives do not consistently improve over a carefully tuned MSE baseline. Two approaches stand out on MLP: Gaussian-smoothed binning objectives and quantile loss. Of these, quantile loss yields more consistent gains across different backbones.
Quantile Loss as a Strong Alternative
The authors compare 18 regression objectives across 41 datasets. Most objectives do not provide consistent improvements over a carefully tuned MSE. Soft Binning, HL-Gaussian, and quantile loss stand out in aggregate results, improving upon MSE in terms of both ranks and relative RMSE improvement. Quantile loss, which predicts conditional quantiles using the pinball loss and averages them as the scalar prediction, yields more consistent gains across different backbones compared to HL-Gaussian. On MLP with Muon optimizer, quantile loss achieves a median gain of 2.18% over MSE, while HL-Gaussian achieves 2.34%. The results also hold under AdamW optimizer, though Muon configurations achieve larger median gains, highlighting the importance of optimizer choice when evaluating regression objectives.
Limitations of Global Early Stopping
Having identified quantile loss as a strong regression objective, the authors investigate a limitation that remains under standard training: regions with different levels of data uncertainty can reach their best predictive performance at different epochs. Consequently, a single global early-stopping checkpoint can impose a compromise between uncertainty regions. Such input-dependent variation in target noise is common in tabular data. Examining this conflict through controlled synthetic experiments and uncertainty-based checkpoint selection on real datasets reveals that different uncertainty regions favor different stopping times. Per-cohort early stopping, which selects checkpoints separately for groups of comparable data uncertainty, improves median RMSE over global selection by 0.18% for five cohorts and 0.114% for ten cohorts across the seven largest benchmark datasets.
Adaptive Quantile Regression (AdaQR)
Motivated by the limitation of global early stopping, the authors propose Adaptive Quantile Regression (AdaQR), which incorporates jointly learned, input-dependent smoothing into quantile training. By applying stronger smoothing in high-uncertainty regions, AdaQR aims to postpone overfitting there while allowing learning to continue in lower-uncertainty regions. The loss function replaces the point mass in quantile loss with a Gaussian distribution centered around the target, where the variance is predicted from the input. A larger conditional variance yields a larger predicted smoothing parameter, and the second term acts as a regularization penalty scaling the predicted smoothing parameter by a tunable hyperparameter.
AdaQR is designed as one potential remedy for the mismatch induced by a shared stopping point. On MLP across 28 datasets with at least 1,000 training samples, AdaQR achieves positive median RMSE improvements over MSE on all four backbones (MLP, TabPack, RealMLP, TabM). Its median gains over MSE are also larger than those of quantile loss, with the clearest differences on TabM and RealMLP and a small difference on TabPack. An ablation experiment on RealMLP demonstrates the importance of instance-wise adaptivity, as replacing per-sample smoothing with a single global smoothing parameter results in substantial performance degradation.
Uncertainty Estimation
The authors examine whether AdaQR learns meaningful data uncertainty on real-world tasks. Using CatBoost as a trusted reference for ground-truth uncertainty (unavailable), they find strong correlations between AdaQR's uncertainty estimates and CatBoost's, suggesting AdaQR captures meaningful input-dependent variation. This indicates that AdaQR not only improves point predictions but also learns informative uncertainty estimates that reflect the heteroskedastic nature of tabular data.
Effect on the Global Stopping Compromise
Repeating the cohort-based experiment from the global early stopping analysis for AdaQR on the same seven largest benchmark datasets with the MLP backbone shows a lower benefit from cohort-based selection under AdaQR compared to standard quantile regression. The median improvement decreases from 0.18% to 0.092% for five cohorts and from 0.114% to 0.085% for ten cohorts. This is consistent with findings on synthetic data, suggesting that adaptive smoothing does not fully resolve differences in preferred stopping times across uncertainty regions but does provide additional gains beyond standard quantile regression.
Conclusion
In this paper, the authors systematically revisit regression objectives for tabular deep learning, comparing 18 objectives on 41 datasets and evaluating leading approaches across multiple backbones. Quantile loss emerges as a consistently strong alternative to carefully tuned MSE. Yet a strong objective does not eliminate the limitations of global early stopping: regions with different noise levels can peak at different training epochs, making a shared checkpoint a compromise. Motivated by this, the authors introduce AdaQR, which uses jointly learned, input-dependent smoothing as one possible remedy and improves predictions in several settings, with benefits varying across backbones and datasets. Their findings highlight that progress in tabular regression requires attention not only to what models optimize, but also to how different parts of the data are learned over time.