Production recommender systems exhibit feature-lifecycle asymmetry: adding new features must produce gains on some metric, while feature removal must not regress any metric. This asymmetry makes deprecation difficult to prioritize. A feature can fail to be removed if it interacts with other features, its importance changes daily, it cannot be evaluated before production models change or data drift occurs, it competes for A/B test space with other launches, it causes regressions in online tests, or it is consumed by multiple models that must coordinate its removal. FORTE addresses these challenges as a model-agnostic production framework for discovering, validating, and removing features at industrial scale.
Feature-lifecycle asymmetry in production recommenders
In large-scale recommendation systems, feature sets grow continuously as teams launch new signals. Each addition must demonstrate measurable improvement in an A/B test. However, removing a feature carries stricter constraints: no metric can regress. This asymmetry creates a ratchet effect where features accumulate even when their marginal utility diminishes. Over time, bloated feature sets increase training costs, serving latency, and model complexity without proportional benefit. Existing approaches to feature selection often rely on offline importance scores that fail to capture feature interactions, daily importance drift, or cross-model dependencies. FORTE was built to overcome these gaps in a production environment where evaluation must be frequent, interaction-aware, and predictive of online outcomes.
FORTE architecture: perturbation, grouped ablation, and lineage tracing
FORTE operates in three phases. First, an efficient perturbation mechanism evaluates every feature daily by measuring the change in model loss when the feature is randomly corrupted. This perturbation is model-agnostic and scales to millions of features by using a low-rank approximation of the feature covariance matrix. Second, to account for interactions, FORTE constructs candidate deprecation sets from all low-rank features and evaluates them through grouped ablation. Rather than testing features in isolation, it removes subsets together and measures the combined effect. This captures synergistic or redundant relationships that single-feature ablations miss. The offline grouped ablation results are calibrated against historical A/B test outcomes, making them sufficiently predictive to allow deprecation candidates to proceed to online testing with confidence. Third, lineage tracing maps each feature to every downstream consumer model in the production graph. When a feature is deprecated, FORTE identifies all models that consume it, enabling coordinated retraining and realizing the full computational savings of removal from the training pipeline.
Validation on search-relevance and ranking models
The authors validate FORTE on two large-scale recommender systems at an industrial deployment. On a search-relevance model, FORTE produced three nested candidate deprecation sets of 32, 48, and 59 features. Online guardrails rejected the largest set due to a marginal regression signal. The 48-feature set was selected for deprecation, removing 43.3 percent of the model's features without any metric regression. On a large ranking model, FORTE identified eight features for removal. Retraining followed by A/B testing confirmed zero regression. The launch reduced trainable parameters by 29.9 percent, improved serving throughput by 19.8 to 25.5 percent, and reduced annual training costs by hundreds of thousands of dollars. The efficiency impact varies with model architecture; sparse linear models benefit more from parameter reduction than deep dense architectures.
Trade-offs and operational considerations
FORTE's grouped ablation requires compute proportional to the number of candidate sets evaluated, which can be substantial for models with thousands of low-rank features. The framework mitigates this through the low-rank covariance approximation and by prioritizing candidates with the lowest individual perturbation scores. The offline-to-online calibration depends on a history of A/B tests with known feature removals; cold-start systems without this history may need a bootstrap phase. Lineage tracing assumes a well-maintained feature registry and model dependency graph, which may not exist in less mature ML platforms. The authors note that FORTE does not replace the need for careful A/B testing but reduces the candidate surface so that test slots are spent on high-confidence deprecations rather than exploratory removals.
What this means for ML platform teams
Teams operating large-scale recommender systems can adopt FORTE's principles even without the full framework. Daily perturbation scoring provides a lightweight health signal for feature drift. Grouped ablation, even at a smaller scale, catches interaction effects that single-feature importance misses. Lineage tracing between features and models should be a first-class capability in any ML feature store. The calibration of offline ablation against online A/B results creates a feedback loop that improves deprecation confidence over time. FORTE demonstrates that systematic feature deprecation is not only possible but yields measurable infrastructure savings—reduced parameters, faster serving, lower training costs—without sacrificing model quality.
Read the paper on arXiv