Test-time adaptation helps a deployed model respond to changing inputs, but the same intervention can rescue one batch and damage the next. Updating normalization statistics, for example, can improve corrupted images yet fail when a batch contains mostly one class. Average accuracy hides these failures and cannot tell a model when to use an intervention. Target labels can guide this decision, but they are scarce and some choices are already clear. We introduce PORTAL, a controller that chooses when to deploy an intervention, which inputs to label, and when to stop querying. Each available prediction rule is a candidate, and its predictions are fixed before current feedback arrives. Every acquired label therefore evaluates all candidates on the same input. PORTAL combines these shared comparisons and directs queries toward candidates that can still overtake the current leader. It stops when the remaining labels cannot change the choice or when the possible accuracy gain no longer justifies another query. Our analysis links the required feedback to accuracy gaps and shared mistakes, explaining why some choices need much less evidence. Experiments show that this control turns an intervention that loses accuracy overall into a useful choice on selected batches. It prevents severe failures caused by changes in arrival order and preserves gains across complete corruption streams. When a fixed rule already works well, PORTAL uses little feedback to retain its performance. Periodic label checks reveal when past evidence needs refreshing and help the controller recover as conditions change. The acquired labels also train stronger candidates for future batches, and selection brings nearly all of their accuracy gains into deployment. The same feedback therefore improves both the choice made now and the predictions available to later batches.
Test-time adaptation under shifting inputs
Deep learning models deployed in production often encounter inputs that differ from the distribution on which they were trained. Test-time adaptation (TTA) methods attempt to update model state – such as normalization statistics, ensemble weights, or other parameters – so the model can produce accurate predictions on the new data. A central challenge is deciding whether an adaptation step helps or harms performance on the current batch. Interventions that improve accuracy on corrupted or noisy inputs may degrade performance when the batch happens to contain easy examples from the original distribution. Conversely, skipping an adaptation that would have helped a difficult batch leaves accuracy on the table. Prior work has largely focused on designing better adaptation operators – layer-wise learning rate tuning, batch statistics updates, entropy minimization – while assuming the decision of when to apply them is either always-on or triggered by a simple heuristic.
This paper identifies a gap: even a well-designed adaptation can be harmful if applied at the wrong time or to the wrong batch. The authors argue that the decision itself should be learned or reasoned about, rather than treated as a binary global flag. The key observation is that different adaptation strategies excel under different conditions, and a controller that selects the right strategy for the right input subset can recover accuracy that a single fixed strategy would lose.
Why average accuracy fails to guide intervention choices
Reporting average accuracy across a test stream masks per-batch variations. A model that achieves 85% average accuracy may do so because it performs well on the majority class while severely failing on a minority subset. In such a case, an intervention that improves minority-class accuracy at the expense of majority-class accuracy might actually decrease the average, even though it helps the harder subgroup. Similarly, if inputs arrive in a correlated order – say, all corrupted images first, then clean ones – a one-size-fits-all adaptation policy will rescue the first group and damage the second, and no post-hoc average can reveal this asymmetry. The paper notes that target labels, when available, can indicate whether an intervention should be applied, but labels are expensive to obtain and some deployment scenarios preclude them entirely.
The fundamental problem, then, is to determine, for each incoming batch or micro-batch, whether an intervention will improve or degrade the model's performance, using as few additional labels as possible. This is the problem PORTAL addresses.
PORTAL: a controller for when and what to label
PORTAL operates as a decision layer sitting between the incoming data stream and the adaptation module. At each step, a set of prediction rules – or "candidates" – is available. Each candidate represents a possible way to adapt the model: perhaps a different normalization update rule, a different ensemble combination, or a different threshold for triggering an adaptation. The critical design choice in PORTAL is that each candidate's predictions are fixed before any current feedback (labels) arrives. This means that for a given input, all candidates produce their predictions first, and only afterward are labels acquired and used to evaluate every candidate on that same input.
Because predictions are fixed ahead of time, every label acquired becomes a shared evaluation point. If three candidates are running in parallel – call them C1, C2, and C3 – and a label y is obtained for input x, then the loss of C1 on (x, y), the loss of C2 on (x, y), and the loss of C3 on (x, y) are all computed. This shared evaluation eliminates a subtle source of bias: each candidate is judged on exactly the same inputs, so differences in performance reflect the candidate's logic rather than differences in the data it happens to see.
PORTAL combines these shared comparisons across labels to maintain a running estimate of which candidate is leading in terms of accuracy. The controller also tracks which candidates have the potential to overtake the current leader given the remaining unlabeled inputs. Queries are directed toward inputs that are most informative for distinguishing among candidates – specifically, those where the current leader's advantage is narrow or where another candidate could surpass it if it were to perform well on that input.
Directing queries toward candidates that can still overtake
Not every input is equally useful for evaluating candidates. Some inputs are easy for all candidates, providing little discrimination. Others are hard for everyone, again providing little distinction. The most informative inputs are those where the current leader is vulnerable – perhaps another candidate would get them right while the leader gets them wrong. PORTAL uses the accumulated label information to compute, for each remaining unlabeled input, a measure of how much a label there could shift the relative ordering of candidates. Inputs that could potentially change the leader receive higher priority for labeling.
This direction of queries is not random; it is focused on the boundary of what is still uncertain. If the current leader has a substantial margin over all other candidates across all already-labeled inputs, then many remaining inputs become less relevant because even worst-case performances on them would not overturn the leader. PORTAL monitors this margin and uses it to decide when further labeling is unlikely to flip the decision.
Stopping when the remaining labels cannot change the choice
PORTAL has two stopping criteria. The first is logical: if, given all labels acquired so far, no remaining unlabeled input could possibly cause the current leader to lose its position, then querying stops. This is determined by examining the worst-case scenario for the leader on remaining inputs and comparing it to the best-case scenario for any challenger.
The second criterion is economic: if the possible accuracy gain from switching to a different candidate is smaller than the cost – in labels, compute, or latency – of acquiring one more input, then querying also stops. This ensures that PORTAL does not spend excessive resources on labeling when the expected benefit is minimal. The paper's analysis shows that the number of labels needed before stopping is closely related to the accuracy gaps between candidates and the degree to which they make the same mistakes (shared mistakes). When candidates disagree on many inputs, fewer labels are needed to identify the best one. When they tend to err on the same inputs, more labels may be required to find the one that gets those inputs right.
Analysis linking feedback to accuracy gaps and shared mistakes
The theoretical analysis in the paper connects the number of labels PORTAL acquires to two fundamental quantities: the accuracy gap between the current leader and the best alternative candidate, and the shared mistake rate among candidates. If two candidates consistently make errors on the same inputs, then labeling those inputs does not help distinguish them; PORTAL must look elsewhere. Conversely, if candidates make disjoint sets of errors, then relatively few labels are needed to identify which one is superior. The analysis also shows that when the accuracy gap is large, PORTAL stops quickly because even unfavorable performances on remaining inputs cannot overcome the leader's advantage. When the gap is small, more labels are required to confirm that the leader is genuinely better rather than merely lucky on the observed inputs.
This explanation accounts for the empirical observation noted in the abstract: some choices need much less evidence than others. If a candidate is clearly ahead in accuracy from the start, PORTAL requires few labels to confirm this. If the candidates are closely matched, more labeling effort is needed – but the controller's query direction ensures that each label is maximally informative.
Experiments: turning an intervention that loses overall accuracy into a useful choice
The experimental results center on corruption and distribution shift scenarios. In one set of experiments, a particular test-time adaptation – updating running batch normalization statistics – improves accuracy on corrupted image subsets but harms accuracy when a batch is dominated by a single class. When PORTAL is applied, the adaptation is deployed only on subsets where it is predicted to help, and skipped or replaced on subsets where it would harm. The result is that the overall average accuracy, which would decrease if the adaptation were applied uniformly, is instead preserved or improved because the controller avoids the harmful subsets. More importantly, the subsets on which the adaptation would have been detrimental are identified and either receive a different intervention or none at all, preventing the severe accuracy drops that would otherwise occur.
In another set of experiments, inputs arrive in a correlated order – for instance, all heavily corrupted images appear first, followed by clean images. Without a controller, the adaptation that helps the corrupted images would be applied early, and when clean images arrive, the adaptation would degrade their accuracy. PORTAL's query-efficient labeling prevents this by detecting the shift and switching to a different adaptation or no adaptation as appropriate. The paper reports that PORTAL prevents the severe accuracy failures that arise from changes in arrival order, and instead preserves the gains achieved by the adaptation on the subsets where it is appropriate.
Preserving gains across complete corruption streams
When the test stream consists of a complete sequence of corruptions – say, Gaussian noise, shot noise, motion blur, etc. – applied consecutively, a fixed adaptation strategy may excel on some corruptions and fail on others. PORTAL's ability to stop querying when further labels cannot change the decision means that on each corruption type, it quickly settles on the best-performing candidate. Across the full stream, the cumulative accuracy is higher than with a fixed strategy because PORTAL selects the right candidate for each segment. The paper emphasizes that this is not merely a matter of averaging – PORTAL actively avoids applying the wrong candidate at the wrong time, and the labels acquired on earlier segments also serve to train better candidates for future segments.
Minimal feedback when a fixed rule already works well
An important practical finding is that when one candidate already performs well relative to the others, PORTAL uses very few labels to confirm this and retain the current leader. The controller does not waste resources on unnecessary labeling. This is significant because in many deployment scenarios, a default adaptation (e.g., standard batch normalization without any special test-time update) may already be near-optimal. PORTAL detects this situation quickly and refrains from extensive querying, thereby preserving the status quo while still providing the option to switch if conditions change.
Periodic label checks for refreshing past evidence
Conditions can change over time – the nature of the distribution shift may evolve, or new types of inputs may appear. PORTAL addresses this by incorporating periodic label checks. These are not continuous labeling efforts but rather intermittent evaluations on newly arriving inputs. If the periodic check reveals that the current leader has been overtaken or its margin has shrunk, PORTAL resumes active querying to identify a new leader. The paper shows that this mechanism allows the controller to recover as conditions change, without requiring a full restart of the labeling process. The acquired labels from previous periods continue to inform the controller, but the periodic checks ensure that outdated evidence does not persist indefinitely.
Training stronger candidates for future batches
One of the more subtle contributions of the paper is the observation that the labels acquired by PORTAL for decision-making also serve to train stronger candidates. Each label provides supervision that can be used to update or fine-tune the candidate models. Because PORTAL selects inputs that are informative for distinguishing among candidates, the labels acquired are particularly valuable for improving the weaker candidates. The paper reports that after several batches, the candidates have been refined, and when PORTAL selects among them for deployment, it brings nearly all of their accuracy gains into deployment. In other words, the feedback loop does not only decide which candidate to use now – it also makes the candidates themselves better, and those improvements carry over to future test-time batches.
Conclusion
PORTAL introduces a controller for test-time adaptation that decides when to intervene, which inputs to label, and when to stop querying. By fixing predictions before feedback arrives and using shared comparisons across candidates, PORTAL directs labels efficiently and stops when further labeling is unwarranted. The analysis links the required feedback to accuracy gaps and shared mistakes, explaining why some decisions need little evidence. Experiments demonstrate that PORTAL turns an intervention that loses accuracy overall into a useful choice on selected batches, prevents severe failures from changes in arrival order, and preserves gains across complete corruption streams. When a fixed rule already works well, PORTAL uses minimal feedback to retain performance. Periodic label checks allow the controller to recover as conditions change, and the acquired labels also train stronger candidates for future batches. The same feedback therefore improves both the choice made now and the predictions available to later batches.
Read the paper on arXiv