Zero-cost proxies for neural architecture search rank candidate networks from a single forward or backward pass at initialization, avoiding the cost of full training. But their reliability varies across search spaces: a capacity-driven proxy like parameter count excels on size-varying spaces but falters on topology-varying ones, while a structure-driven proxy does the reverse. Every proxy has a weak regime, and no single proxy robustly beats the trivial parameter count baseline across both structure-varying and size-varying spaces under one protocol. Yifan Yang, Zhaoyan Wang, Zheng Gao, Xiaoyu Li, and Jiaojiao Jiang at the University of New South Wales and KAIST introduce CoRA-NAS to close this gap: a two-stage framework that combines a static ranking prior with low-cost learning-curve refinement, using a single configuration across all search spaces.

Three axes of signal, two static and one dynamic

The starting point is an observation about what drives a trained network's accuracy. Two signals are static and near-zero-cost: the network's capacity (a representational budget) and how well its structure uses that capacity at initialization (the quality of the wiring). The third signal is dynamic: the learning dynamics, visible only once training begins. These three axes are complementary, not redundant. Capacity and structure are decoupled by construction in NATS-Bench, where the size variant (SSS) fixes topology and varies width while the topology variant (TSS) does the reverse. Each axis alone fails in the other regime: capacity dominates size spaces, structure dominates topology spaces. Static proxies cannot observe optimization; the early training curve catches the "good at init but trains poorly" cases that static proxies miss.

CoRA-NAS decomposes these axes into two stages. Stage 1 (Rank) aggregates the two static axes into a zero-cost coarse prior. Stage 2 (Refine) injects the dynamic axis as a residual correction to that prior. The key insight is that the residual is well-posed: the dynamic axis is orthogonal to the static prior, so the curve residual carries exactly the information the prior structurally lacks. A clean prior keeps this residual learnable from cheap features.

Stage 1: Equal-weight rank consensus with a label-free gate

Stage 1 produces a label-free coarse ranking from five off-the-shelf zero-cost proxies covering both static axes: parameter count and Synflow for capacity, and jacov/NASWOT plus the expressivity and trainability views of AZ-NAS for structure at initialization. No new proxy is introduced. For each proxy, a percentile rank is computed over the entire search space.

Before combining, a target-free consensus gate adapts the bank to each space without reading any labels. Each proxy's agreement with the leave-one-out consensus of the others is measured via Spearman correlation. Proxies that anti-correlate with the majority are dropped. On NB201 and NATS-SSS, all five proxies agree so the gate is a no-op. On NB101, it drops jacov, which genuinely anti-correlates there (alpha = -0.41), independently rediscovering jacov as the least transferable proxy. The threshold is robust: any value in [-0.35, -0.2] yields the same partition. This is the key contrast with LIBRA-NAS, which selects per-space proxies by agreement with ground-truth accuracy. CoRA's gate uses only inter-proxy agreement, remaining fully label-free.

The kept proxies are combined by equal-weight log-rank consensus: the sum of log percentile ranks, where the log penalizes any proxy ranking an architecture near the bottom (a soft AND). Equal weighting means no learned weights and no per-space tuning. An ablation shows that learned adaptive weighting and a two-axis decoupling both fail to improve the result and often hurt it.

Stage 2: Anchor-based residual with target-free curve extrapolation

Stage 2 spends roughly 1% of full training on a small anchor set to inject the dynamic axis. About 1,000 architectures (approximately 6.4% of the space) are stratified across the prior percentile to cover the full range evenly. Each anchor is trained for a few epochs (5 to 30, depending on the space) and its early validation-accuracy curve is recorded.

From each curve, a final-accuracy estimate is formed without any ground-truth target by rank-averaging two parametric extrapolations: a log-linear fit and a bounded power law. The power law supplies a bounded asymptote that prevents top-end overshoot. The residual on each anchor is the gap between the dynamic ranking and the static prior: where the prior is wrong and by how much.

An Extremely Randomized Trees regressor is fitted on the anchors using features that include the prior percentile, four cheap raw signals (Synflow, jacov, L2 weight norm, parameter count), and the architecture encoding. The residual is propagated to every architecture in the space, combined with the prior at half weight (to hedge against propagation error), and smoothed by averaging each score with its one-edit neighbors to denoise the head of the ranking.

Results: no weak regime across four vision spaces

On NAS-Bench-201 (15,625 architectures, three datasets), CoRA-Rank (Stage 1 alone) achieves the best Spearman correlation among all training-free methods on every dataset: 0.938 on CIFAR-10, 0.932 on CIFAR-100, 0.911 on ImageNet16-120, with a mean of 0.927. This is ahead of AZ-NAS (0.897), MeCo (0.886), Dextr (0.874), LIBRA-NAS (0.872), SWAP (0.859), and ZiCo (0.790). Stage 2 (CoRA-Refine) pushes this further to mean Spearman 0.946. On CIFAR-100, the selected architecture reaches 73.32% accuracy, near the ground-truth best of 73.37%.

Across four vision benchmarks with a single untuned configuration, CoRA-Refine has no weak regime. Its per-space Spearman correlations are 0.946 (NB201), 0.715 (NB101), 0.786 (TransNAS-Bench-101), and 0.894 (NATS-SSS). The worst-space correlation of 0.715 is the highest floor of any compared method, including target-aware LIBRA-NAS. Every baseline collapses on at least one space: capacity proxies fail on topology spaces, structure proxies fail on size spaces, and even the strongest combined proxies floor at 0.55 to 0.61 somewhere.

On the real DARTS cell space (via the NB301 surrogate), Refine on real 15-epoch curves recovers the capacity-dominated boundary, beating the diluted prior (+0.046) and matching parameter count within noise. Trained from scratch with 5 seeds, the selected cell reaches 2.81% CIFAR-10 error, beating AZ-NAS and random using the same candidate pool and evaluation protocol.

Why the clean prior matters

The authors show that a clean, signal-transparent prior is essential for the refinement to work. An ablation feeds two different priors into the same refinement: the equal-weight consensus and AZ-NAS's opaque non-linear score. AZ-NAS ranks well on its own (mean Spearman 0.897 on NB201), but its non-linear aggregation yields a residual that the tree ensemble cannot learn from cheap features (cross-validated R-squared = 0.37 versus 0.43 for CoRA's consensus). The clean consensus produces a residual that is structured and predictable from the architecture encoding, which is why the refinement can correct it.

This also explains why predicting a residual rather than accuracy from scratch works. The residual is small and structured (the gap between the prior and the dynamic ranking), while predicting accuracy from scratch requires learning a much larger, noisier function. The target-free estimate from early curves means the method can run on any space without fully trained performance labels.

The honest limitations

On the pure size space, the unsupervised consensus prior does not beat parameter count. It is the low-cost curve residual that recovers it, yielding one configuration robust across all four vision spaces. The size-space result is a tie with the best capacity proxies, not a win. The architecture encoding is space-specific, as for any NAS method. The gate catches sign-flips but not dilution (weak disagreement that does not cross the anti-correlation threshold). Refine is not zero-cost: it spends approximately 1% of full training, which is small but nonzero.

The no-weak-regime result is demonstrated over four vision spaces. On a structurally distinct recurrent space (NAS-Bench-NLP), both the prior and the residual are weak, so CoRA hits an honest weak regime. On a vision-transformer space (ViT-Bench-101), the label-free consensus is competitive with the ViT specialist L-SWAG (approximately 0.03 lower mean Spearman) and ahead of the general ranker AZ-NAS. These results delimit the claim to vision rather than universality.

What this means for NAS practitioners

CoRA-NAS demonstrates that cross-space robustness is achievable without per-space tuning or ground-truth accuracy labels. The single-configuration design means a practitioner can apply the same method to a new search space without re-running hyperparameter optimization or needing labeled architectures for calibration. The 1% training budget for refinement is small enough to be practical for real NAS workflows, where the alternative is training thousands of candidates to convergence.

The framework's boundary is clear: it benefits most when the prior is informative and the architecture encoding captures meaningful structural variation. On spaces where the encoding is weak (like the recurrent NAS-Bench-NLP space), the residual is not learnable and Refine cannot lift the prior. The method is proxy-agnostic by design: any bank of zero-cost proxies covering capacity and structure can be plugged in, and the gate will adapt the bank per space. This makes it a framework rather than a single method, and its benefit scales with the quality of the underlying proxies.

Read the paper on arXiv