A New Way to Morph One Neural Network Into Another
Finding a promising neural architecture is only half the battle. Once you have one that works, how do you get from it to a better one without throwing everything away and starting over? Researchers at Flx AI have proposed a method that treats architecture design like a genetic crossover problem, borrowing techniques from bioinformatics to systematically combine the best parts of two existing networks.
The core idea is deceptively simple: if you can figure out exactly which edits transform one architecture into another, you can then sample combinations of those edits to produce offspring that blend properties of both parents. The challenge is that those edits are not independent. Adding a layer shifts everything beneath it. Branching structures impose constraints on which changes are valid and in what order they must occur. The paper, titled Evolutionary Architecture Search through Grammar-Based Sequence Alignment, addresses all of this with a grammar-aware alignment procedure the authors call Constrained Smith-Waterman Crossover, or CSWX.
Why Position-by-Position Comparison Fails
Neural architectures are not like strings of text where you can compare characters at fixed positions. Inserting a single layer shifts every subsequent layer, making a naive position-by-position comparison misleading. The authors solve this by representing each architecture as a derivation tree, a hierarchical record of how the network is assembled from modules arranged in sequence, in parallel branches, or wrapped in routing constructs.
These trees are serialized into token sequences, with special separator tokens marking the boundaries of branches and routing modules. The separators matter because they preserve structural information that would otherwise be lost. Without them, a change to a branching node might appear valid on its own but leave the rest of the sequence describing an inconsistent or incomplete architecture.
The Alignment Engine
CSWX works by building a dynamic-programming matrix where one parent's token sequence forms the columns and the other forms the rows. Moving right through the matrix deletes a token from the first parent; moving down inserts one from the second; moving diagonally matches or substitutes the two. The algorithm keeps the cheapest paths at each cell while discarding any that violate the grammar's structural rules.
When multiple paths carry the same cost, all of them are retained because earlier decisions constrain which later moves are valid. The accumulated cost at the bottom-right corner gives the edit distance between the two architectures, and tracing back through the recorded choices recovers the exact sequence of edits needed to transform one into the other.
Handling Branch Order
One subtlety the authors address is branch ordering. When two branches are combined by addition, swapping their order leaves the computation unchanged, yet the token sequences differ. A straightforward alignment would still assign a positive edit distance for what is functionally identical architecture.
The recursive extension, RCSWX, solves this without recomputing the entire alignment for every possible branch ordering. Instead, it reuses the unaffected regions of the matrix and evaluates alternative orderings only within submatrices delimited by branching nodes. When a separator closes a branching block, the algorithm merges alternatives by keeping the cheapest paths to each cell. This process recurses through every level of nesting.
From Alignments to Offspring
Once edits are recovered, the sampling procedure generates offspring by selecting compatible combinations of those edits. The method, called Shortest Edit Path Crossover, randomly picks roughly half the edits from a shortest path between the parent graphs. Dependencies between edits are recorded and enforced: deleting all operations inside a routing module would leave it empty unless another edit removes the enclosing module or fills it with new content.
Edits are sampled according to their total cost using a truncated skew-normal distribution. At zero skewness, this reduces to a truncated Gaussian, but adjusting the skewness lets the system bias offspring toward one parent or the other.
What It Costs in Practice
The computational cost depends on sequence length and nesting depth. CSWX runs one ordered alignment in polynomial time. Enumerating all branch orderings and repeating the full alignment for each combination is far more expensive. RCSWX strikes a middle ground by collapsing alternatives locally, though deeply nested architectures still push it toward exponential worst-case behavior.
In measured runtimes on architectures from evolutionary searches, SEPX and RCSWX found the same edit paths whenever SEPX completed. But SEPX became impractical at around 15 nodes, with comparisons taking hours. RCSWX pushed that boundary to substantially larger architectures. Notably, when the authors compared ResNets with MLP-Mixers, some pairs with larger edit distances aligned faster than pairs with smaller distances, suggesting that similarity alone does not predict computational cost.
Beyond Architecture Search
The alignment procedure does more than produce new architectures. The same edit-distance calculation gives researchers a way to measure how diverse a search population is and to study whether changes in architecture track changes in performance. The authors explore these questions alongside evolutionary search in the grammar-based space introduced by einspace.
The full method and experimental results are available on arXiv, and the reimplementation is on GitHub. For teams building on existing neural designs rather than starting from scratch, this approach offers a principled path from one architecture to the next.