The compositional generalisation problem has haunted Transformer research for years. Train a seq2seq model on "John drinks coffee" and it learns to translate that just fine. But give it "John drinks coffee on a mat" when only "on a mat" appeared in object position during training, and it falls apart. This failure pattern, where structural generalisation (new grammatical constructions) is harder than lexical generalisation (new words in familiar constructions), has been widely cited as evidence that Transformers lack something fundamental about compositional reasoning. A new paper from Aalto University and the University of Helsinki argues the evidence was misleading all along.
The Structural vs. Lexical Divide
Compositional generalisation splits into two categories. Lexical generalisation means encountering familiar structures with new words: you have seen "drink coffee" and "eat bread" and now you must handle "drink bread." Structural generalisation means encountering new structural patterns: you have seen "the cat" and "the dog drinks" but never "the cat drinks" (a particular construction). Previous work, especially on the COGS benchmark, found that Transformers handle lexical cases well but fail on structural ones. This led to a widespread belief that Transformers have a specific architectural limitation with structural compositionality.
Three hypotheses competed to explain the gap. The architecture hypothesis blamed Transformers directly. The unfair task hypothesis argued structural generalisation is simply a harder, less well-defined task. The concrete syntax hypothesis suggested incidental surface properties of the logical form (variable numbering, token ordering) were the real culprits. The new paper proposes a fourth explanation: the problem is not with the model or the task definition, but with the datasets themselves.
Type Diversity: A Precise Definition
The authors introduce the concept of type diversity, measured using Grammatical Framework (GF), a formalism that separates abstract syntax (what structures exist) from concrete syntax (how they are written as strings). In GF, linguistic types are categories like NP (noun phrase), VP (verb phrase), and Imp (imperative sentence). Each type has constructor functions that build trees of that type. For example, the NP type might have constructors for simple determiner-noun phrases ("the cat"), prepositional phrases ("the cat on a mat"), relative clauses ("the cat that sleeps"), and adjectival phrases ("the big cat").
Type diversity is the count of constructors of a given type in the training grammar. This is not the same as lexical diversity (how many different words appear) or general structural diversity (how complex the sentences are). It is a precise, type-level measure of how many different ways the grammar can build trees of a particular category. The original COGS dataset, for instance, has about 3 NP constructors but roughly 20 to 400 constructors for lexical types like verbs and nouns. This imbalance, the authors argue, is why structural generalisation appeared harder.
Using GF to Build Diverse Dataset Variants
The practical challenge is testing the type diversity hypothesis. You cannot simply vary type diversity in existing datasets without controlling other factors. The authors use GF as a data generation tool. For the COGS and SLOG benchmarks, they write GF grammars that extend the original datasets with new constructors: adjectives, plural noun forms, verb tenses, participial adjectival phrases, and relative clauses. They also add artificial NP structures to push diversity further. Dataset sizes are kept approximately constant across variants so that the effect of type diversity is isolated from dataset size.
For SCAN, a purely artificial dataset, they generate 120 training set variants by systematically varying the number of constructors at three hierarchical levels: V (lexical verbs), VP (verb phrases with modifiers), and Imp (imperative sentences). Each level gets a grid of diversity values: V at 1, 10, 30, 60, 90 constructors; VP at 1, 10, 30, 60, 90; Imp at 1, 4, 8, 16. Every combination is generated, and each is trained with 14 random seeds. This gives a clean factorial design for measuring how type diversity at one level interacts with generalisation at every level.
COGS Results: From Two Percent to Ninety-Two
The COGS experiment tests PP-in-Obj to PP-in-Subj generalisation: the model sees prepositional phrases only in object position during training and must handle them in subject position at test time. The original dataset achieves 1.7% accuracy on this structural task. The results track type diversity step by step:
- Adding participial adjectival phrases in subject position: 6.2%
- Adding participial phrases in both subject and object: 16.8%
- Adding adjectives: 26.0%
- Adding relative clauses: 55.4%
- Adding plural forms: 69.3%
- Adding verb tenses: 73.3%
- Adding artificial NP structures: 92.4%
The trend is clear and monotonic. Each new NP constructor increases structural generalisation accuracy. Verb tenses, which add diversity at the VP level but not the NP level, contribute less than half a standard deviation (69.3% to 73.3%). A dataset size control (60% augmentation with word swaps) changes accuracy from 2.1% to 1.7%, confirming that size is not the confounding variable.
The preliminary experiment with PAPs further clarifies the mechanism. When PAPs use "acl" semantics (making them syntactically distinct from PPs), accuracy reaches 32.0% when present in both positions but drops to 0.2% when only in the subject. When PAPs use "nmod" semantics (making them behave like PPs), accuracy reaches 81.8% even when only present in the subject position. This eliminates the concrete syntax hypothesis: the benefit comes from adding a new NP constructor, not from surface-level changes to the logical form.
SCAN Results: No Inherent Lexical-Structural Gap
The SCAN experiments are where the core claim is tested most rigorously. With 120 variants and 14 seeds each, the results show that type diversity correlates with compositional generalisation equally in lexical and structural test cases. Without much type diversity, accuracy hovers around 0 to 3%. With increased diversity, accuracy climbs to 60 to 90% across all three generalisation levels.
The correlations reveal interesting interactions between type levels. At low V diversity (10 constructors), increasing VP diversity actually helps lexical generalisation (Spearman's rho = 0.86). But at high V diversity (90 constructors), increasing VP diversity hurts lexical generalisation (rho = -0.67). This suggests that sub-tree type diversity helps when the type itself is scarce but can introduce noise when the type is already well-represented. Similar patterns appear for Imp generalisation: VP diversity helps at low Imp diversity (rho = 0.58) but hurts at high Imp diversity (rho = -0.65).
The key finding is that there is no significant difference in how type diversity correlates with lexical versus structural generalisation. The previous observation that structural cases are harder was an artifact of low structural type diversity in the specific datasets used, not a property of the Transformer architecture itself.
The Compound Divergence Contradiction
One of the more provocative findings concerns the Distribution-Based Compositionality Assessment (DBCA) metric introduced by Keysers et al. (2020). DBCA measures atom divergence (how different the individual tokens are between train and test) and compound divergence (how different the combinations are). Keysers et al. found a strong negative correlation between compound divergence and accuracy on the CFQ dataset, interpreting it as compound divergence capturing the core difficulty of compositional generalisation.
The SCAN experiments show the opposite. Compound divergence correlates positively with accuracy: Pearson r = 0.94 for lexical generalisation, r = 0.96 for structural, and r = 0.89 for second-order structural. Atom divergence also correlates positively (r = 0.84, 0.97, 0.82 respectively). The resolution is that compound divergence is ambiguous about which side of the train-test split is being diversified. In Keysers et al.'s experiments, more compound divergence meant more novel test combinations (harder task). In this paper's experiments, the test set stays fixed while the training set gains diversity (easier task due to broader support). The metric is too vague to serve as a general compositionality measure.
SLOG: Surface Properties Still Matter
The SLOG experiments add another dimension. SLOG extends COGS with more complex constructions (relative clauses, centre embedding, wh-questions). The authors test how various dataset properties affect generalisation beyond type diversity. Overall accuracy increases from 27.7% to 47.7% when adding NP diversity. PP-in-Subject accuracy jumps from near zero to about 35%. Shallower centre embedding improves from near zero to about 80%. But relative clauses in subject position barely improve: from 0.0% to 1.8%.
The surface property experiments are revealing. Reordering semantic terms in the logical form (aligning them with surface word order instead of GF's internal order) causes the question mark in indirect object wh-questions to be placed incorrectly in 63.2% of aligned-order models versus 18.1% of GF-order models. Adding verb tenses reduces variable binding errors in wh-questions from 22.9% to 2.6%. These results confirm that both type diversity and surface properties of the logical form matter, and neither alone explains the full picture.
What This Means for Benchmarking
The paper's discussion on benchmarking methods makes a pointed argument. Diagnostic datasets like SCAN and COGS were created to isolate compositionality, but multiple studies have shown they are not robust. SCAN has few target-side dependencies, allowing simple models to do well without real composition. Adding more verbs to SCAN achieves near-perfect accuracy on the "add jump" test case. Different compositional generalisation benchmarks rank models in different orders. The authors argue that systematic compositionality is an abstract property that necessarily co-exists with other dataset properties. It is impossible to strip away everything non-compositional.
Instead of trying to purify benchmarks, the recommendation is to map the variables that affect compositional generalisation and control for them when comparing models. Type diversity is one such variable. Surface properties of the logical form are another. Dataset size is a third. Ignoring these factors and attributing performance differences to model architecture is a mistake.
Limitations and Open Questions
The authors are upfront about what the paper does not show. Only Transformers are tested, and results may differ for LSTMs or other seq2seq models. Only one hyperparameter setup is used per experiment (taken from prior work for comparability), and results would likely shift with different configurations. High variance across random seeds is a persistent issue, and conclusions are based on patterns visible despite that variance.
The paper also does not claim that Transformers reach perfect or human-level compositional generalisation with enough diversity. Accuracy typically plateaus before 100%, consistent with other work showing neural networks learn many aspects of grammar without learning to use them systematically. Type diversity explains the gap between lexical and structural generalisation but does not solve the broader problem of achieving truly systematic compositionality.
For practitioners, the takeaway is concrete: when evaluating models on compositional generalisation benchmarks, pay attention to the type diversity of your training set. A benchmark where structural types are underrepresented will make any model look bad at structural generalisation, regardless of its actual capabilities. The code and dataset variants are publicly available, making it straightforward to generate diversified versions of standard benchmarks for more fair evaluation.