IMPORTANT: yes
When Correct Answers Hide Distinct Internal Worlds
Consider a simple mathematical chain: F(3) = 7, then G(7) = −2. Given the input 3, the correct answer is −2. Now shuffle the rule order so that G(9) = 4 appears first, followed by G(7) = −2, F(5) = 9, and F(3) = 7. The logical dependencies are unchanged, the chain of reasoning is identical, and the answer remains −2. To a human, these two presentations feel like the same problem. But inside a large language model, they may look very different.
Zhixu Silvia Tao of Princeton University investigates precisely this question in a paper submitted on September 23, 2026. The central finding is striking: models that solve reordered mathematical problems more accurately also encode different rule orderings more distinctly in their internal representations. The paper introduces a novel metric called permutation signal-to-noise ratio (SNR) to quantify this phenomenon and evaluates 16 language models spanning 1B to 8B parameters across three synthetic settings.
The Rule-Shuffling Task: Controlling for Answer, Varying Presentation
The experimental design is carefully constructed to isolate a single variable. Each problem consists of a chain of numerical function mappings, such as F(3) = 7 and G(7) = −2, alongside distractor chains that use the same function symbols with different values. Given an input, the model must follow the relevant chain to reach the correct answer. The distractor chains, with distinct values at each stage, ensure that only one path from the queried input leads to the answer.
The key manipulation is presentation order. The same problem is presented under multiple permutations of its rule lines. One permutation places the relevant rules first, followed by distractors. Another scatters the relevant rules among the distractors. A third reverses the order entirely. Every permutation preserves the exact same rules and the exact same correct answer. The only thing that changes is the order in which the rules appear on the screen.
This construction creates what the paper calls an answer-invariant task: different orderings require the same conclusion. The design is inspired by prior work showing that premise order can affect deductive and mathematical reasoning accuracy, but it goes further by enabling a direct examination of what models represent internally, not just what they output.
The experiments evaluate three settings, parameterized by composition depth D and number of distractor chains m. Setting (2,4) uses two-step composition with four chains, (2,7) uses two-step composition with seven chains, and (3,4) uses three-step composition with four chains. Each setting contains 100 problem instances, each presented under 50 permutation patterns, yielding 5,000 prompts per model. All 16 models receive the identical prompts.
Measuring Representational Distinctness: The Permutation SNR
The paper's methodological contribution is a statistic that quantifies how distinctly a model encodes different presentation orders in its hidden states. The metric, permutation SNR, compares two sources of variation in the model's internal representations.
Between-cluster variation measures how far apart the average representations are for different permutation patterns. If a model treats all orderings as equivalent, these cluster centers should collapse to the same point. Within-cluster variation measures how much the representations of individual problem instances vary within the same permutation pattern. This captures natural differences between problems: different numerical values, different chains, and different queries.
Permutation SNR is the ratio of between-cluster to within-cluster variation. A high SNR means that the model reliably separates representations belonging to different orderings, relative to the natural variation between problem instances. A low SNR means that ordering patterns blend together in the model's representation space.
To compute this statistic, the author extracts hidden states from each model at every layer, mean-pooling over nonpadding prompt tokens. The representations contain only the prompt text, with no answer tokens included, so they capture what the model has encoded before generating its response. Layer-wise SNR values are averaged across all transformer layers and the embedding output to produce one value per model per setting.
The Results: Accuracy and Representational Distinctness Are Linked
Across all three synthetic settings, a clear pattern emerges. Models with higher accuracy on the rule-shuffling task exhibit greater permutation-pattern separability in their hidden representations. The Spearman correlation between layer-averaged permutation SNR and accuracy is 0.86 for the (2,4) setting, 0.73 for (2,7), and 0.62 for (3,4).
The direction of this relationship is counterintuitive and worth dwelling on. If the task is answer-invariant, one might expect that a model which answers correctly regardless of rule order would also develop a representation that is invariant to rule order, collapsing different orderings into the same internal representation. The data suggests the opposite: the models that perform best are also the models that most sharply distinguish between different orderings in their hidden states.
This finding draws a sharp conceptual line between two properties that are often conflated in the evaluation of reasoning systems. Answer invariance means that different presentations of the same problem yield the same output. Representation invariance means that different presentations yield the same internal encoding. The paper's results show that these are not the same thing, and that the former does not imply the latter. Successful mathematical rule composition can coexist with distinct internal representations of equivalent orderings.
Controlling for Model Size
A natural concern is whether the observed correlation simply reflects that larger models are both more accurate and have richer representational structure. The paper addresses this by computing partial Spearman correlations that control for ranked parameter count. The adjusted correlations remain positive across all three settings, with Spearman partial correlations of 0.815 for (2,4), 0.647 for (2,7), and 0.469 for (3,4).
The association remains statistically significant at the 0.05 level in the two-step settings. In the three-step setting, the partial correlation is positive but does not reach significance (p = 0.078), leaving the evidence after size adjustment inconclusive for that condition. The paper acknowledges this limitation and notes that the analyses adjust for parameter count but do not account for shared model ancestry, training data, or specialization.
What the Models Are Actually Doing
The paper's models span a diverse set of architectures and specializations: Qwen2.5 and Qwen2.5-Coder families, Qwen2.5-Math variants, Llama 3.1 and 3.2 series, Mistral 7B, Gemma 3, and NVIDIA's AceInstruct and AceMath families. This diversity makes the observed pattern more robust, since it is not driven by a single architecture or training paradigm.
The problem design is also worth examining more closely. In the (2,4) setting with composition depth 2 and 4 distractor chains, each problem contains 5 rule lines (4 chains × 2 mappings each). The model must identify the chain relevant to the queried input while ignoring 4 distractor chains. As the number of distractors increases to 7 in the (2,7) setting, the reasoning task becomes harder, requiring the model to more reliably suppress irrelevant information. As the composition depth increases to 3 in the (3,4) setting, the chain-following problem becomes more complex, requiring deeper logical chains.
Interestingly, the correlation between SNR and accuracy weakens as the task becomes harder: from 0.86 to 0.73 to 0.62. This could suggest that in harder settings, other factors beyond representational distinctness begin to constrain accuracy, or that the relationship between internal encoding and output performance becomes noisier.
Limitations and Open Questions
The paper is explicit that its evidence is correlational and limited to synthetic tasks. It does not establish that encoding order improves reasoning, nor does it test whether the observed relationship between permutation SNR and accuracy would hold on broader, real-world mathematical tasks. The models share partial ancestry, and the analyses do not control for all potential confounds.
The author identifies several directions for future work. Testing the relationship through interventions on ordering-related representations would help establish whether permutation SNR has causal bearing on reasoning performance. Extending the analysis to broader mathematical tasks, particularly those involving multi-step proofs or formal reasoning, would test the generalizability of the finding. The paper also raises the question of whether training models to be order-invariant, perhaps through explicit data augmentation over equivalent orderings, would change their internal representations and whether such changes would improve downstream performance.
The related work discussion positions the paper at the intersection of several active research areas: symbolic and rule-based reasoning, order sensitivity in large language models, and the geometric structure of model representations. Prior work by Chen et al. (2024) demonstrated that premise order affects reasoning accuracy, while He et al. (2025) used order-centric augmentation to improve performance. McIlroy-Young et al. (2024) studied order independence through set-based prompting. The current paper adds a representational lens to this body of work, asking not just whether order matters for accuracy but what order means inside the model.
Why This Matters
The distinction between answer invariance and representation invariance has practical implications for how we evaluate and design reasoning systems. If accuracy alone is the metric, a model that answers −2 regardless of rule order passes the test. But if the model internally treats the two orderings as completely different problems, it may be less robust to perturbations, less sample-efficient when learning new tasks, and harder to interpret or debug. A model whose representations respect the answer-invariance of the task might generalize more cleanly and be more amenable to verification.
Permutation SNR offers a diagnostic tool that goes beyond final-answer scoring. It can be applied to any task where the same content is presented under different permutations, making it a potentially general-purpose measure of how deeply a model has internalized the structure of a problem versus merely matching inputs to outputs. For developers of reasoning systems, the finding that higher accuracy coexists with greater representational distinctiveness suggests that optimizing for accuracy alone may not produce models whose internal representations reflect the true symmetries of the problems they solve.
The paper does not resolve whether representational distinctiveness is a cause or a consequence of better reasoning. But it opens a productive line of inquiry: perhaps the path to more robust mathematical reasoning in language models runs not through forcing representational invariance, but through understanding and controlling the representational structures that already accompany accurate answers.