The Problem: Type Safety Guarantees the Form, Not the Meaning
A growing class of production language models now sells decisions rather than text. Instead of generating free-form prose, these systems return a probability distribution over a predefined set of options, with illegal candidates masked before normalization. By construction, every output conforms to the required schema. Vendors advertise this as a structural guarantee, listing "type-safe" beside "self-consistent" as properties of the same artifact.
But a guarantee about the form of an output tells us nothing about whether the model interpreted the options correctly. A typed decision head that returns "yes" when it should return "no" still passes every type check, yet produces a semantically wrong answer. This is the gap that Yu Sun of the National University of Singapore and Junhao Xu of Fudan University expose in their paper "Type-Safe Is Not Error-Free."
Background: Why Schema Validity Is Not Semantic Validity
Language model sensitivity to prompt formatting, option identifiers, and label words has been documented extensively. Prior work has shown that small models ride semantic priors rather than override them, and that schema-constrained outputs can be semantically wrong even when structurally valid. Studies have also shown that schema descriptions can override explicit instructions.
What prior work has not done is isolate this effect with surgical precision. The key challenge is one of experimental design: in a typed decision interface, each option consists of an option name and a textual rubric that defines what the option means. Both are visible to the model in the input. But they are always bound together, making it impossible to tell whether a model is reading the name, the rubric, or some combination.
Sun and Xu solve this by constructing a clean controlled experiment. They can hold the question, the state, the rubric text, and the set of option names byte-identical between conditions, and exchange only which rubric is bound to which name. A model that reads the definition should answer identically in both arms. A model that reads the name should invert. The remapping is stated declaratively in the same input; nothing is hidden.
The Method: The Label-Rubric Swap
The core instrument is elegant in its simplicity. Each option is rendered as its name followed by its definition, like "no:
The aligned arm binds each name to its own rubric as shipped. The swapped arm exchanges the two rubrics between the two names. Gold is keyed to the rubric throughout, so a rubric-reading model is unaffected by the swap and a name-reading model inverts. Nothing else changes: identical question, identical state, identical rubric strings, identical option set, identical number of options.
To isolate the word from the channel, the authors also repeat the identical structural operation with neutral option names that carry no polarity: 0/1 and A/B. These serve as controls. Any movement they show is what re-binding costs when the name means nothing.
The evaluation is conducted on 1200 binary workflow decisions across four predicates: invoice reconciliation, agent-trace triage, security-alert classification, and customer-service escalation, with 300 items each. The gold-positive rate is .5750. Every item carries the dataset's own task-specific rubric text. A competence screen confirms that the aligned arm achieves balanced accuracy between .7857 and .9310 across the four predicates, so all reported effects are measured on decisions the head demonstrably makes correctly.
Results: Confident Inversion, Not Confusion
The headline result is stark. Swapping the rubrics behind no and yes changes 76.92% of the returned answers, while the identical exchange behind 0 and 1 changes only 6.50%, and behind A and B only 6.00%. The controlled difference-in-differences is 70.42 percentage points, with a 95% confidence interval of [67.58, 73.08].
More telling than the flip rate is what happens to ranking quality. Balanced accuracy falls from .8719 to .2839, moving from well above to well below chance. AUROC collapses from .9376 to .2315, far below the 0.5 threshold. This is the signature of confident inversion rather than confusion: the ranking is intact and pointed the wrong way. Inverting the swapped scores recovers AUROC .77. The head is not uncertain about the predicate; it is confidently answering a different question, the one its option names suggest.
The finding holds across all four predicates independently. For no/yes, the minimum flip rate across predicates is 56.7% (customer-service escalation), against a neutral control that never exceeds 11.3%. Every polar pair flips at least 7.4 times as often as the neutral control. The paper quotes the minimum rather than the mean because relabelings differ in how far they preserve meaning, and the minimum is the conservative bound that survives scrutiny.
Polarity is the driving force, but training-vocabulary familiarity amplifies it. The five polar pairs split into two groups: no/yes and false/true are words this head emitted during training and the words the interface ships, whereas absent/present, negative/positive, and rejected/accepted are polar words it never emitted. Moving from neutral names to unfamiliar polar names costs 41.00 percentage points; moving from unfamiliar polar names to the two familiar ones costs a further 16.04 points. The effect is not reducible to memorized output vocabulary, since it acts on words the head was never trained to produce.
Above binary cardinality, the problem intensifies. On 683 genuine multi-way questions, renaming members to neutral letters changes 52.42% of answers and drops accuracy from .5637 to .2782. Rotating names one step against descriptions, so each name advertises its neighbor's content, changes 79.65% and drops accuracy to .1552. At k=16, neutral renaming lands at .141, exactly the share of items whose gold member happens to sit first positionally rather than semantically.
The Effect Replicates Across Architectures and Vendors
Critically, this is not a quirk of one checkpoint. The authors run the identical swap through three distinct read-out families. The marker read-out, which scores a single [MASK] marker token per option, shows the largest effect: 80.50% of answers change for no/yes. The span-mean read-out, which averages over the entire rendered span including the rubric, shows a 4.1 times smaller effect at 19.50%. The hosted vendor model, queried over HTTP, shows 32.50% of answers changing for no/yes.
The span-mean geometry provides a clear mechanism: when an option's score is the mean over its entire rendered span, a one-token name is averaged against a rubric an order of magnitude longer, and its influence is diluted in proportion. The marker geometry has no such dilution because the score comes from a single token that attends over the span. Consistently, the span-mean head is strongly name-driven only where there is no rubric to dilute the name.
The hosted model exhibits the same fundamental failure even though its AUROC falls from .8146 to .5806, toward chance rather than past it. The vendor model's type-error rate is 0% by Lemma 1, just like the open-weight checkpoints. Across all three families, neutral names move the fewest answers, polar pairs outside the marker head's vocabulary move more, and the two inside it move most. The ordering holds everywhere; only the magnitudes differ.
The Decisive Test: Random Names Eliminate the Problem
To prove that the failure depends on semantic polarity rather than the act of renaming itself, the authors introduce a fourth name class: each option is named by a random 5-character string over letters and digits, drawn so neither name is a word or prefix of the other. These names carry no polarity and no content at all.
Opaque names land on the neutral class in all three families. They change 6.86%, 11.67%, and 2.02% of answers across the marker, span-mean, and hosted models respectively, essentially indistinguishable from the 0/1 and A/B controls. Aligned balanced accuracy is retained throughout, meaning the low flip rate reflects a decision still being made, not a channel switched off. Polar names on the same items and read-outs change 32.21% to 70.72%. What the swap exploits is therefore the option name's polarity: with polarity absent, the name's semantic content buys nothing over an identifier that means nothing in particular.
Practical Implications
The paper offers two direct mitigations. First, use neutral option identifiers and carry the meaning in the rubric. On the data here, this costs 6.50% instability instead of 76.92%. Second, bind the decision to the rubric during training by randomizing option names, which is computationally cheap for a head of this size. Both are practical interventions available today.
For anyone deploying typed decision models, the central lesson is that a 0% type-error rate is a property of the decoder's read-out, not a measurement of reliability. The paper argues that every typed decision deployment should be accompanied by a name-invariance number. The flip rate against neutral option names requires only two extra forward passes per item and no labels, yet on these checkpoints it would have surfaced a 76.92% instability that accuracy on the shipped schema never reveals.
Limitations
The study audits two encoder checkpoints in English. Whether the dilution mechanism transfers quantitatively to other architectures is untested. The third family, a hosted vendor model, is a black box reached over a network at a single point in time; the model served under that name may change, and only returned distributions are observable, not internal weights. Polarity and training-vocabulary familiarity are not cleanly separated by the experimental arms, though the direction of the confound is acknowledged. The multi-way arm rests on a pool whose shipped accuracy is .5637, so it is treated as directional support rather than a second headline. The paper reports the vulnerability and two mitigations but does not evaluate them.
Conclusion
The paper's title states its central finding with precision. A typed interface guarantees the form of a decision, and that guarantee is analytic: it holds under every perturbation, including those that invert the decision. On a shipped decision head, with the rubric text fixed and gold keyed to it, exchanging which rubric is bound to no and yes changes 70.4 answers per hundred more than the identical exchange behind 0 and 1, and turns a .94-AUROC decision into a .23-AUROC one. Type-safe, and not error-free. The structural guarantee is real, but it constrains only the support of the output distribution, not which element of that support receives the mass. For practitioners who depend on typed decisions in production pipelines, this distinction between valid form and valid meaning is everything.