When a probing study shows that a speech foundation model can classify word identity from its internal representations, the standard interpretation is that the model has learned something about words. Robin Huo and Ewan Dunbar at the University of Toronto argue this interpretation is too quick. A word like "cat" is both a sequence of phonemes (/kæt/) and a lexical unit with syntactic and semantic properties. If the model encodes phonemes well, it can discriminate words by their sound form without ever representing the word as an abstract unit. The distinction matters because it determines whether these models are genuinely learning linguistic structure or merely building better acoustic phonetic analyzers. The paper uses a simple residualization technique to partial out phoneme information and asks: does word identity information survive?
The confound in probing studies
Self-supervised speech models like HuBERT and wav2vec 2.0 are trained on prediction objectives: predict masked frames, cluster representations, predict discrete units. Probing studies have shown that their internal representations encode phonemes, speakers, and words. But the word probing results are ambiguous. In Saussure's framework, a word token is both a signifier (the phonological form) and a signified (the lexical meaning). Since these models clearly encode phonemes well, successful word probes could be entirely explained by phonological encoding. A representation that captures /k/ + /æ/ + /t/ can identify "cat" without ever treating "cat" as a unit.
Previous work by Choi et al. compared near-homophones with synonyms using cosine similarity, finding weak non-phonological encoding. But cosine similarity has a fundamental flaw: information can be drowned out by irrelevant dimensions even if it is represented with high fidelity on a subspace. A linear classifier operating on specific dimensions can extract information that a global similarity metric misses.
Residualization: subtracting phoneme information
The method is straightforward. For each frame in the representation, fit a ridge regression predicting the representation features from a one-hot encoding of the phoneme label (or diphone, or triphone). Subtract the regression predictions from the true representations. The result is a residualized representation where linear phoneme information has been removed. The approach is not perfect: it may fail to fully remove phoneme information, or may remove more than just phoneme information. But the authors address both concerns. For the first, they validate by probing phoneme identity on the residuals. For the second, removing extra information would only strengthen the conclusion.
The validation experiment on LibriSpeech dev-clean shows the method works. Raw HuBERT representations yield 87% phoneme classification accuracy in later layers. After residualization, accuracy drops to 27-29% in layers 5-11, approaching but not reaching the most frequent phoneme baseline of 11.6%. The failure to reach chance is attributed to the exclusion policy: frames where the phoneme/diphone/triphone fully contains the word are excluded from the regression fit, preventing the regression from inadvertently learning word-level patterns. Without standardization, the method fails completely (accuracy stays at 55-100%), confirming that the standardization step is essential for matching distribution shapes across phoneme classes.
Words persist after phoneme removal
The main experiment probes word identity (8,217 word categories plus silence) from raw and residualized representations across all 13 layers (convolutional layer plus 12 transformer layers) of HuBERT-base and wav2vec 2.0-base. The results are stratified by word length.
In raw representations, both models show the same pattern: little word classification ability in the convolutional layer, rising sharply through the transformer layers, peaking at over 90% accuracy in layers 9-10 (HuBERT) or 7-8 (wav2vec 2.0). Residualizing phonemes does not change this overall trajectory. The drop from raw to phoneme-residualized is modest in the peak layers, and residualized accuracy remains well above the length-specific baselines (14.9% for 3-letter words, 0.4% for 4-letter, 0.1% for 5-letter).
Removing diphones decreases accuracy slightly more than removing phonemes alone, especially in earlier layers. Removing triphones produces the largest drops, but in the peak layers (9-10 for HuBERT, 7-8 for wav2vec 2.0), the picture is mixed: words of length 3-6 show substantial drops but still perform well above baselines. This is the key finding: even after removing information about three consecutive phonemes, the model can still classify word identity with reasonable fidelity. The representation encodes word information that goes beyond encoding short phoneme sequences.
The layer-wise pattern is informative. Early layers (1-4) lose most of their word classification ability after triphone residualization, suggesting they rely heavily on local phonetic context. Later layers retain more word information, consistent with the hypothesis that they integrate broader contextual cues to form word-level representations. This aligns with the general understanding that transformer layers in speech models progressively abstract from acoustic to linguistic information.
Word discovery improves without phoneme information
The practical test is whether removing phoneme information helps downstream tasks that depend on higher-level linguistic structure. The authors evaluate unsupervised word segmentation and clustering on LibriSpeech dev-clean using HuBERT layer 9 representations. The pipeline follows Malan et al.: word boundary detection via peak detection on frame dissimilarity, followed by k-means clustering into hypothesized word categories.
Results are measured by normalized edit distance (NED) of discovered word classes, token F1-score for boundary detection, and R-value capturing the trade-off between recall and oversegmentation. Phoneme residualization improves all three metrics across multiple hyperparameter settings. For the best NED configuration: raw NED = 0.443, phoneme-residualized NED = 0.439. For the best RR configuration: raw NED = 0.527, phoneme-residualized NED = 0.478, raw F1 = 0.166, residualized F1 = 0.175, raw RR = 0.531, residualized RR = 0.537.
The improvement is robust across segmentation hyperparameter settings, which is particularly important for unsupervised lexicon discovery in low-resource settings where little data is available for tuning. The authors note that directly applying this approach requires aligned phonemic labels, which are not typically available in practice. The results should be interpreted as validation of the core insight that removing phonemic information can enhance accessibility of higher-level information, not as a practical system.
Residualization from diphones and triphones, unlike phonemes, degrades segmentation performance. This makes sense: phoneme transition probabilities are predictive of word boundaries (statistical learning research shows infants use transitional probabilities to segment speech). Removing local phonemic context strips away boundary-relevant information, even if it also removes confounding word-form information.
What the models are actually encoding
The results paint a specific picture of what HuBERT and wav2vec 2.0 learn in their later layers. They do encode word identity with reasonable fidelity, and this encoding is not fully explained by local phonetic content. The representations integrate information from surrounding context, essentially creating a "token" that allows words to be identified locally at any frame throughout the word.
This is not a trivial property. HuBERT is trained to predict masks spanning at least 10 frames (200 ms), which at an average phoneme duration of 65 ms contains only about 3 phonemes. The prediction targets are derived from clustering individual, contextless frames. The model could have constructed representations predictive only of local phonemic context. Instead, it predicts words, including quite long ones, with high accuracy.
However, the authors are careful about what this does and does not demonstrate. It shows the model treats words as units, but does not reveal whether it encodes syntactic or semantic properties of words beyond their phonological form. The residualization removes linear phoneme information; non-linear phoneme information may remain. And the integration of context that enables word identification could be a byproduct of the masked prediction objective rather than evidence of abstract lexical representation.
Implications for speech understanding
The result has implications for several active research directions. For speech-aware language models that tokenize speech into discrete units, understanding what those units encode is critical. If the units carry word-level information beyond phonemes, they may be more linguistically rich than assumed. For zero-resource speech processing, where unsupervised lexicon discovery is a key challenge, the finding that phoneme removal enhances word discovery suggests that explicitly disentangling form and meaning in speech representations could improve performance.
The gap between text-based models and speech-derived discrete units in unsupervised tasks like lexicon discovery and textless language modeling remains substantial. This paper suggests one reason: speech representations carry both phonological and lexical information entangled together, and tasks that depend on one type of information are confounded by the other. Simple disentanglement via residualization improves the signal for tasks that need lexical information without phonological noise.
For interpretability research, the paper demonstrates that residualization is a viable tool for understanding what speech models encode. Unlike subspace-based methods that require choosing arbitrary numbers of dimensions to remove and arbitrary importance scores, residualization operates on the full representation and has a clear statistical interpretation: it removes the linear component of one factor while preserving everything else. The approach is limited to linear relationships, but for probing purposes this is often sufficient.
The practical limitation is the need for aligned phonemic labels. The authors acknowledge this and suggest that future work could investigate approximate or inferred labels. Given that forced alignment tools are increasingly accurate and available for many languages, this may be less of a barrier than it appears. The more fundamental question is whether the linear residualization captures the right kind of entanglement, or whether non-linear interactions between phoneme and word information require more sophisticated disentanglement methods.