Automatic speech recognition systems and audio language models now report low error rates on monolingual benchmarks. But a large fraction of the world's speakers are bilingual and routinely code-switch, alternating between two languages within a single utterance. This breaks the monolingual assumptions baked into most systems: tokenizers, language priors, and decoding all degrade at language boundaries. The degradation is especially acute when one language is low-resource and orthographically rich, as is the case for many African languages. Yet evaluation practice has not kept pace with the models. Aggregate word error rate, the default metric, sums errors over an utterance that is usually dominated by the matrix language, diluting errors at the rarer switch points and embedded-language tokens. A model can appear strong by WER while failing precisely where code-switching happens.

An evaluation-first approach to English-Yoruba code-switching

Chibuzor Okocha and Christian Grant at the University of Florida take an evaluation-first stance. Rather than proposing a new model, they ask how well current systems actually transcribe English-Yoruba code-switched speech, and whether standard WER explains their behavior. They benchmark eleven systems under a shared, deterministic protocol on the AfriCodeSwitch corpus: 2,000 utterances from 100 speakers across 13 domains, averaging 4.2 seconds and 8.52 words per utterance with 2.6 switches on average. The corpus is balanced across language dominance, with 2,799 English-dominant, 3,336 balanced, and 2,046 Yoruba-dominant utterances.

The six ASR models span transducer (Parakeet-TDT-0.6B-v2), attention-encoder-decoder (Whisper-large-v3), and LLM-decoder (Qwen3-ASR-1.7B, Canary-Qwen-2.5B, Granite-Speech-4.1-2B) designs. The five audio LMs are Kimi-Audio-7B-Instruct, Audio Flamingo 3, Audio Flamingo 2, Qwen2-Audio-7B-Instruct, and Qwen2.5-Omni-7B. All are evaluated on the same 2,000 utterances with a shared scoring pipeline.

Metrics that localize errors to switch boundaries

The paper introduces several switch-aware diagnostics beyond WER. The Switch-Entry Token Error Rate (SETER) scores only the first token of each new language run, measuring whether a system "lands" each switch. The Switch-Point Error Rate at window k (SPER@k) expands the scoring window to include tokens within distance k of the nearest switch entry, localizing errors to the boundary region. Language-specific error rates (EN-ER and YO-ER) score only English or Yoruba tokens respectively. A diacritic-insensitive WER strips tone marks and sub-dot characters before scoring, isolating the error attributable to Yoruba diacritics.

WER hides where systems actually fail

The central finding is that aggregate WER ranks systems differently than switch-aware metrics. Parakeet-TDT-0.6B-v2 achieves the best WER at 66.1%, but Kimi-Audio-7B-Instruct is statistically indistinguishable on WER (66.2%, p=0.76) while winning on every switch-localized metric. Kimi-Audio's SETER is 66.8% versus Parakeet's 68.9% (p less than 0.001), its SPER@1 is 65.0% versus 67.2% (p less than 0.001), and its SPER@3 is 63.2% versus 65.2% (p less than 0.001). WER rank is essentially uncorrelated with Yoruba token error (Spearman rho = -0.10) and only moderately correlated with SETER (rho = 0.67, not significant at alpha=0.05). WER is a poor proxy for code-switching fidelity.

Yoruba recognition collapses across every system

Among the eight faithful systems (those that produce verbatim transcriptions rather than over-generating), Yoruba token error rates range from 96.6% to 99.4%. English token error rates range from 33% to 46%. Systems recognize English tokens roughly two-thirds of the time and Yoruba tokens almost never. The error is not uniform across the utterance: it spikes sharply at switches into Yoruba (switch-token error 0.95 to 1.00 across systems) while remaining much lower at switches into English (0.40 to 0.53). Denser switching hurts boundary fidelity more than aggregate WER, with SETER rising from 66.7% at low switch density to 72.1% at high density.

Stripping diacritics barely changes WER for most systems (Parakeet +0.06 points, Canary -0.05, Qwen3-ASR -0.29), with the exception of Kimi-Audio (+1.85 points, consistent with it producing more Yoruba content). Diacritics are a second-order concern. Base Yoruba recognition must improve first.

Generative audio LMs fail as exact transcribers

Three audio LMs have WER above 100%, indicating systematic over-generation: Audio Flamingo 2 (119.8%), Qwen2-Audio (137.1%), and Qwen2.5-Omni (202.9%). The failure modes are distinct. Qwen2-Audio has a 31.9% prompt leakage rate, echoing instruction text in its output, with insertions accounting for 46.2% of errors. Qwen2.5-Omni has a 3.1% repetition rate and 3.7% length expansion rate, with insertions at 64.9% of errors. These models frequently translate or summarize instead of transcribing, and occasionally emit text in unrelated scripts.

Prompt design has a dramatic effect on these models. Qwen2-Audio's WER drops from 135.6 with the primary prompt to 103.3 with a direct transcription prompt (cutting 32.3 points), and its prompt leakage drops from 33.7% to 0.2%. But the anti-translation prompt actually increases WER to 139.6. Qwen2.5-Omni shows a similar pattern: the direct prompt cuts WER from 183.0 to 125.9. A single prompt can understate or overstate a model's true transcription ability, which means leaderboard rankings based on a single prompt are unreliable for these models.

What this means for the field

The paper's contributions are concrete and actionable. The released manifests, metric implementations, and evaluation scripts provide a reusable infrastructure for switch-aware benchmarking of African code-switched speech. The switch-localized diagnostics reveal that systems failing at Yoruba switches are not captured by WER, which means current leaderboards are incomplete for multilingual applications. The prompt sensitivity results for generative audio LMs are a warning: these models should not be evaluated on transcription tasks with a single prompt, and their over-generation behavior needs separate handling from faithful ASR evaluation. For practitioners deploying speech systems in bilingual African contexts, the message is that WER on a monolingual or aggregate benchmark does not predict real-world performance on code-switched speech. Read the paper on arXiv