When a politician's speech is taken out of context, a podcast host misstates a statistic, or a social media video claims something that never happened, the claim lives in audio — but the evidence that could refute it lives in text. Fact-checking systems have largely been built around written claims and textual evidence, leaving a growing category of misinformation in spoken media poorly served. A team of researchers from IISER Bhopal, King's College London, and MBZUAI tackles this gap with VeriSpeak, a probe benchmark and systematic study of whether Large Audio Language Models can verify factual claims presented in speech. The paper, submitted September 24, 2026, introduces both a new evaluation benchmark and a set of striking findings about how current speech-capable models handle fact verification.
Why Speech Fact-Checking Is a Different Problem
Text-based fact verification has matured considerably over the past several years. Benchmarks like FEVER, HoVer, and SciFact have established a reliable paradigm: retrieve relevant textual evidence, then classify whether a written claim is supported or refuted by that evidence. The retrieve-and-verify pipeline works because the claim and the evidence share the same modality — both are text. The model never has to translate between representations; it simply compares semantic content.
Speech fact verification breaks this assumption. The claim arrives as audio, encoded in acoustic patterns that a model must first decode into linguistic meaning. The evidence arrives as text, drawn from knowledge bases or retrieved documents. The model must maintain a clear separation between what was said and what the evidence says, then compare the two. This cross-modal setting introduces failure modes that have no direct analogue in text-only verification.
The stakes are real. Misinformation increasingly circulates as audio — podcast clips edited to imply false statements, political speeches decontextualized on social media, deepfake interviews. A system that can only verify written claims is blind to a large and growing fraction of online misinformation. VeriSpeak is designed to probe exactly this vulnerability.
Constructing a Controlled Testbed
VeriSpeak contains 3,879 spoken claims spanning three fact categories: temporal facts (1,451 items involving years and dates), geographical facts (2,226 items involving locations and nationalities), and relational facts (202 items involving employment, kinship, and organizational affiliations). Each item is balanced between true and false labels, preventing models from exploiting label imbalance as a shortcut.
The construction pipeline is deliberately rigorous. Starting from celebrity biographies, the authors enrich each record with structured metadata from Wikipedia, resolving birth countries, genders, and historical entity names. Facts are flagged by category, then extracted as single unambiguous atomic sentences by a smaller language model. To generate plausible false counterparts, the authors apply category-specific perturbations: temporal facts have years shifted within a ten-point range, geographical facts swap locations with same-type candidates from the knowledge base, and relational facts either swap relation words or replace entities entirely. This ensures that false claims are deceptive enough to require genuine reasoning rather than surface-level cues.
Critically, all speech is synthesized using a single speaker to prevent models from exploiting speaker identity as a spurious verification cue. Each item also includes an automatic transcript generated by Wav2Vec 2.0, so researchers can separate speech recognition errors from reasoning failures.
The benchmark covers 659 subject entities across 60 countries, with roughly balanced gender representation. The test suite evaluates five LALMs from three model families — Qwen, Qwen2.5, and Phi — under six controlled conditions ranging from text-only verification to speech-only verification to retrieval-augmented verification with and without chain-of-thought reasoning.
The Text-Speech Modality Gap
The first major finding is a large and consistent gap between how well models verify written claims versus spoken ones. Qwen-7B achieves 75.3 percent accuracy in text-only mode, but its speech-capable counterpart Qwen-Audio-Chat reaches only 50.4 percent on the same claims — a 24.9 point drop. Phi-4-multimodal drops from 68.0 percent to 49.2 percent, an 18.8 point gap. Audio-Flamingo-3, which performs best among the LALMs in speech-only mode at 58.5 percent, still falls 13.3 points below its 71.8 percent text-only baseline.
The authors decompose this gap into two components. The first reflects whether the multimodal adaptation itself degrades text-side factual ability — that is, does the LALM perform worse than its text-only backbone when both receive written claims? For Qwen-Audio-Chat, this text-mode degradation is 9.7 points, suggesting that adding speech capabilities can cannibalize text verification. For Qwen2-Audio-7B and Phi-4-multimodal, this component is negligible or even slightly negative, meaning the LALMs match or slightly outperform their text backbones on written claims.
The second component — the drop caused by switching from text input to speech input within the same LALM — is uniformly large across all models. It ranges from 6.5 to 27.2 points. This suggests that the core problem is not that multimodal adaptation destroys existing knowledge, but that the speech interface itself fails to reliably elicit factual verification behavior. The relevant factual knowledge may exist in the model's parameters, but the path from acoustic signal to verification decision is unreliable.
Retrieval Helps, But Only Partially
If the speech interface cannot reliably access parametric knowledge, perhaps retrieved textual evidence can compensate. The results show that retrieval alone provides a modest improvement. Transcript-based retrieval using the multi-e5 retriever lifts average accuracy by only 2.3 points across standard LALMs — from 50.4 to 51.8 percent for Qwen-Audio-Chat, from 49.2 to 50.9 for Qwen2-Audio-7B, and from 49.2 to 53.9 for Phi-4-multimodal.
More troubling is why the gains are so limited. When the authors sampled 50 transcript-retrieval examples and asked the model to extract what claim it was verifying, only 31 percent of extractions matched the original spoken claim. A striking 65 percent matched the retrieved evidence instead. In other words, the models were more likely to treat the retrieved passage as the statement to verify than as evidence for verifying the spoken claim. This claim-evidence conflation means that even when the retriever finds the right document, the model uses it for the wrong purpose.
Audio-query retrieval — where the audio claim itself serves as the retrieval query via a CLAP-based retriever — performs worse than transcript-based retrieval. Its average lift is negative at -0.3 points, consistent with CLAP's extremely low Recall@1 of approximately 0.3 percent. The top-ranked passage rarely provides relevant evidence, leaving the model without useful context.
Among transcript-based retrievers, multi-e5 is the most reliable overall with Recall@1 values of 85.6, 78.2, and 49.8 percent across the three model families. The e5-large-v2 retriever performs second-best, and msmarco-MiniLM is competitive for some models. But even the best retriever cannot compensate for the model's inability to properly use retrieved context.
When Reasoning Makes the Difference
The relationship between reasoning and fact verification is more nuanced than the common intuition that chain-of-thought prompting always helps. On its own, CoT does not improve speech-only verification. For standard LALMs, adding CoT actually hurts performance: Qwen-Audio-Chat drops from 50.4 to 44.9 percent, and Phi-4-multimodal collapses to just 16.7 percent due to format failures. The average Δ increases, meaning the models move further from their text-only baselines.
This counterintuitive result reveals an important principle: reasoning is not useful for recovering factual knowledge from speech — that job belongs to retrieval. Reasoning is useful for comparing what was said against what the evidence says. When CoT is paired with retrieved evidence, the picture changes dramatically. Transcript-RAG with CoT improves every standard LALM: Qwen-Audio-Chat gains from 51.8 to 52.8 percent, Qwen2-Audio-7B from 50.9 to 57.0 percent, Audio-Flamingo-3 from 60.0 to 65.5 percent, and Phi-4-multimodal from 53.9 to 60.7 percent. The average improvement from adding CoT to retrieval is 4.9 points.
The most dramatic results come from Audio-Flamingo-next-think, a thinking-tuned LALM trained with explicit reasoning. Unlike standard models, it benefits from CoT even in the speech-only setting, improving from 66.4 to 71.8 percent. With transcript-RAG it reaches 81.4 percent, and with RAG plus CoT it achieves 86.1 percent accuracy — a 19.7 point lift over its speech-only baseline and 14.3 points above the text-only non-retrieval baseline. This means that for a reasoning-trained LALM, speech-based verification with retrieval can actually outperform text-only verification without retrieval.
However, even the best model remains 6.3 points below the text-only RAG upper bound of 92.4 percent. A measurable gap persists between speech-based and text-based verification, indicating that the speech interface introduces irreducible friction.
Where the Errors Actually Come From
A word error rate analysis helps distinguish speech perception failures from reasoning failures. Across all models, WER ranges from 0.13 to 0.23, with Audio-Flamingo-next-think achieving the lowest rate at 0.14. For comparison, human transcribers achieve a WER of approximately 0.105 on the same speech. The difference is relatively small, suggesting that the core speech is intelligible.
More revealing is the composition of the errors. Approximately 74 percent of edit operations involve proper nouns — subject names like "Richie" versus "Ritchie" or "Bachelet" versus "Bachellet." Non-name tokens have an error rate of only about 3 percent. For fact verification, where names and locations are often label-critical, these entity recognition errors are directly consequential. A model that mishears a politician's name cannot correctly verify claims about that politician, regardless of how well it reasons over retrieved evidence.
The additional diagnostic condition C0′, where the model's own transcription is fed as text to the text LLM, quantifies how much of the performance gap is attributable to recognition errors versus reasoning errors. This decomposition provides evidence that while recognition errors contribute to the modality gap, they are not the sole or even primary cause — the verification reasoning itself is also degraded by the speech interface.
Implications and Open Questions
The findings from VeriSpeak carry practical implications for anyone building or deploying speech-based fact-checking systems. First, speech input alone is not sufficient. A model that performs well on text cannot be assumed to perform equivalently on speech, even when the same model family is used. The modality gap is real and large, ranging from 13 to 27 percentage points depending on the model.
Second, retrieval-augmented approaches must address the claim-evidence conflation problem. Simply feeding a model a spoken claim alongside retrieved documents is not enough. The model needs explicit reasoning prompts or training that teaches it to treat the spoken claim as the object of verification and the retrieved text as evidence. The chain-of-thought prompting approach works, but primarily when combined with retrieval — reasoning without evidence is destabilizing.
Third, thinking-tuned models represent a meaningful advance. Audio-Flamingo-next-think's 86.1 percent accuracy demonstrates that reasoning-specific instruction tuning can substantially improve speech fact verification, though it does not eliminate the gap to text-only RAG.
The authors note several limitations. The benchmark currently covers celebrity facts only, which limits the range of domains and claim types. The single-speaker synthesis, while controlled, does not represent the diversity of real-world speech. The reliance on automatic transcripts introduces transcription noise, though the authors argue this reflects realistic deployment conditions. Future work could explore whether entity-specific recognition improvements close more of the gap, and whether domain adaptation to political or news speech narrows the modality divide.
VeriSpeak provides the research community with a controlled testbed for systematically analyzing speech fact verification. Its structured category design, balanced labels, and controlled evaluation conditions make it a valuable resource for probing the boundaries of current LALM capabilities — and for diagnosing exactly where those capabilities fall short when the claim lives in speech and the evidence lives in text.