Reading Between the Lines: Why Conspiracy Detection Needs Agents That Ask Questions
Detecting conspiracy theories on social media is not just a matter of identifying certain words or phrases. A tweet that says "the vaccine is part of a secret government plot" could be genuine endorsement, a sarcastic mockery, a legitimate concern expressed in hyperbolic language, or a debunking reference. The surface text alone is often insufficient to determine which. A new paper from Ben-Gurion University of the Negev argues that the task must be reframed as a socially embedded interpretation problem -- and that an agentic framework, capable of selectively retrieving social context on demand, outperforms every text-only approach tested.
The paper introduces an agentic system that uses four tools to query tweet metadata, author profiles, prior user posts, and the user's social network. Evaluated on a manually annotated dataset of 504 Hebrew tweets drawn from a corpus of 270 million public tweets spanning 2018 to 2023, the agentic framework achieves an F1 score of 0.730, compared to 0.535 for a text-only LLM baseline -- a 36% relative improvement. The system significantly outperforms even a degenerate agent that receives the same contexts as input but cannot actively decide which to query.
Why Text Classification Fails at This Task
The core difficulty is not recognizing conspiracy-related content. It is determining the speaker's illocutionary force -- whether the utterance expresses endorsement, criticism, satire, or mockery. The same surface markers that signal a conspiracy narrative can serve entirely different communicative functions depending on the social context.
The paper illustrates this through five carefully selected examples. In one case, a tweet about crop-duster planes and dirty skies is labeled as conspiracy endorsement (the user genuinely believes in chemtrails). In another, a tweet that explicitly references Pfizer's "secret plot" and the "New World Order" is labeled as non-conspiracy -- it is sarcasm, mocking a conspiracy theory by exaggerating it to absurdity. Both GPT-5 and Gemini 3 Flash classify the sarcastic tweet as conspiracy endorsement when given the text alone. Only when provided with contextual information -- the date of the tweet, the author's profile stating they like to "talk nonsense," their prior weather-related posts, and their ego network discussing the actual storm -- do the models arrive at the correct label.
Table 1 in the paper shows this contrast starkly: two state-of-the-art LLMs agree on the correct label for only two of the five examples. They disagree on the remaining three, including the sarcastic tweet and a post about Israeli hospitals and political grievances that could be either genuine complaint or political satire.
Three types of context matter. Temporal context may be needed when a tweet indirectly refers to a recent event. Conversational context matters when a tweet responds to or mocks another post. Social context -- the author's usual stance, community affiliation, and repeated narratives -- may be the most decisive signal of all.
The Agentic Framework
The authors design a system based on the ReAct paradigm, interleaving reasoning with tool use in an iterative loop. The agent receives a target tweet and a task definition, then can call one of four tools to retrieve additional context as needed.
The first tool retrieves tweet metadata: timestamp, text length, hashtags, URLs, and source. This can situate a tweet in a temporal or topical context, or flag unusual posting behavior. The second tool retrieves author profile information: username, bio, account creation date, follower and following counts, verification status, and location. This provides cues about the author's identity and self-presentation.
The third tool samples up to 20 tweets posted by the same author in the 30 days preceding the target tweet. This reveals prior stance, recurring narratives, and ongoing themes. The fourth and most sophisticated tool recovers the user's ego-network based on retweet interactions in the 14 days preceding the target tweet. It returns the five most frequently retweeted accounts along with their profile information and recent posts, giving the agent a window into the immediate community the author moves in.
Critically, the tools only access data timestamped before the target tweet was posted. There is no leakage from future information. The agent decides on a per-case basis which tools to call and in what order, asking only for evidence relevant to its current reasoning step.
Evaluating Context-Aware Classification
The evaluation is designed to separate three factors that are often conflated: the availability of contextual evidence, the type of evidence provided, and the mechanism through which it is accessed. The authors therefore evaluate two main experimental settings: a preloaded mode where contexts are appended to the input for a degenerate agent, and an agentic mode where the agent actively retrieves contexts through tools.
The results are unambiguous. The text-only LLM baseline achieves accuracy of 0.603 and F1 of 0.535. The degenerate agent with all contexts preloaded achieves accuracy of 0.707 and F1 of 0.670. The agentic framework with all tools available achieves accuracy of 0.790 and F1 of 0.730.
The comparison between the degenerate agent and the agentic framework, both with access to all the same contexts, is the most revealing. Both have the same information available to them, yet the agentic framework performs significantly better. The reason, the authors argue, is that providing all contexts to every tweet introduces noise. A tweet that is genuinely a conspiracy endorsement may be obscured by irrelevant network information, while a sarcastic tweet may be clarified by just one piece of context -- the author's profile or a single prior post. The agent finds what it needs and ignores the rest.
Looking at individual tools, user history alone (the degenerate setting with prior tweets preloaded) achieves accuracy of 0.748 and F1 of 0.698, outperforming the full preloaded-context degenerate agent. This suggests that the author's own prior discourse is the single most informative context signal, but even that signal is best exploited by an agent that can reason about when to consult it.
The two-agent debate setting, inspired by Socratic questioning, pairs Gemini 3 Flash as Analyst 1 with GPT-5 as Analyst 2. Analyst 2 critiques Analyst 1's reasoning and classification, with particular attention to non-literal interpretations. This achieves accuracy of 0.683 and F1 of 0.558 -- better than the text-only baseline but falling short of the single-agent approaches. The debate format exposes alternative interpretations but introduces its own noise, and the fixed workflow cannot selectively retrieve the specific evidence each case requires.
What the Errors Reveal
Error analysis provides further insight into how the framework succeeds and fails. When the agentic framework misclassifies tweets that the text-only baseline classified correctly, 78% of those errors are false positives -- the agent incorrectly labels a non-conspiracy tweet as conspiracy. This suggests that incomplete or poorly selected contexts can sometimes distort interpretation, particularly when a tweet contains strong conspiracy markers that the agent overweights.
The shared errors -- cases where both the agent and the baseline fail -- tend to involve deeply ambiguous cases where even a careful human reading would be uncertain. These cases often involve niche conspiracy topics with in-group jargon, or tweets where the communicative function is genuinely indeterminate.
The confusion matrices tell a clearer story than aggregate metrics. The text-only baseline produces 142 false positives and 58 false negatives. The full agentic framework reduces these to 76 false positives and 30 false negatives. Context helps both identify conspiratorial discourse and reject non-conspiratorial tweets that happen to contain similar surface-level language.
Efficiency and the Token Economy
A practical concern with agentic systems is efficiency. Every tool call adds latency and tokens. The authors provide an analysis of the token economy tradeoffs between the agentic framework and the degenerate approach.
The degenerate agent with all contexts preloaded receives the full text of all tool outputs in every case, regardless of whether they are relevant. The agentic framework retrieves context selectively, often using only one or two tools per tweet. The paper reports that the agentic framework achieves better performance with fewer total tokens consumed than the degenerate approach using all preloaded contexts, demonstrating that selective retrieval is not only more accurate but also more economical.
The analysis of tool-use patterns shows that different tools are used with different frequencies. The tweet-information tool is invoked most often, since metadata provides a quick initial signal about timing and topicality. The ego-network tool is used least frequently, but when it is used, it tends to be decisive -- the social circle evidence often tips the classification in ambiguous cases.
Why This Approach Generalizes Beyond Hebrew
The choice of Hebrew as the evaluation language is both a constraint and a feature. Hebrew is a low-resource, morphologically rich language, and conspiracy detection in any language benefits from social context. But the Hebrew Twitter ecosystem has particular characteristics that make the task challenging: a politically polarized environment spanning multiple election cycles, a global pandemic with intense vaccination debates, a tight-knit community with distinctive in-group jargon, and a culture of political satire that blurs the line between mockery and endorsement.
The paper's argument is not specific to Hebrew. The core claim -- that conspiracy detection is an illocutionary task requiring social context -- applies to any language, any platform, any community. The tools described are generic: metadata retrieval, profile lookup, historical tweet sampling, and network analysis. Any platform with equivalent data access could support the same framework.
Limitations and Ethical Considerations
The paper acknowledges several limitations. The evaluation dataset, while unique in its coverage, is limited to 504 annotated tweets -- a small fraction of the 33,121 candidate tweets in the adversarial pool. The manual annotation process is labor-intensive and requires domain expertise, which constrains the speed at which larger labeled datasets can be produced.
The system's performance depends on the availability of social context data. On platforms where author profiles are private, tweet histories are restricted, or network data is unavailable, the agentic framework would have fewer tools at its disposal. The paper does not evaluate the framework in such degraded settings.
There are also ethical considerations. The system classifies tweets as conspiracy or non-conspiracy, which could be used for content moderation, research, or surveillance. The authors note that the system treats context as interpretive hints rather than proof of intent, and that the binary classification scheme inevitably collapses the rich spectrum of communicative functions into two categories. They also note that the task of conspiracy detection is complicated by the fact that discussing a conspiracy does not necessarily imply endorsing it, and that challenging a dogma with alternative narratives can sometimes expose genuine wrongdoing.
Nevertheless, the paper makes a compelling case that the future of conspiracy detection lies not in better text classifiers but in systems that can reason about social context. The findings support a simple but powerful idea: effective classification depends not only on access to context, but on adaptive reasoning that uses tools selectively, asking only for the evidence relevant to each reasoning step.