White-box Deception Detection via Scalable Probes
Recent incidents have highlighted the challenge of monitoring LLM agents and the danger of models deceiving people. We show that white-box deception detection via probes can be scaled up to frontier monitoring settings by collecting the largest deception dataset to date for training probes and introducing a novel probe architecture which can aggregate information across many layers and tokens. Our probes achieve 98.8% AUC in SHADE-Arena, surpassing an Opus 5.5 text-monitoring baseline, and show improved efficacy as the underlying model is scaled up. To push our probes to their limit, we test them on several cases where deception cannot be determined from the context alone. In these cases, which we refer to as introspective deception, the ground truth can only be determined through careful elicitation or thorough knowledge of a model's training data. In one such evaluation, we show that probes can distinguish transcripts containing a model's true hidden goal from other goals with an AUC of up to 99.7%. Our probes also readily detect deception on prominent open-weight models which lie about politically sensitive topics, and about their beliefs when put under pressure. We release our training dataset, dubbed FIBS, to help drive frontier deployment of effective probes, and encourage the community to expand upon it with further examples of deception and sabotage.
The FIBS Dataset: Largest Deception Collection to Date
To investigate the effect of diversity and data scale in lie detector training, the authors created Falsehoods in Broad Settings (FIBS), a contextual deception dataset of 163,016 transcripts spanning 28 families. FIBS combines existing datasets with synthetic constructions and on-policy examples from multiple models. A large fraction of rows, 76%, are synthesized from existing datasets with appended follow-up questions; another 18% come from on-policy deception games, pressure to lie, or persona-driven transcripts; the remaining 6% cover bespoke cases such as misrepresented tool calls or extremely long-context examples. Labels were validated through extensive LLM-as-a-judge filtering, with an estimated label noise rate of 1.4%. The dataset is balanced exactly by label, controls for length and message count, and caps families at 8,000 rows each to ensure even coverage across settings.
A Probe Architecture That Spans Layers and Tokens
The paper introduces several probe architectures built on top of frozen activations from modern open-weight models. The most basic form is the linear probe, which applies logistic regression on the residual stream at a single layer and token. Higher-capacity single-span probes include MLP probes, two-layer perceptrons with hidden width 256. For ensemble-style per-layer averaging, the LinearEnsemble and MLPEnsemble probes train a probe per layer whose training loss stays below a threshold and average their outputs.
The medium-capacity, long-span probe is the EFC (early-fusion covariance) probe. At each token, EFC concatenates activations across layers, projects them to a small dimension using a learnable projection, and aggregates across tokens via the mean and covariance of the projected vectors. The output is a linear projection of those moments. An axial probe applies factored causal attention over the full layer×token grid, attending over both axes simultaneously. All probes map hidden states to a [0,1] output indicating the presence of a knowing falsehood.
SHADE-Arena: 98.8% AUC Beating Opus 5.5
Probes trained on FIBS were evaluated on SHADE-Arena, a benchmark of multi-step on-policy trajectories where an untrusted model acts as an agent in an environment with tool access. In this setting, probes achieved 98.8% AUC, surpassing an Opus 5.5 text-monitoring baseline. The results hold across 14 SHADE-Arena scenarios, with performance improving as the underlying model scale increases. On Qwen3.5-397B-A17B, all four probe types (LinearEnsemble, MLPEnsemble, EFC, and axial) scored above the Opus 5.5 baseline, with self-monitoring included as a comparison point. Figure 1(a) in the paper shows AUROC across scenarios, with 95% episode-bootstrap intervals.
Introspective Deception: When Context Alone Is Not Enough
The authors distinguish two categories of falsehood based on how belief is established. A falsehood is contextual when belief can be determined from the transcript alone—for instance, when a tool result in the context directly contradicts the model's statement. Otherwise, a falsehood is introspective: the ground truth can only be determined through careful elicitation or thorough knowledge of the model's training data. Identical answers can be a lie for one model and honest for another, as illustrated in Figure 1(b) where two model organisms both deny having a hidden goal, but the denial is false for one and true for the other; the axial probe classifies the two transcript sets with 98.0% AUC.
To evaluate introspective deception, the authors used belief elicitation and model organisms with hidden goals. In one evaluation, probes distinguished transcripts containing a model's true hidden goal from other goals with an AUC of up to 99.7%. This result shows that probes trained primarily on contextual falsehoods can generalize to cases where the label depends on additional knowledge about the model rather than visible context alone.
Detecting Deception on Open-Weight Models
Probes trained on FIBS also readily detect deception on prominent open-weight models. The dataset includes cases where models lie about politically sensitive topics, and cases where models misrepresent their beliefs when placed under pressure. In these settings, probes identify knowing falsehoods without requiring access to proprietary system prompts or black-box monitoring infrastructure. The transfer from FIBS-trained probes to open-weight models demonstrates that the dataset's diversity enables generalization beyond the specific models used during construction.
From Dataset to Deployment: FIBS and What Comes Next
The FIBS dataset and the accompanying probe architectures provide a scalable pathway for white-box deception detection in frontier monitoring settings. By collecting the largest deception dataset to date and training expressive probes that aggregate information across many layers and tokens, the authors demonstrate that probes can compete with, and surpass, much more expensive black-box monitors. The improved efficacy observed as the underlying model scales up suggests that the approach will remain effective as future, more capable models are deployed. The authors release FIBS to the community and encourage expansion with further examples of deception and sabotage, hoping the resource will drive broader adoption of effective probes in real-world agent deployment.