When an LLM inside an agent workflow produces a recommendation or judgment, it often comes with an explanation naming the factors that drove the decision. Operators use these explanations to monitor systems, diagnose errors, or decide when to escalate. The assumption is that the named factors actually reflect how the model reached its output. This paper tests that assumption and finds it unreliable in practice.
What "Explaining a Decision" Actually Means in Agent Workflows
Consider an LLM that recommends financial advisors to clients. It returns a recommendation and cites the top three client features that most influenced it: financial goals, investible assets, age. An operator reviewing this explanation assumes those features are the primary drivers. If the recommendation looks wrong, the operator might adjust one of the cited features or escalate based on what was named. But what if the model's actual decision depends more strongly on a feature it never mentioned, like communication preference? What if changing financial goals wouldn't change the recommendation at all?
The paper introduces two precise interpretations of what a cited feature means. Necessity means changing the feature would change the output. Sufficiency means retaining the feature while removing other changeable information would preserve the output. These are distinct properties. A feature can be necessary without being sufficient (changing it flips the decision, but keeping it alone isn't enough to maintain it). A feature can be sufficient without being necessary (keeping it preserves the decision, but changing it doesn't necessarily change the outcome because other features compensate).
Controlled Interventions on Black-Box Models
The evaluation uses two synthetic use cases. Advisor recommendation presents 100 client profiles with 18 features (financial goals, investible assets, net worth, income brackets, demographics, communication preferences, planning priorities) against a fixed pool of 13 advisors. The model selects the best match and cites the top three client features. Prompt-risk monitoring presents 100 prompts that combine ordinary requests with risk segments from eight categories (PII, illegal advice requests, jailbreak attempts, harmful requests, etc.). The model assigns a risk score from 1 to 5 and cites the top three risk features.
For necessity, the method constructs modified inputs where one feature is changed while everything else stays the same. In advisor recommendation, this means contrast substitution: replacing a profile field with the most different valid alternative. In prompt monitoring, it means blank removal: deleting the risk segment associated with that feature. Each modified input is queried three times, and the necessity score is the fraction of trials where the output changes.
For sufficiency, the method retains one feature at its original value while removing or masking all other changeable features. In advisor recommendation, the other 17 client features are replaced with [UNKNOWN] while keeping the advisor pool. In prompt monitoring, only the target risk segment and the base request are retained. The sufficiency score is the fraction of trials where the original output is preserved.
Eight models are evaluated: Claude Opus 4.6, Sonnet 5, Haiku 4.5; GPT-5.4 with reasoning disabled and at Low, Medium, and High effort; and Gemini 3.5 Flash with Medium Thinking. All queries use temperature zero with identical inputs and prompt structure.
The Cited Ranking Is a Weak Signal
The mean Spearman correlation between the cited ranking and necessity scores is 0.349 for advisor recommendation and 0.431 for prompt monitoring. For sufficiency, it is 0.354 and 0.580 respectively. These are positive correlations, meaning earlier-cited features tend to score higher on average. But the correlation is far from perfect, and it varies wildly across models. Individual model-level correlations span from 0.014 to 0.755 across the two criteria.
The same explanation can agree with one criterion much more than the other. Sonnet 5 has similar necessity correlations in both use cases (0.571 and 0.573) but its sufficiency correlations differ substantially (0.273 for advisor recommendation, 0.698 for prompt monitoring). Across models, necessity and sufficiency rankings of cited features correlate between 0.18 and 0.35 in advisor recommendation and between 0.38 and 0.87 in prompt monitoring. A factor that makes a decision sensitive to change is not always the factor that best preserves the decision when retained.
In prompt monitoring, there is a salience gap. The most frequently cited risk category is jailbreak_attempt, but harmful_request and requests_illegal_advice score higher under both necessity and sufficiency. The explanations emphasize a holistic or more critical risk category rather than the categories with the strongest measured influence. Citation frequency alone gives reviewers a misleading picture of which risks drive the score.
Uncited Features Often Outscore the Weakest Cited Feature
This is the most consequential finding for practitioners. In advisor recommendation, an uncited feature scores strictly above the lowest-scoring cited feature in 57.6% of responses under necessity and 58.1% under sufficiency. In prompt monitoring, the rates are 25.8% and 8.9%. The cited top three do not reliably identify the three features with the strongest measured influence.
The highest-scoring uncited feature in advisor recommendation has a mean necessity score of 0.698, which is close to cited rank 1 at 0.731 and exceeds cited ranks 2 and 3 at 0.583 and 0.482. Its mean sufficiency score of 0.492 exceeds all three cited positions. The first citation often carries substantial information, but the full cited set can miss features that are equally or more influential.
At the set level, the picture is more nuanced. The cited-set mean necessity score exceeds the uncited-set mean by only 0.023 at both k=2 and k=3. The cited and uncited sets have similar average scores. Individually strong features in advisor recommendation are commonly uncited, but the cited set as a whole is not dramatically weaker than the best possible uncited set.
Reasoning Effort Helps Feature Selection, Not Ordering
GPT-5.4 shows an interesting pattern. Higher reasoning effort reduces the rate at which a higher-sufficiency feature is omitted: from 76% without reasoning to 40% at High effort in advisor recommendation, and from 25% to 3% in prompt monitoring. But the correlation between the cited ranking and intervention scores does not consistently improve. It rises from 0.194 to 0.452 at Medium reasoning then falls to 0.353 at High. Better feature selection and better feature ordering are distinct properties.
The Claude models show the same distinction. In prompt monitoring, Sonnet 5 has stronger ranking agreement than Haiku 4.5 (0.598 versus 0.069) and fewer omissions (14% versus 31%). But in advisor recommendation, no model dominates on both. Opus 4.6 has fewer omissions than Sonnet 5, yet Sonnet 5 has stronger ranking agreement. These are separate reliability concerns, and improving one does not guarantee improvement in the other.
Full Rankings Show Instability Beyond the Top Three
An exploratory pilot asked GPT-5.4 to rank all available features for 40 samples in each use case. In advisor recommendation, scores generally decrease with rank but are not strictly monotonic: rank 4 exceeds rank 3 in 7.7% of responses. In prompt monitoring, mean necessity actually increases from 0.202 at rank 3 to 0.667 at rank 4, and mean sufficiency jumps from 0.030 to 0.389. The aggregate trend of concentration near the top coexists with individual rank inversions below the boundary.
What This Means for Agent Oversight
The practical implication is direct. If you are using LLM explanations to monitor agent behavior, debug errors, or decide when to escalate, treat the cited factors as testable claims rather than verified accounts. The explanation does not reliably identify the factors with the strongest measured influence on the decision. An operator who acts only on cited features may miss the actual drivers of a wrong output.
The framework proposed here provides a black-box reliability check. For any LLM decision point in an agent workflow, you can run controlled interventions (changing one feature, retaining one feature) to estimate necessity and sufficiency scores for each input feature. Compare these scores against what the model cites. Where they disagree, the explanation is misleading about the model's actual behavior.
This does not mean LLM explanations are useless. The cited features contain useful information, and the first citation often carries substantial signal. But the full cited set should not be trusted as a complete or accurate account of what drove the decision. In high-stakes agent workflows, the gap between explained and actual influence is large enough to matter.
The study is limited to synthetic inputs, temperature-zero decoding, and two specific use cases. Other domains, prompt formats, sampling settings, and model versions may produce different patterns. The evaluation also does not assess whether the underlying decisions are accurate or fair, only whether the explanations agree with observable decision behavior.