Safety benchmarks typically report one overall score for a suite of datasets, each of which may target one or more safety-related attributes, so models with similar overall scores can have very different attribute profiles. Comparing models is more tractable at the level of individual attributes, yet it is often unclear whether even a single dataset's scores isolate any single attribute. One plausible candidate for such an attribute is harmful refusal, a model's tendency to refuse dangerous or policy-violating prompts. This paper examines whether harmful refusal constitutes a single, measurable attribute in HELM Safety. Using a construct validity framework that stipulates an attribute must exist before a test can measure it, the analysis starts with HELM Safety's four datasets that might plausibly target harmful refusal, but finds that three are saturated. The remaining dataset, HarmBench, is subjected to two psychometric tests to determine if a single attribute like harmful refusal could stand behind its score. First, multidimensional item response theory modeling strongly suggests that HarmBench does not measure a singular attribute. Second, a differential item functioning analysis finds items where models from different developers with the same refusal ability score differently. These flags largely disappear under scope-specific matching, a pattern consistent with aggregation effects but not sufficient to rule out domain-specific developer differences. Zooming out, HarmBench collapses distinct harm behaviors into one score, and the overall HELM safety aggregate further collapses HarmBench and scores from other datasets into a single top-line number. Any safety score that averages over datasets and items can hide saturation and conflate behaviors this way. The paper argues that a score should earn its single-attribute reading before models are compared with it.
Saturation in Three of Four HELM Safety Datasets
HELM Safety aggregates results from multiple datasets, each designed to probe different dimensions of model safety. When a dataset becomes saturated, it means that additional model variation provides little to no discriminative power—scores cluster tightly around high or low values, leaving little room to distinguish among models that differ in meaningful ways. The paper identifies that three of the four datasets that might plausibly target harmful refusal are saturated. Saturation occurs when the prompt set, though varied in surface form, maps onto a narrow set of underlying refusal behaviors, so that nearly all models either refuse or comply across the board. This saturation masks attribute-level differences that could otherwise inform model selection or improvement.
The saturation of three datasets does not merely reduce their individual usefulness; it has cascading effects on the aggregated HELM safety score. When multiple saturated datasets contribute to a composite score, the resulting top-line number can be dominated by the non-saturated dataset, or by noise from items that do not cohere under a single attribute. The paper's construct validity approach insists that before any score can be interpreted as measuring a single attribute, that attribute must be demonstrably present and measurable within the dataset. Without that foundation, comparisons across models rest on an unstable base.
Construct Validity and the Threshold for Attribute Measurement
Construct validity is a framework for evaluating whether a test measures what it claims to measure. A core stipulation is that the attribute in question must exist prior to and independent of the test instrumentation. In the context of harmful refusal, the attribute is a model's propensity to decline prompts that are dangerous or violate stated policies. The paper applies this framework to HELM Safety's suite, asking whether harmful refusal is a coherent, isolable construct or whether it is a heterogeneous mix of behaviors that different datasets capture in inconsistent ways.
The framework demands that each item or prompt in a dataset reliably probes the target attribute, and that models varying on that attribute produce varying scores. When three of four candidate datasets fail this stipulation—because saturation means items no longer discriminate—the remaining dataset must bear the burden of either supporting or refuting the single-attribute claim. The paper's analysis shows that burden is not met without serious qualification.
Multidimensional Item Response Theory on HarmBench
The first psychometric test applied to HarmBench uses multidimensional item response theory (IRT) modeling. IRT models the probability of a particular response (here, refuse or comply) as a function of the model's ability on specified attributes and the item's characteristics with respect to those attributes. In a unidimensional model, a single "refusal ability" parameter suffices to explain response patterns. The multidimensional extension allows multiple latent attributes to influence responses simultaneously.
The multidimensional IRT results for HarmBench strongly suggest that a single refusal ability parameter does not adequately capture the response data. Instead, item patterns are better explained by multiple latent dimensions—perhaps distinct refusal behaviors corresponding to different prompt categories, threat types, or policy domains. This finding undermines the claim that a single HarmBench score can represent a unified harmful refusal attribute. Models may refuse for reasons that are not captured by a single continuum, and the score's meaning depends on which underlying dimension is engaged.
Differential Item Functioning and Developer Differences
The second psychometric test is a differential item functioning (DIF) analysis. DIF examines whether items function differently for groups of models that have the same underlying level of the target attribute. In this case, the analysis compares models from different developers—such as OpenAI, Anthropic, Google, and others—at equivalent refusal ability scores. The test finds items where two models with the same refusal ability score differently, simply because of the model's origin or training lineage.
Importantly, these DIF flags largely disappear under scope-specific matching. When items are restricted to a narrower prompt scope—or when models are compared within rather than across developer groups—the apparent differential functioning diminishes. This pattern is consistent with aggregation effects: broad, mixed-domain prompts cause apparent differences that vanish when the comparison is confined to a more focused semantic space. However, the disappearance under scope-specific matching does not fully rule out domain-specific developer differences; it merely shifts the interpretation from item-level artifacts to broader domain or developer characteristics that shape model behavior in particular regions of the prompt space.
Aggregation Effects in HarmBench and HELM Safety Totals
Zooming out from the item-level analysis to the dataset- and aggregate-level, the paper observes that HarmBench collapses distinct harm behaviors into one score. The dataset combines prompts spanning coercion, deception, disallowed content, and other harm categories, each of which may tap different model competencies. The resulting score is a composite that may be high because a model refuses coercive prompts but complies with deceptive ones, or vice versa. This intra-dataset heterogeneity means the single HarmBench number obscures such trade-offs.
The overall HELM safety aggregate compounds this problem by merging HarmBench's score with results from other datasets, some of which are saturated and others that measure different attributes. The top-line number becomes a conflation of saturation effects, multidimensional item behavior, and developer-specific patterns. Model comparisons that rely on this single number risk being misled about where models actually differ or resemble each other on the harmful refusal attribute.
Implications for Model Comparison and Benchmark Design
The paper's central argument is that a safety score should earn its single-attribute reading before models are compared with it. In practice, this means that benchmark designers and users must verify that (a) the target attribute is not saturated across items, (b) the items cohere under a unidimensional IRT model, (c) differential item functioning is either absent or fully understood and adjusted for, and (d) the aggregate score does not collapse meaningfully distinct behaviors. Without these checks, a high or low safety score tells the observer less than it might appear to.
For model developers, the implications are clear: relying on a single top-line safety score for marketing or regulatory purposes is risky if the underlying benchmark structure is unstable. For researchers, the paper offers a methodological template—multidimensional IRT and DIF analysis—that can be applied to other safety benchmarks to assess whether their scores hold up under construct validity scrutiny.
The findings also suggest that benchmark improvement efforts should focus on item pool diversification, explicit attribute labeling, and the documentation of saturation thresholds. Datasets that have become saturated should be retired or substantially revised, and new datasets should be stress-tested for multidimensionality before being incorporated into aggregates.
Conclusion
Harmful refusal, as captured by HELM Safety, does not emerge from the analysis as a clean, single measurable attribute. Three of four candidate datasets are saturated, and the remaining dataset, HarmBench, fails multidimensional IRT unidimensionality and exhibits differential item functioning that, while partially explained by aggregation effects, still hints at developer- or domain-specific structure. The aggregation of HarmBench and other datasets into a single HELM safety total further obscures the attribute's true nature. The paper concludes that safety scores must undergo rigorous psychometric vetting before they are deployed as the basis for model comparison, and that the field should move toward more granular, attribute-specific reporting rather than depending on top-line numbers that may conceal more than they reveal.