Measuring bias in large language models has often relied on comparing the log-likelihoods of two contrastive stereotype sentences. This approach can produce unreliable results because simply rewriting the same stereotype with an alternative attribute may yield logically inconsistent preferences. To address this shortcoming, the authors propose a dual minimal pair setup that introduces two axes of comparison for more robust stereotype evaluation.
A data-augmentation framework for expanded coverage
The framework begins with a data-augmentation strategy that fills critical gaps in existing stereotype datasets. Paraphrases and alternate attributes are generated to expand coverage. The authors apply this framework on a set of stereotypes across four languages: English, Russian, Spanish, and Chinese. This multilingual scope reveals patterns that single-language evaluations might miss and provides a broader foundation for comparing stereotype strength.
Two evaluation metrics tailored to the dual minimal pair setup
Two evaluation metrics are introduced, each tailored to the dual minimal pair setup. One of these metrics models the mutual information between social groups and stereotyped attributes. This mutual information-based metric is better suited for aggregation and enables more robust comparisons of stereotype strength across different languages and models. By focusing on mutual information, the metric captures the statistical relationship between group membership and attribute association in a way that single-pair log-likelihood comparisons cannot.
Experimental results and code availability
Experimental results demonstrate that the dual minimal pair approach yields more stable and interpretable bias measurements. The mutual information metric, in particular, provides consistent results when evaluating models across the four languages studied. The authors' code is available at the provided URL, allowing researchers to apply or extend the framework.