The most popular AI benchmarks have become an exercise in theatrical number-crunching — but what happens when you test these models on a task that has no strategic workaround? On September 25, 2026, a new benchmark called random-bench was published that strips large language models down to their statistical core, asking a deceptively simple question: can an LLM generate actual randomness?

Why Testing Randomness Matters

The benchmark arrives at a moment when the AI industry is mired in "benchflation" — a term describing the arms race of ever-inflated scores on increasingly contrived tests. While many of these benchmarks serve as marketing collateral, they often correlate with real capabilities, for better or worse. Random-bench takes a different philosophical stance, borrowing an idea from a 2020 astrology article: it doesn't matter if a system is "truly" random; it matters if the output is useful.

The test is deliberately unglamorous. The researcher sends each model a prompt to select a number between 0 and 255 — a single byte — and repeats this 255 times. The resulting bytes are concatenated into a 2048-bit sequence. This process is then repeated 54 times per model, which is the minimum threshold set by the NIST statistical test suite. The final score is a measure of how "random" each model's output actually is.

The Entropy Results

The benchmark uses two primary metrics: Shannon entropy (measured in bits per byte) and the NIST random bit generation test suite. For reference, a cryptographically secure pseudo-random number generator — the kind built into Python's standard library — scores at the theoretical maximum of 8 bits per byte.

The results reveal a counterintuitive pattern. More modern frontier models tend to score lower on the entropy scale, meaning their outputs are measurably more regular than older or less sophisticated models. This tracks with a broader observation: as language models have grown more capable, their outputs have also grown more predictable.

The numbers get more striking when examining response distribution across different models. A single number — 173 — accounts for 60.1% of all responses aggregated across every provider tested. Eight models chose essentially nothing but 173 across the entire test run, and all eight of those are OpenAI models. Four Anthropic models similarly cluster around the same one or two preferred numbers, indicating the problem is not isolated to a single company's training pipeline.

The BERT Problem

To illustrate the spectrum of failures, the benchmark includes non-LLM models like BERT for comparison. The results are inverted. Where frontier models over-correct toward numbers that look random to a human, BERT under-corrects toward numbers that appear in written text. Specifically, 76% of bert-base-uncased responses and 53.4% of ModernBERT-large responses fell between 0 and 10 — the single digits that dominate everyday writing.

The explanation is straightforward. A base BERT model, trained only to predict the next token in text, has no instruction-following layer. When asked to "pick a number," it defaults to the numerical tokens it encounters most frequently in its training corpus: 0, 1, 2, and 3. Frontier models, by contrast, have been shaped by reinforcement learning from human feedback (RLHF), and humans have a peculiar bias toward numbers like 137 and 173, which they perceive as looking "very random." This anthropomorphic imprint gets baked into the model's understanding of what a random number should look like.

Per-Bit Bias and the NIST Reality Check

The benchmark also examines per-bit bias — the average value of each individual bit across all generated numbers. Frontier models exhibit a distinctly human-like pattern of alternating bits in predictable ways. Non-instruct BERT models, meanwhile, produce mostly 0s until the final few bits, consistent with their preference for low numbers. The instruct variants of ModernBERT show reduced bias but have not escaped it entirely.

The final and most damning analysis comes from the NIST test suite, which measures whether an attacker could practically exploit the output of a given generator. Here, every model tested — including the 800MB ModernBERT-large-instruct — scores poorly. The NIST tests detect structural vulnerabilities that Shannon entropy alone might miss, and no LLM passes. As the author notes bluntly, anyone considering using a large language model's weights as a cryptographic random number generator should reconsider.

What This Actually Tells Us

Random-bench is not a practical tool for anyone building production systems. It is, as the author describes it, "very much vibecoded" — built for curiosity and insight rather than engineering utility. But it does serve as a fascinating lens into how language models think, revealing that the very training techniques which make them helpful conversationalists also make them fundamentally bad at producing true randomness. The gap between BERT's literal-mindedness and frontier models' performative randomness maps neatly onto the pipeline of post-training human feedback — and that mapping tells us something about how deeply human biases shape even the most seemingly objective-seeming outputs of these systems.

The source code for the benchmark is publicly available for anyone who wants to reproduce or extend the results.