Design benchmark reveals gap in AI detection
A new online benchmark called Real Design Benchmark has put AI-generated interfaces up against real product designs, asking participants to tell them apart. The results show that 41 percent of people cannot reliably identify which design comes from an AI generator. The site invites users to compare two designs side-by-side, pick the one that better accomplishes a task, and then guess which was machine-made. The gap between perception and reality raises questions about how quickly AI tools have entered the design workflow and whether design teams are equipped to spot the difference.
How the comparison game works
The benchmark presents two design options side by side. One originates from a live product, the other from an AI generator given only a written brief describing what the feature does. The AI never sees the real screen; it receives only product context—fonts, icons, and colors from the original product—and builds a new version from scratch. After viewing both options, participants first pick which design better accomplishes the described task. Only then do they guess which design is AI-generated. The answer reveals itself after both judgments are recorded. This blind format ensures that preferences guide the identification step rather than prior knowledge.
Methodology behind the AI design challenge
According to the site's methodology page, each task features a real feature extracted from a live web product, captured from an actual user account. The AI receives a written brief that describes what the feature does, not how it looks. It also gets access to the product's existing fonts, icons, and color palette. The real design is withheld entirely. This means the AI must produce a viable alternative using only functional description and style tokens. The comparison then asks human participants to judge which version serves the task better, and whether they can distinguish the AI output from the human-made original.
Preference and identification are measured separately
The benchmark tracks two distinct metrics. Preference records which design participants judge as better for the task, regardless of origin. Identification measures whether participants correctly point to the AI-generated design. These two data points do not always align. It is possible to prefer an AI-generated layout while still recognizing it as machine-made, or to favor the real design while misidentifying its origin. The 41 percent non-detection rate suggests that many participants appreciate AI outputs without realizing their source.
What the results suggest about current AI design capability
The benchmark's founder frames the experiment as a way to gauge how far AI design tools have come. When an AI can produce a screen that 59 percent of people identify as AI-made, and 41 percent miss the signal entirely, it indicates that visual fidelity and layout competence have reached a point where the output is indistinguishable from real product design for a significant majority of viewers. This does not mean AI can replace designers, but it does suggest that the gap between human and machine-generated interfaces has narrowed enough that end users often cannot tell the difference.
Implications for design teams and product teams
For teams that commission custom work or evaluate third-party design agencies, the benchmark highlights a new consideration: if stakeholders cannot reliably spot AI-assisted work, the traditional quality-assurance step of "checking for machine-made output" may no longer function as intended. Teams may need to audit design provenance explicitly, perhaps by tracking tool usage in source files or by establishing internal guidelines about when AI tools are appropriate for which project phases. The benchmark also suggests that AI-generated designs can compete on merit in common UI tasks, which means product teams should evaluate AI outputs on the same criteria they apply to human designs rather than dismissing them solely on origin.
Real Design Benchmark provides a public, repeatable way to test these dynamics. By playing round after round, participants contribute data that maps how often AI design slips past human observation, and under what conditions detection becomes easier or harder. The site's ongoing operation will accumulate a dataset that could help the broader design community understand where AI fits into current workflows and where human oversight still adds unique value.