The way people discover businesses online is shifting under our feet. Search engines are no longer the sole gatekeepers. AI assistants, shopping agents, and research tools now sit between a customer's question and your homepage, deciding whether to mention you at all. Cloudflare is betting that this change is big enough to warrant a new category of web tooling.
The company has launched two products aimed at site owners trying to understand how AI agents interact with their sites. One diagnoses whether agents can technically read and navigate a site. The other measures whether AI assistants actually recommend that site to users. Together they represent a push to redefine what "being found online" means when machines do the browsing.
Less Than Half of Web Requests Come From Humans
Cloudflare's internal data paints a stark picture: fewer than half of all HTML page requests now originate from a human. Not every non-human request comes from an agent acting on someone's behalf, but that share is climbing. Answer engines, shopping assistants, and research tools are becoming the de facto interface through which customers evaluate options.
The traditional SEO playbook, built around ranking on a results page, assumes a human scans links and clicks through. Agents do not work that way. They read structured data, follow protocol cues, pull content through machine-readable formats, and synthesize an answer before the user ever sees a URL. If your site is not legible to that process, you are invisible to a growing slice of your potential audience.
Agent Readiness: The Technical Diagnostic
The first tool, Agent Readiness, functions as a technical audit. It crawls a site the way an agent would: checking robots.txt for crawler rules, parsing XML sitemaps, evaluating response headers, looking for Markdown versions of content, and scanning for published metadata around authentication and tool interfaces.
Results roll up into a readiness tier from "Not Ready" to fully agent-native. Each check returns pass, fail, or neutral, with an explanation and evidence showing the exact request and response Cloudflare observed.
The checks are organized by implementation effort. Quick wins cover the basics most sites skip: a crawler-readable robots.txt, an XML sitemap, explicit AI-crawler rules, and serving clean Markdown to agents. Technical groundwork adds content signals stating how your material may be used, an API catalog, link headers, and agent login instructions. Advanced integration introduces OAuth discovery, MCP (Model Context Protocol) and A2A (Agent-to-Agent) agent cards, a skills index, Web Bot Auth, and WebMCP. A commerce tier covers emerging payment standards like x402, ACP, and Universal Commerce Protocol, though that section is informational for now.
Every failed check includes a remediation path. Where Cloudflare has a native feature that helps, a direct link to that setting appears. For everything else, a button copies a prompt designed for coding agents to build the missing piece. The feedback loop is intentional: make a change, re-scan, and watch the status update.
Answer Engine Optimization: Measuring Recommendations
The second tool, Answer Engine Optimization (AEO), addresses the harder question: when a customer asks an AI assistant about your category, does it recommend you or a competitor?
Unlike search rankings, there is no impression count or click-through report for AI recommendations. When a competitor gets named instead of you, the sale is lost and nothing in your analytics reflects it.
AEO works by first inferring a site's industry and category from its content, then probing leading AI assistants (currently Anthropic's Claude and OpenAI's GPT) with realistic customer prompts. These are not generic queries. They mimic actual discovery scenarios: requests for recommendations, product comparisons, and category-specific advice.
The tool returns several metrics. Citation Rate measures how often your site appears as a source in answers within your category. Prominence tracks how much of the answer content is attributed to you and how early your citation appears. Mention Rate counts how often assistants name your brand in their response, regardless of whether they cite your URL. Comparing mention rate to citation rate reveals a specific gap: your brand is recognized but not yet earning the reference. Share of Voice puts your citation share against competitors, showing who dominates the prompts you are losing.
Pre-Computed Baselines Instead of Live Queries
AI assistants rarely produce identical answers to the same question twice. To handle that variance, Cloudflare runs the evaluation panel once per category and reuses the baseline across all site owners in that domain. This approach has practical advantages: results load instantly from a snapshot, compute costs stay manageable at scale, and the shared corpus lets Cloudflare derive an Industry Fit score measuring whether an assistant groups your site with your actual competitors.
When deeper evaluation is needed, Cloudflare AI Gateway prompts each assistant multiple times across different models. Workers AI then reads the responses natively on Cloudflare's infrastructure, extracting signals about citation placement, attribution, and prominence. The system uses exact text analysis rather than model-graded scoring to avoid circular evaluation.
Operator Activity: Who Crawls Without Sending Traffic
A companion view breaks down crawl and referral traffic by operator: OpenAI, Google, Anthropic, and others. It shows which operators read your content, which ones actually send visitors back, and what errors they encounter along the way (403 blocks, 404 dead links).
The pattern worth acting on is the operator that crawls thousands of pages but refers zero visitors. That operator is consuming your bandwidth and content without returning any audience. Cloudflare frames this as a visibility problem site owners did not previously have the tools to identify.
The First-Mover Advantage Is Still Available
Cloudflare's pitch echoes the early days of SEO: the sites built for search engines were rewarded before everyone else caught up. The company argues the same dynamic exists now for agents. Sites that are easy for agents to find, parse, and trust will get recommended. The rest will not appear in the answer at all.
Both tools are available in the Cloudflare dashboard. Agent Readiness is live; AEO is available through early access requests. For teams building on the open, agent-ready web, Cloudflare is also soliciting feedback through its Developer Discord.