A developer who sells AI visibility services to local businesses discovered that his own website, and 36 demo pages he had been sending to prospects, were invisible to every major AI assistant. The culprit was not a deliberate configuration or a coding error. It was a Cloudflare setting he never touched, applied automatically, that blocked GPTBot, ClaudeBot, and PerplexityBot from reading any page on his domain. The robots.txt file also carried blanket disallows for eight AI crawlers and a Content-Signal header telling them not to train on the content. None of it was his doing.

The incident exposed a gap between what site owners think is happening and what is actually happening. Bot-protection products, including Cloudflare's default settings, are treating AI crawlers as threats to block, on sites whose owners have never opened the relevant configuration panel. The result is that businesses that want to be found by AI assistants may already be invisible, and they have no idea.

The training crawler versus search agent split

The distinction that matters is between training crawlers and search-time agents. GPTBot, ClaudeBot, and PerplexityBot fetch pages in bulk to build training datasets. OAI-SearchBot, ChatGPT-User, Perplexity-User, and Claude-SearchBot fetch a single page at the moment a person asks a question. Blocking the first group does not prevent an AI assistant from recommending your business. The assistant can still read your page when someone asks about dentists in your area. It just cannot use your content to train its next model.

This split is documented by the vendors themselves. It is also the subject of large-scale research, including "Consent in Crisis: The Rapid Decline of the AI Data Commons" from the Data Provenance Initiative, published at NeurIPS 2024. That paper audits robots.txt across the web domains that power major training corpora. But neither the vendor documentation nor the academic research had been applied to local business websites, which are the actual subject of the question "will an AI assistant recommend me."

So the developer built a scanner. The first attempt produced a clean result: 29 local business websites, nothing blocking. That result was wrong. A parser bug meant the scanner could not read User-agent: *, the most common way sites block everything. A second bug meant it missed noindex tags, which WordPress has used since version 5.3 instead of robots.txt disallows to discourage search engines. And the host list was never saved, making the scan unreproducible. The number could not be checked, so it was retired.

What 310 local business sites actually returned

The second scan covered 310 local business websites across Fairfax, Loudoun, and Arlington in Virginia. Host names were pulled from OpenStreetMap and written to disk before anything ran. The scanner hit each site with real user agent strings for every major AI crawler, plus control requests from an ordinary browser, Googlebot, curl, and an unknown user agent. 293 hosts responded.

Fifteen sites block an AI crawler in robots.txt. Six of those serve the same file, byte for byte, across six different dental practices. The file allows Googlebot, bingbot, OAI-SearchBot, ChatGPT-User, Perplexity-User, and Claude-SearchBot. Everything else gets a blanket disallow. Someone drew a deliberate line: stay out of training sets while staying readable when a customer asks a question. Fourteen of the fifteen robots.txt blockers draw exactly that line.

Fifteen out of 293 is 5.1%. Reported as a single number, "5% of sites block AI" sounds meaningful. It describes almost nothing that costs a business a customer, because the sites that block training crawlers are still readable by search-time agents. Treating a robots.txt count as the finding misses the actual problem.

Silent blocking without robots.txt rules

Separately from robots.txt, 18 sites returned a refusal to AI crawlers while serving browsers, Googlebot, curl, and an unknown agent normally. Eleven of those refusals came back as 403 errors. The other seven returned persistent 429 rate-limit responses that never let up, even when re-tested one request at a time, three seconds apart. A browser received a 200 from the same address moments later.

Twelve of the 18 sites have no robots.txt rule about AI at all. On the stricter count, using only 403 responses, five of eleven have no rule. Most of the sites doing the blocking never wrote it down. The two groups split cleanly on response type: every site that wrote an explicit AI rule returns a plain 403. Every one of the seven persistent 429s has no rule. Deliberate refusal announces itself with 403. The silent kind shows up as a rate limit.

The hosts span six different infrastructure setups. Four sit behind Cloudflare, three on nginx, two on LiteSpeed, one Apache, one a CDN identifying only as hcdn, and one that sends no Server header at all. Three of the nginx sites identify as WP Engine. No single vendor's default explains the pattern. What explains it is bot-protection products treating AI crawlers as something to stop, on sites whose owners have never opened the setting.

Checking your own site

The distinction between stated policy and actual behavior is the core problem. Robots.txt surveys measure the easy one. Actual HTTP responses measure the one that matters. A site that says nothing about AI in its robots.txt but returns 403 to every AI crawler is invisible in practice, regardless of what the file says.

To check your own site, fetch your front page as an AI crawler and observe the status code:

curl -s -o /dev/null -w "%{http_code}\n" -A \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot" \
  https://yoursite.com/

A 200 means you are readable. A 403 or 429 means something is turning them away. If your robots.txt does not mention AI crawlers, you did not ask for it. Something at the edge is making the decision for you.

Then read the robots.txt file itself. If the only names in the disallow are GPTBot, ClaudeBot, and PerplexityBot, you have opted out of training data while staying readable by assistants. That may be exactly what you want. If OAI-SearchBot or ChatGPT-User appear in the disallow, that is the one worth reconsidering, because those are the agents that answer questions when someone asks an AI for a recommendation.

The developer's own product had a separate problem. The plugin he ships labeled GPTBot as "ChatGPT," so a business that deliberately opted out of training data was told "ChatGPT: blocked." That reads as "you are invisible" when the truth is "you chose not to be training data and you are still perfectly recommendable." He shipped a fix. The broader point stands: the distinction between training and search matters, and most tools and most site owners are not making it.