OpenAI, Google, Microsoft, and Anthropic have agreed to support a shared signal that lets publishers control how their content is used for AI training. The signal, called a policy token, will be published in a website's robots.txt file. Kevin Kelley, a technical lead for standards at Anthropic, proposed the token as part of the Internet Engineering Task Force (IETF). Previously, site owners had to set AI training preferences separately for each crawler. The new approach lets one preference apply across multiple AI companies.
A unified opt-out token for AI crawlers
The policy token lives inside robots.txt, the long-standing standard that tells web crawlers which parts of a site they may access. Historically, robots.txt controlled crawling for search indexing and archiving. AI training preferences were not part of the original spec, so each AI provider required its own directive. This new token consolidates those preferences into a single line that crawlers can read and respect.
Publishers grapple with a growing volume of AI crawlers that scrape content and feed it into models. Some have turned to the Really Simple Licensing (RSL) standard, which lets publishers charge crawlers that visit their sites. Cloudflare blocks AI crawlers by default. The new policy token adds a standardized way for publishers to express their wishes without maintaining separate configs for every company.
How the token works in practice
User-agent: *
Allow: /
Disallow: /private/
AI-Training: opt-out
The example above shows a robots.txt entry that marks all crawlers as opting out of AI training. The AI-Training directive is the policy token. When crawlers from participating companies encounter this token, they are expected to restrict access to the site's content for model training purposes. The token does not affect whether the site appears in search results; it only governs use of the content for training or grounding.
Company positions and rollout timelines
Anthropic has already adopted the signal. Google and Microsoft plan to support it next year. OpenAI says it will implement it in the coming months. Microsoft will offer granular controls that let publishers opt out of AI training or AI grounding, which is the practice of pointing an AI model to a verified source of information to reduce hallucinations. Google previously gave sites controls to manage whether content appears in AI search features or is used for training, without affecting visibility in standard search results. Microsoft notes that websites will still appear in Bing's search results regardless of their preferences.
Meta's communications head, Andy Stone, voiced support for a common signal that conveys preferences to multiple providers at once. He said the industry should work together on standards and their implementation. Stone did not confirm whether Meta will honor the token.
What this means for development and content teams
Teams that manage large publisher sites can now add one line to robots.txt instead of maintaining separate directives for OpenAI, Google, Anthropic, and other crawlers. This reduces operational overhead and the risk of misconfigured settings. For developers building tools that interact with AI crawlers, the token provides a predictable signal to read and respect, potentially simplifying compliance workflows. The gradual rollout means early adopters using Anthropic's model will see the effect sooner, while others will need to update their crawler logic as Google, Microsoft, and OpenAI integrate support.
The token also opens the door for tooling that automates robots.txt generation for AI training preferences, similar to how some platforms already generate sitemap entries. Until broader support is in place, publishers may still need to monitor crawler behavior manually.
A update on October 30th added a comment from Meta, reflecting ongoing industry discussion about the standard.