Innhold Show
Website owners will be able to opt out of AI model training by OpenAI, Google, Microsoft and Anthropic with one shared rule in robots.txt. It is a welcome step away from the growing tangle of company names and crawler-specific rules.
The new policy token is called __usage_ai-training. It lets a publisher say that all or part of a website should not be used to change the learned parameters of generative AI models. Every provider that adopts the system can read the same preference.
This is sensible. A content creator should not have to maintain an ever-expanding list of bot names just to express one simple choice. But call the mechanism what it is: a voluntary preference for model training. It does not block crawling, remove data already used for training or take effect at all four companies immediately.
What is the new AI training opt-out?
The new opt-out is a regular robots.txt group where the User-Agent field contains a shared policy token instead of the name of one crawler. The Internet-Draft gives this example for reserving an entire site from model training while allowing a press kit:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Disallow: / expresses a preference against AI training across the whole website. The longer /press-kit/ rule wins for that directory and allows training there. The mechanism follows the familiar longest-match behavior of robots.txt, so publishers can set a preference by path without introducing another file format.
Until now, a publisher has had to deal with names such as GPTBot, Google-Extended and ClaudeBot, along with separate definitions and controls from each company. OpenAI describes the shared token as a way to send one training preference to several providers. Its existing crawler controls will continue to work alongside the new system.
What does the rule control – and what does it not control?
The token covers AI training in a fairly precise sense: using content to change the learned parameters of a generative AI model. It does not automatically control whether a bot may fetch the page, whether the page can appear in ordinary search results or whether a service may retrieve it as a current source for a generated answer.
That distinction is more than technical hair-splitting. Crawling is the collection of content. Training changes a model’s parameters. Grounding means retrieving current information and using it to support an answer, often with a link back to the source. A publisher may want to remain visible in search and AI answers while refusing permission for model training.
The new group is not a crawler that will visit under the name __usage_ai-training. The draft explicitly treats content-use preferences and traffic controls as independent signals. If you want to block an actual bot, you still need a rule for that crawler’s product token.
An existing wildcard group such as User-Agent: * does not count as a preference under this proposal. If the __usage_ai-training group is absent, no choice has been expressed. Existing robots.txt files will not suddenly be interpreted as consent or refusal simply because the draft exists.
When will the four companies respect the signal?
The four companies support the direction, but their rollouts are not synchronized. Anthropic has already adopted the signal. OpenAI says support will arrive over the next few months. Google is targeting 2027, while Microsoft has set February 1, 2027 as the effective date for its new preferences.
That means you should not delete your current company-specific rules and declare the job finished today. A preference has no practical effect at a provider until that provider can read and honor it. OpenAI and Google also say their existing controls will continue to work.
Google has explained that a shared disallow preference will be treated as a block for Google-Extended when the site has no separate Google-Extended rule. At the same time, Google’s announcement says a specific Google-Extended rule can give Google a different instruction from the broad shared preference.
There is a practical catch. Google-Extended is connected to both training and grounding in some Gemini experiences. A publisher who wants to refuse training but remain eligible to be used and linked as a current source will need to read Google’s final implementation carefully. One shared line makes the basic choice easier, but it does not magically make every company’s products and definitions identical.
Microsoft separates model training from AI answers
Microsoft draws the clearest line between training and grounding. The shared robots.txt rule will control AI training for a website or path. A separate on-page preference will control whether the content can ground generated answers in services including Bing, Copilot and Microsoft 365.
Under Microsoft’s implementation plan, a publisher can set training to Disallow and grounding to Allow. The page may then remain available as a current source for a generated answer without being available for future generative model training. That choice does not by itself exclude the page from ordinary Bing search results.
Microsoft is also ending the use of NoArchive and NoCache as controls for training and grounding. The new preferences will apply when content is crawled from February 1, 2027. After March 1, 2027, the old tags will no longer be interpreted as those preferences.
If your site currently relies on NoArchive or NoCache to limit AI use, you therefore have a real migration task. Training must be addressed in robots.txt or on the page. To preserve a grounding restriction during the transition, Microsoft says the new on-page preference must be published and a temporary setting registered in Bing Webmaster Tools by March 1, 2027. That temporary setting lasts until the page is crawled again or September 30, 2027 at the latest.
Is this a finished standard?
No. The document was published on October 30, 2026 as an individual Internet-Draft by Anthropic’s Kevin Kelley. It is marked as work in progress and expires on May 3, 2027 unless it is updated or replaced. The IETF Datatracker makes clear that an individual draft is not approved by the IETF and has no formal standing in the standards process.
That does not make the idea worthless. Quite the opposite. Early backing from four major companies can make the signal useful well before every formal process is complete. Using familiar robots.txt syntax is also a smart choice because software that does not recognize the policy token can simply ignore the group without breaking existing crawler rules.
Still, the word “standard” needs a brake pedal for now. This is a draft and a voluntary preference, not a technical barrier or enforcement authority. The draft also allows more specific provider rules and private agreements to override the broad preference. Whether companies honor the signal depends on their commitments and any applicable legal obligations, not on robots.txt enforcing anything by itself.
The same limitation applies to historical training data. The preference is tied to the robots.txt file in effect when the content is retrieved. It can guide future use by companies that respect it, but it cannot turn back time and remove a work from a model that has already been trained.
My verdict: Good direction, but keep your current controls
A shared opt-out for AI training is a good idea. The web needs fewer proprietary little kingdoms where every provider invents its own token, definition and documentation. When OpenAI, Google, Microsoft and Anthropic support the same basic signal, the choice becomes much easier for small websites and large publishers alike.
But nobody should sell this as a giant red stop button for “AI scraping.” It is not. The token addresses one specific use of content, while crawling, search indexing, grounding and existing training data remain separate questions. If that difference is blurred, people will believe they have disabled far more than they actually have.
My advice to website owners is simple: keep your existing provider-specific rules while the rollout continues. Document what you want to allow for crawling, training and grounding as three separate decisions. Add the shared token when it matches your choice, but also check the rules for the providers that matter to you.
This is not the final answer. It is, however, concrete, understandable and possible to adopt without rebuilding the web. That is a much better start than four more special rules.
Frequently asked questions
What should I add to robots.txt to opt out of AI training?
Use User-Agent: __usage_ai-training followed by Disallow: / to express the preference for the entire site. The signal is still a draft and providers are adopting it on different schedules, so keep your existing GPTBot, Google-Extended and other relevant crawler rules until support is active and verified.
Does the new policy token stop AI bots from visiting my website?
No. The policy token expresses a preference about using content for generative model training. It is not a crawler block. To restrict collection itself, you still need robots.txt rules for the product token used by the relevant crawler.
Can my pages still appear in search results and AI answers?
Yes, depending on the provider’s implementation. Training, ordinary search indexing and grounding are separate uses. Microsoft offers separate choices for training and grounding. Google applies the shared preference through Google-Extended when no more specific Google-Extended rule is present.
Will the opt-out remove my content from models already trained on it?
No. The draft applies the preference associated with the robots.txt file in effect when content is retrieved. It does not describe a mechanism for deleting historical training data or changing a model that has already learned from the content.
