A shared AI training control is coming to robots.txt
Google plans to support a single robots.txt token, proposed at the IETF, that tells participating AI providers whether a site's content may be used for training. Here is how it works and what to do now.
On this page
On 30 October 2026, Google announced that it plans to support a shared AI training control in robots.txt. The idea is simple: one group in your robots.txt file states whether you want your content used to train generative AI models, and every AI provider that takes part reads the same line. Today, a site owner who wants to opt out has to list each provider's own token, one by one.
The control comes from a proposal at the Internet Engineering Task Force (IETF), based on work in its AI Preferences (AIPREF) working group. Google says it aims to support it, along with other AI providers, in 2027.
How the control works
The proposal adds no new directives and no new syntax to robots.txt. Instead, it defines a policy token, __usage_ai-training, that you write in a User-Agent line like any crawler's name. The rules under it then express your preference rather than a crawling instruction:
User-Agent: __usage_ai-trainingDisallow: /Allow: /press-kit/
In the draft's own example, this states a preference against AI training for every path on the site, and a preference in favour of it for everything under /press-kit/. Paths work exactly as they already do in robots.txt, so you can opt out of one folder, a whole site, or everything except one section.
A few details in the draft matter for anyone editing the file:
- The wildcard group does not count.
User-Agent: *must not be read as matching the policy token. If your file has no__usage_ai-traininggroup, no preference has been expressed. - It is separate from crawling. The token is not a crawling instruction. Search engines and other crawlers keep following the rules for their own names, exactly as before.
- Specific rules win. The draft says specific expressions should take precedence over general ones, and providers may treat their own named tokens as more specific than the shared one.
"AI training" here follows the definition the AIPREF working group has agreed: using content to change the learned parameters of a generative AI model. Training models used only for tasks like classification, ranking or scoring is not included.
What Google says it will do
Google already offers its own control, Google-Extended, which it has supported since 2023. Google's crawler documentation says it lets publishers manage whether content Google crawls may be used to train future Gemini models, and that it does not affect a site's inclusion or ranking in Google Search.
Under the plan, if your robots.txt uses the shared token to opt out of training, and you have not written a Google-Extended group yourself, Google will read it as if you had disallowed Google-Extended. Existing company-specific controls keep working alongside the shared one, so you can set a broad choice with the shared token and still give individual providers their own instructions.
Google also notes that sites which specifically allow Google-Extended are eligible to appear in responses and receive clicks through grounding in Gemini Apps. It describes the proposal as consistent with the EU Code of Practice for general-purpose AI model providers, which recognises robots.txt as a main way for publishers to state machine-readable preferences.
Where it stands
This is a proposal, not a finished standard. The IETF document is an individual Internet-Draft, published on 30 October 2026, and as the IETF itself notes, it has no formal standing in the standards process yet. Discussion continues in the AIPREF working group. Google's timeline is support in 2027, and it says it will share updates on progress.
What it means for your site
- Nothing breaks today. Parsers that do not know the token ignore its group, as they ignore any unknown crawler name. Adding it will not change how search engines crawl your site.
- Decide your position now. Agree with your clients whether their content should be available for AI training, and whether any sections, such as press material, should be treated differently.
- Keep your current rules. If you already block or allow specific AI providers by name, leave those groups in place. They continue to work, and the draft treats them as more specific.
- Don't use the wildcard for this. A
User-Agent: *rule does not express an AI training preference under the proposal. Use the named token. - Check what search needs.
Google-Extendeddoes not affect Google Search ranking, but allowing it is what makes a site eligible for grounding links in Gemini Apps. Weigh that before opting out of Google specifically. - Watch the draft. Wording and details may change before providers adopt it. Recheck the IETF page before you roll it out across client sites.
