A site will be able to state its preference about AI training once, in one robots.txt group, and have Google read it the same way as Microsoft and OpenAI. That is what the three companies said on 30 October 2026, and none of them supports the token yet. Google says it aims to support it in 2027.
What the policy token is
robots.txt is the plain text file at the root of a domain where a site tells crawlers which paths they may fetch. A policy token is a line in that file that is addressed to a purpose instead of to a named crawler, so the same group can carry a preference that several companies read.
The token is __usage_ai-training. It comes from an Internet-Draft by Kevin Kelley of Anthropic, published on 30 October 2026, and the draft is careful about what it is not. It adds no new directive names and no new grammar. The path language, the longest-match rule and the group structure of robots.txt stay exactly as they were.
A group written with the token looks like any other group, with the purpose standing where a crawler name usually sits.
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Read literally, that file expresses a preference against AI training on every path except the ones under the press kit, where it expresses a preference in favour. The double underscore at the front is there so that a loosely written parser is unlikely to match the token by accident.
What Google said it will do
Google’s post on 30 October 2026 describes the mechanism in one sentence. If a site opts out of training through the shared token and has not mentioned Google-Extended, Google plans to treat that as if the publisher had disallowed Google-Extended. Google-Extended is the company’s own control, in force since 2023, for whether its AI products may use a site’s content.
The same post keeps a door open that matters more than the token itself. Sites that specifically allow Google-Extended stay eligible to appear in responses and receive clicks through grounding in Gemini Apps. So a site can refuse training and still be quoted in an AI answer, which is the opposite of what most people assume a training opt-out does.
The full wording sits in Google’s announcement on the Search Central blog, and the mechanism itself is in the Internet-Draft. Both are short. The draft is the one worth reading if you have to explain this to a client.
Why a wildcard group does not count
This is the part that will catch people. A group addressed to a star, the one most sites already have at the top of the file, must not be read as matching the policy token. The draft says it plainly, so an existing catch-all block expresses no AI training preference at all.
Two more rules in the draft decide real cases. A consumer must match the token exactly apart from case, and a group may name several tokens across consecutive User-Agent lines when a rule serves more than one purpose. Specific expressions should win over general ones, so a site that names a company’s own control keeps that instruction.
The token is not a crawl block
A training preference and a crawl instruction are separate signals in the same file. The draft states that the policy token is not a directive for crawling behaviour, though a consumer may use a preference to inform what it fetches. If you want a page left alone, that is still a rule aimed at a crawler’s own name.
Getting this backwards is expensive in both directions. A site that adds the token expecting its pages to vanish from AI answers will be disappointed, and a site that blocks crawling to stop training also loses the citation it wanted.
What Google, Microsoft and OpenAI each plan
The three announcements were made within a day of each other and they do not mean the same thing. Search Engine Journal summarised them on 7 November 2026, and the differences are the useful part.
| Provider | What the shared token will mean | Caveat to plan around |
|---|---|---|
| Opting out through the token is read as disallowing Google-Extended when that is not specified | Support aimed at 2027, and allowing Google-Extended keeps a site eligible for Gemini Apps grounding | |
| Microsoft | The shared control will be used for training purposes | A separate page-level grounding preference is coming, and NoArchive and NoCache stop counting as training preferences after 1 March 2027 |
| OpenAI | Plans to honour the shared control | Published as a plan, with no date attached in the summary |
The Microsoft row is the one with a deadline. A site that has been relying on NoArchive or NoCache to keep its content out of training has until 1 March 2027 to move to the new preference, and after that the old instruction stops doing that job.
What to write in robots.txt before 2027
Nothing in the file needs to change today, because no provider reads the token yet and a file that adds it now expresses a preference to nobody. The useful work is deciding what the preference should be, then writing it once the support lands.
- Decide what you are protecting. Training and citation are different uses. A blog that wants to be quoted in AI answers has little reason to disallow grounding, and a paid content business has every reason to refuse training.
- Keep the company controls you already have. Google-Extended and the equivalents stay in force alongside the shared token, so a specific instruction is not replaced by a general one.
- Fix the wildcard illusion. If the only group in the file is addressed to a star, no AI training preference has been expressed, whatever the comment above it claims.
- Write it down for the person who inherits the file. A robots.txt nobody can explain is how a site ends up blocking a crawler it wanted, which is a failure I have seen more than once in an audit.
The AI search and getting cited hub collects the posts that track this side of search, and two of them go further into the mechanics. The llms.txt experiment in 97% of llms.txt files get no requests is what a preference signal looks like when nothing reads it. The Cloudflare pilot in what changes for ChatGPT search crawlers asks the same question about access rather than use.
One more thing worth keeping in view. A crawler rule governs a crawler, and the Brave deal covered in Brave selling 30 billion pages to AI companies shows an index being licensed rather than crawled. If your content reached a model through a licensed index, a robots.txt edit in 2027 will not undo it. The token is a lever, and it is one lever, which is why why blog posts never show up in AI answers has more to do with the page than with the file.
