IETF draft · robots.txt

One robots.txt line for AI training

Anthropic, Microsoft, Google and OpenAI have said they will read a single shared group in robots.txt as your preference about training AI models on your pages. Here is what it does, what it does not, and how to add it without hurting your search or AI visibility.

The line

The draft adds no new directive. It reserves a name, __usage_ai-training, that you use in the User-agent line of an ordinary group, with ordinary Allow and Disallow rules. This example states a preference against training on the whole site except a press kit:

User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Spell it exactly: two underscores, then usage_ai-training. Providers are told not to rely on loose matching, so a near-miss reads as an unknown crawler and states nothing.

What it does not do

It does not block crawling. It is about how content is used. Crawlers that honour it can still fetch every page. Blocking a crawler still takes a group for that crawler by name.
User-agent: * does not count. A wildcard group never matches the token. A file without a __usage_ai-training group states no training preference.
It does not reach backwards. It applies to content fetched while it is published. Earlier crawls are untouched.
It is a request, not a lock. Compliance is voluntary, and the draft lets private licensing agreements override it.
It is not a standard yet. It is an individual draft the IETF has not adopted, and it lapses on 3 May 2027 unless revised.

Who says they will honour it

Status as of 12 November 2026. These dates are announcements, not guarantees.

Provider	Position	When
Anthropic	Has started implementing support; the draft's author works there.	No date given
Microsoft (Bing)	Will honour it while crawling; also adds page-level training and grounding controls.	From 1 February 2027
Google	Plans to honour it; reads an opt-out as a Google-Extended opt-out when that group is absent.	Early 2027
OpenAI	Will honour it for foundation-model training; existing GPTBot rules keep working.	In the coming months, no date
Meta	Called it an important step but has not committed to honouring it.	Not committed

The Google catch

Google plans to treat a shared opt-out as if you had disallowed Google-Extended, when your file has no Google-Extended group of its own. Google-Extended also decides whether Gemini Apps may use your pages to ground answers, so opting out of training this way can cost you Gemini citations too. If those matter more, add an explicit Google-Extended group: Google follows it instead of the shared token.

How it fits with the rest of your robots.txt

Keep your per-vendor groups. GPTBot, ClaudeBot, Google-Extended and the rest keep working. Where a vendor's own group disagrees with the shared token, the vendor's group wins.
Search and retrieval agents are separate. OAI-SearchBot, Claude-SearchBot, ChatGPT-User and similar agents fetch pages to answer questions and cite sources. The training token does not change how they treat you.
llms.txt is unaffected. Inviting agents to read your site and opting out of training are compatible: one is about answering questions, the other about building models.

Should you add it?

If you already refuse training crawlers one by one, yes: the shared group reaches participating providers you have not listed, in one place. If you want maximum reach in AI answers, leaving it out is a valid choice: no group means no preference, which is how things worked before the draft. Either way, decide deliberately, and check the Google catch above.

How Lab451 handles it

Generator: an opt-in training preference for robots.txt. No preference is the default; choose to opt out of the whole site or all but some paths. When you opt out, the per-crawler training groups (GPTBot, ClaudeBot, Google-Extended and the rest) are written to match, because a crawler's own group would otherwise override the shared one. Search and retrieval agents are not touched.
Checker: reports your current training preference, a misspelled token, per-vendor rules that contradict it, and the Google-Extended coupling.
Monitoring: tells you when your training preference changes, because the preference that counts is the one published when a page was fetched.

Check your robots.txt Generate one with the training option

Sources

AI training FAQ

Questions about the shared AI training token

Does __usage_ai-training stop AI crawlers from visiting my site?

No. It states how fetched content may be used, not whether it may be fetched. Crawlers that honour it can still read every page. To refuse a crawler, you still need a group for that crawler by name (for example User-agent: GPTBot with Disallow: /).

Does User-agent: * cover AI training?

No. The draft says a wildcard group never matches the token. A robots.txt with no __usage_ai-training group states no training preference at all, whatever the * group says.

Should I remove my GPTBot, ClaudeBot or Google-Extended rules?

No. Per-vendor groups keep working, and when a vendor's own group disagrees with the shared token, the vendor's group wins. Keep them for crawl control; add the shared group for providers you have not listed.

Will opting out remove my site from Gemini answers?

Possibly. Google plans to treat a shared opt-out as a Google-Extended opt-out when your file has no Google-Extended group, and Google-Extended also governs grounding in Gemini Apps. If citations there matter more to you, add an explicit Google-Extended group; Google then follows that group instead.

Does it remove my content from models already trained?

No. It only applies to content fetched while the line is in your robots.txt. Nothing in it reaches back to earlier crawls, and the draft lets private licensing agreements override it.

Is this a standard?

Not yet. It is an individual Internet-Draft, draft-kelley-aipref-training-preference-signal, which the IETF has not adopted. It lapses on 3 May 2027 unless revised. The token name could change in a revision; Lab451 will update its generator and checker if it does.