New to this? Read it in simple words
- Websites tell robots what they may do in a file called robots.txt.
- A new draft adds one entry that says “do not train AI on my pages”.
- Four big AI companies say they will respect it, Google and Microsoft from 2027.
- It does not stop robots from visiting, and nobody has to follow it.
- robots.txt
- A file on a website that tells automated programs which pages they may use.
- IETF
- The Internet Engineering Task Force, a group that writes many of the internet’s technical standards.
- AI training
- Teaching an AI model with large amounts of text, pictures or other data.
Websites use a file called robots.txt to tell automated visitors what they may do. On October 30, an Anthropic engineer filed a draft at the IETF that adds a shared entry for AI training, and four AI companies said they would follow it.
How the signal works
A website adds a group for the special name “__usage_ai-training” to its robots.txt file, with “Disallow” for the pages it wants kept out of training. The draft says the usual “User-Agent: *” line does not count as a choice about training.
The signal is about training generative AI models. It “is not a directive for crawling behavior”, the draft says, so crawlers may still visit the pages. Private agreements between a site and an AI company can override it.
Kevin Kelley of Anthropic wrote the draft for the IETF’s AI Preferences working group, which has not adopted it yet. It can still change.
From Microsoft’s and Google’s posts, an IETF mailing-list message, and news reports for OpenAI.
Who will honour it, and when
Microsoft will start honouring the signal on February 1, 2027, its Bing blog says. It also adds a page-level setting, “ai-grounding:disallow”, that keeps a page out of AI answers. Microsoft says these choices do not change whether a site can appear in normal Bing results.
Google says it intends to support the signal in early 2027. A site that opts out this way will be treated as if it had blocked Google-Extended, Google’s existing control for AI training, Search Engine Roundtable reports.
OpenAI plans to support it “in the coming months”, according to SiliconANGLE and Engadget. Kelley wrote on an IETF mailing list that Anthropic has started to build support, adding: “It’ll take us some time.”
What it does not do
The signal does not block crawlers, and it binds no one: the draft says AI companies “MAY” use it. Engadget notes that it does not stop scraping.
It also does not cover AI answers that quote or summarise a page. Microsoft handles that with its separate page setting, and Google says sites that allow Google-Extended can appear in answers in its Gemini apps.
Google says the idea fits the EU’s code of practice for general-purpose AI models, which treats robots.txt as a main way for websites to state their choices.
Sources
Every fact in this story comes from the sources below. Open them to check our work.
- 1Primary source · October 30, 2026Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00) IETF
- 2Primary source · October 30, 2026Message from Kevin Kelley (Anthropic) on the training preference signal IETF ai-control mailing list
- 3Primary source · October 30, 2026Your content, your choice: clearer preferences for training and grounding Microsoft Bing
- 4Primary source · October 2026Supporting a shared AI training control for websites Google Search Central
- 5Research · October 30, 2026Google, Anthropic and others to support new AI crawling opt-out standard SiliconANGLE
- 6Research · October 31, 2026Google joins OpenAI and Anthropic in promising new scraping controls Engadget
- 7
Microsoft’s start date is in its own post, and Google’s plans are as its post and Search Engine Roundtable give them. OpenAI’s plan comes from news reports, and none of the four companies honours the signal yet.
Google, Microsoft, OpenAI and Anthropic back one AI opt-out. Silicon AI News, October 31, 2026. https://siliconainews.com/stories/ai-training-opt-out-signal-ietf
Four big AI companies back one shared robots.txt signal to opt out of training, though it binds no one and doesn’t stop crawling. How we rate the mood
The mood shows how the news frames the event for the tech and AI world: as an opportunity (6 and above) or as a loss (below 5). It is not our verdict on who is right.