Technology and Scienceupdated 1h ago3 min read
Google, Microsoft, OpenAI Plan To Honor Shared robots.txt AI Training Opt-Out
“A shared opt-out will not fix the traffic problem.”
“Microsoft says it will offer granular controls that will give publishers the ability to opt out of AI training or AI grounding”
This article is an automated aggregation of coverage from news outlets. Our editorials offer in-depth analysis curated by the NewsCord team.
Major AI firms Google, Microsoft, OpenAI, and Anthropic back a single robots.txt opt-out line. The shared policy allows publishers to block AI training with one rule across providers.
Save coverage image
A shared robots.txt token
Google, Microsoft, and OpenAI announced plans to honor a shared robots.txt line for AI training preferences, and the line comes from an Internet Draft that Anthropic’s Kevin Kelley submitted to the AI Preferences working group at the IETF. Kelley’s draft uses an ordinary robots.txt rule group where a site writes a policy token named __usage_ai-training under a User-Agent line, with the example showing User-Agent: __usage_ai-training Disallow: / and Allow: /press-kit/. The draft defines AI Training as using content to modify a generative AI model’s learned parameters, and it says models that do only non-generative tasks like classification or ranking are exempted from that definition.
The draft also states that a wildcard group such as User-Agent: * can never match the token, and it says a file without a __usage_ai-training group indicates no preference, which Kelley states is true of every robots.txt file currently published. The draft further says the token does not control crawling, and it frames the mechanism as a preference signal rather than a directive that changes what a crawler fetches.
Google and Microsoft split roles
Google’s post, by John Mueller, says that when a site opts out of training with the shared line and does not specify Google-Extended, Google plans to interpret that “as if the publisher had disallowed Google-Extended.” Google’s crawler documentation says Google aims to support the control in 2027 and that the documentation also says Google-Extended controls whether content can be used to train future Gemini models and whether it can be used for grounding in Gemini Apps and on Vertex AI. Microsoft’s post, by Bing principal product manager Krishna Madhavan, adopts the shared line for training and adds a separate AI Grounding preference that is set only at the page level.
Microsoft said it will begin honoring the new options on February 1, 2027, and it will stop recognizing the older tags as training or grounding options after March 1, 2027. Microsoft also said either restriction will limit visibility of your brand in answers using Artificial Intelligence (AI) and will have no effect on your ability to be listed in search results from Bing.
Implementation timelines and limits
PPC Land reported that Anthropic’s engineer Kevin Kelley told an IETF mailing list on October 30, 2026 that his company had already started building support for the draft, while Microsoft starts on February 1, 2027 and Google plans to start in early 2027. OpenAI said it aims to implement the signal in the coming months, and the draft itself is described as an individual Internet-Draft that does not represent official IETF standardization activity. The Search Engine Journal account says that based on provided timelines, none of the three companies currently supports the line, and it says Google and Microsoft described how each would handle it differently.
The Next Web reported that Microsoft will honor the new preferences from 1 February 2027 and that they will replace the NoArchive and NoCache tags it has used since 2023. The Verge reported that Meta communications head Andy Stone voiced support for a common signal and said, “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” while not saying whether Meta will honor it.