Four leading artificial intelligence firms—OpenAI, Google, Microsoft, and Anthropic—have agreed to adopt an international standard that allows publishers to decline AI training data collection through a unified signal. Kevin Kelley, Anthropic standards technical lead, submitted the policy token specification to the Internet Engineering Task Force on October 30, 2026. Rather than requiring webmasters to track and manually restrict distinct crawler user agents such as GPTBot, ClaudeBot, and Google-Extended, the new robots.txt directive lets site administrators declare an opt-out across all participating systems in one place using a single policy token named __usage_ai-training. The move represents a coordinated industry effort to resolve widespread friction with content publishers over automated scraping without explicit authorization.

This technical consensus emerges amid rising legal risks and tighter regulatory scrutiny across major global jurisdictions. The European Union AI Act mandates copyright compliance and transparent data handling for general-purpose AI developers, while the NIST AI Risk Management Framework highlights respect for content owners as a core element of trustworthy systems. Under legacy robots.txt practices, preventing data extraction often forced administrators to block web crawlers entirely, causing websites to vanish from search engine indices and lose incoming traffic. The newly proposed structure separates ordinary indexing from generative AI model training. Under this model, Microsoft plans to provide granular controls enabling webmasters to preserve Bing search indexing while independently managing permissions for AI training and answer grounding.

Adoption will proceed across varying enterprise timelines before universal technical coverage is achieved. OpenAI plans to integrate the standard into its systems within the coming months, while Microsoft has announced formal crawling support beginning February 1, 2027, and Google is targeting implementation in early 2027. For newsrooms, academic publishers, and independent site operators, the protocol lowers the ongoing administrative overhead of updating disallow lists whenever a new AI crawler enters the market. By providing an explicit mechanism to halt training scraping without forfeiting search visibility, the framework sets a foundation for future data licensing discussions and standardized web governance.