Google joins OpenAI and Anthropic in promising new scraping controls

Google, OpenAI, Anthropic and Microsoft pledge to support a new common signal for robots.txt that blocks scraping for AI training. Google joins OpenAI and Anthropic in promising new scraping controls. News AI Google joins OpenAI and Anthropic in promising new scraping controls The new robots.txt directive will prevent the companies from scraping content for generative AI training.

The companies have decided to adopt a proposal under consideration at IETF, the open standards organization for the internet, written by Anthropic technical lead for standards Kevin Kelley. As OpenAI and Google explained in their announcements, website owners currently have to identify individual crawlers and add separate rules for each provider. Kelley’s proposal will change that by getting multiple AI companies to recognize and honor a single common signal.

Publishers have to set up a new user agent in the robots.txt file that’s specifically for AI training, which will look like this: User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/ That will block crawlers specifically meant to gather data for AI training. The "Allow" section gives website operators the option to allow scraping for AI training for specific parts of the page, such as if there’s a press kit on it. But this new user-agent section doesn’t prevent crawling altogether or prevent content from showing up on Google search.

Back in June, Strava launched the Model Context Protocol (MCP) connector, letting subscribers query their Strava data through Claude for in-depth training insights. Building on that launch, Strava announced a Google Gemini integration that lets subscribers get more from their training.

Strava subscribers who connect directly to Google Gemini will now get more contextual and personalized answers based on their activity history. Once connected, Gemini can analyze Strava users’ data and tell them whether they are getting faster at the same heart rate, how strength training is affecting…

Strava subscribers can now connect directly to Google Gemini for personalized workout insights. Strava subscribers can now connect directly to Google Gemini for personalized workout insights – com news ADVERTISEMENT Strava subscribers can now connect directly to Google Gemini for personalized workout insights Post your comment Sagar 30 October 2026 Strava Google AI Back in June, Strava launched the Model Context Protocol (MCP) connector, letting subscribers query their Strava data through Claude for in-depth training insights.

Gemini can also compare Strava users’ fastest 10K this year against their all-time best. It can also tell them how long their longest running streak on Strava is and if weather affected their performance.

Strava says, "Connecting to Gemini is entirely optional and read-only; users choose to connect, and can disconnect at any time." Gemini integration is available to Strava users across all Gemini plans, but only in the US. To connect Strava to Gemini, open Google Gemini, then navigate to Personal Intelligence > select Strava under connected apps > sign in with your Strava account > authorize Gemini to connect to Strava.

For instance, Google will specifically interpret it as a stop sign specifically for the "Google-Extended" user agent, which crawls websites for materials used to train Gemini models.