Analyze traffic to see if Google ever sends Google-Extended

[davidmytton]

Google's AI crawler is Google-Extended

Google-Extended is a standalone product token that web publishers can use to manage whether their sites help improve Gemini Apps and Vertex AI generative APIs, including future generations of models that power those products. Google-Extended does not impact a site's inclusion or ranking in Google Search.

User agent token Google-Extended

See the full list at https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers

[blaine-arcjet]

It looks like ai.robots.txt includes this so this should also be resolved by #3

[blaine-arcjet]

The google docs state:

Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.

This means that it only works via detection in robots.txt similar to Applebot-Extended.

@davidmytton Does this mean we should categorize all of the google crawler products as ai?

[davidmytton]

@davidmytton Does this mean we should categorize all of the google crawler products as ai?

Started a discussion: #16 (comment)