Google has formally documented a new web-fetching entity called Google-Agent, adding it to its official crawler documentation on March 20, 2026. Unlike Googlebot, which autonomously crawls and indexes the web for Search, Google-Agent is a user-triggered fetcher that navigates the web on behalf of a specific user through AI agents hosted on Google infrastructure, including Project Mariner.
The most significant technical distinction is that Google-Agent does not respect robots.txt directives. Because it acts on behalf of a specific user who has explicitly requested an action, Google classifies its requests as user-initiated rather than autonomous crawling. Blocking Google-Agent in robots.txt has no effect. Googlebot continues to obey robots.txt as it always has.
The two entities use distinct user-agent strings. Google-Agent identifies itself with a Mozilla-compatible string that includes the token “Google-Agent” and a link to its documentation page. Googlebot continues to use its established “Googlebot/2.1” identifier. Each operates from separate IP ranges, with Google-Agent using a dedicated range file called user-triggered-agents.json, distinct from the IP files used by Googlebot and other Google fetchers.
For web developers, the practical implications are immediate. Sites that rely on robots.txt to control automated access will find that Google-Agent bypasses those restrictions entirely when a user asks a Google AI agent to visit a page. The distinction means that access control strategies must now account for a category of traffic that is neither a traditional crawler nor a standard browser session but something in between.
Verification of Google-Agent requests follows the same two-step DNS process used for Googlebot: a reverse DNS lookup on the accessing IP should resolve to a googlebot.com or google.com domain, and a forward DNS lookup on that hostname should return the original IP. Site operators can use this to distinguish legitimate Google-Agent traffic from spoofed requests.
The introduction of Google-Agent reflects the broader shift in how AI systems interact with the web. As AI agents increasingly perform tasks on behalf of users, the traditional crawler-versus-browser distinction is being replaced by a more complex spectrum of automated access patterns that existing web standards were not designed to handle.
