Google added a guide on November 17 that explains how to let Google-Agent through a CDN, firewall or bot-detection service. Google-Agent is the name used by agents running on Google infrastructure that browse sites and take actions when a user asks them to. The crawling documentation changelog lists the page as new today.
Three checks identify a real Google-Agent request
The User-Agent header contains Google-Agent, and the source IP falls inside the ranges in user-triggered-agents.json, a file Google says can change, so IP rules need regular syncing. The third check is experimental: some requests are signed with Web Bot Auth under the identity https://agent.bot.goog, carrying Signature, Signature-Input and Signature-Agent headers that can be verified against the keys Google publishes.
Robots.txt is not the lever here. Google's user-triggered fetchers page says these fetchers "generally ignore robots.txt rules", and the new guide does not say otherwise for Google-Agent.
Allow rules fail on challenges and shared IPs
Most of the guide is troubleshooting. An allow rule has to run before broad cloud or data-center ASN blocks, managed bot rules and AI-agent category blocks. It has to skip CAPTCHAs and JavaScript challenges, not only hard blocks, including in-page CAPTCHAs on search and forms. It has to cover the whole journey – search, filters, product pages, cart, booking and the XHR, GraphQL or api.* calls behind them – not just the homepage. Per-IP rate limits need an exemption for verified ranges, because many users share the same egress pool. Load balancers and proxies must pass on the original client IP and the signature headers.
A site that wants agent bookings or purchases can test this by checking its WAF logs for requests with Google-Agent in the user agent and seeing which rule answered them.
Source: Google Crawling documentation, November 17, 2026