A firewall event says Googlebot requested a public page. The user agent looks right, and the operator is considering an exception. The label is a useful starting point for finding the request, but it was supplied by the client. Before allowing the traffic, confirm the connecting address from a trusted record and apply the verification method documented for the exact crawler or fetcher.
The outcome should be a clear approval decision for a named requester, purpose and path. The existing AI Visibility access checklist explains why sites separate crawler purposes. The firewall troubleshooting guide shows how to identify the layer that denied a request. This walkthrough focuses on the identity check that has to happen before an exception is justified. AI Visibility site access checklist · Google requester verification guide
Start with the request your infrastructure actually received
Find an authorised log record for the exact host, path and time. Capture the request identifier, claimed agent and source address from the component that received the connection. If you have a CDN or reverse proxy, the origin may see the proxy address. Use your provider’s documented method to find the verified client address. Treat a forwarded header as trustworthy only when your infrastructure identifies the trusted proxy and controls how that header is set. A request that supplies its own value needs a different trust decision.
Check how your firewall product interprets the request origin. AWS documents that its WAF Bot Control rule group uses the web request origin’s address for verification and describes a separate configuration consideration for traffic coming through a proxy or load balancer. This is an example of a provider-specific behaviour. Match the verification plan to the place the traffic was observed and the product that will make the allow decision. AWS WAF Bot Control reference
| Evidence | What it supports | What to record |
|---|---|---|
| User-agent string | Claimed crawler or fetcher | Exact label from the request |
| Trusted connecting address | Source observed at the verification layer | Address, layer and proxy method |
| Reverse and forward DNS | A domain and address relationship under the vendor method | Reverse name, permitted suffix and forward result |
| Published IP feed | Membership in a documented address range | Feed URL, capture time, purpose and matched prefix |
| Managed verified-bot signal | A provider classification for the observed request | Product, classification and policy scope |
A browser request or curl test that substitutes a crawler’s user agent can help diagnose a rule that matches text. Label it as a simulation. Even a response that matches the reported error gives you no independent evidence that a vendor sent the original request. Keep simulated and verified requests in separate rows so the security owner can see which evidence supports an exception.
Check the vendor method for the claimed requester
Google documents several categories of crawlers and fetchers and lists ranges for each. For manual checks, it describes looking up the reverse DNS record, validating the returned hostname and checking that a forward lookup resolves to the original address. Compare the hostname with the permitted domains and patterns for the category in the official documentation. A similar-looking suffix from another domain fails that check. For automated verification, use the category’s current published ranges. A match in a broader provider network does not by itself establish the crawler category. Google requester verification guide
For OpenAI, identify whether the claim concerns search crawling, training crawling or a user-triggered fetch. The crawler documentation describes their distinct purposes and links to address feeds. Use the feed associated with the claimed purpose and current guidance, then examine any overlapping results. An address can fall within more than one published range. Record the network evidence alongside the claimed identity rather than treating the address as a unique purpose label. OpenAI crawler documentation
A provider’s managed bot classification can be useful if it is available on the request and its verification method fits the policy. Cloudflare describes several ways a bot may identify itself, including supported signatures, published IP lists with a stable user agent, and reverse DNS. Check which signal your product actually recorded and which actor it applies to. If the signal verifies an intermediary, check what that means for the traffic it sends on behalf of users. Cloudflare verified bots
Reproduce the checks with public evidence
On 6 November 2026, the reverse lookup for Google’s documented example address, 66.249.66.1, returned crawl-66-249-66-1.googlebot.com. A forward A lookup returned the original address. The Google common-crawler feed retrieved that day also contained a matching prefix, 66.249.66.0/27. This is a reproducible check against a public example. The address did not come from an observed request to AI Visibility. For an IPv6 address, use the matching record type and compare the results with the original address. Google requester verification guide · Google common crawler ranges
dig +short -x 66.249.66.1
dig +short A crawl-66-249-66-1.googlebot.com
For an approved log observation, substitute the connecting address from the trusted record. Confirm the hostname in the reverse answer before running the forward check. Compare addresses as parsed values, particularly if you work with different representations of IPv6. Record the DNS result and check time. A lookup timeout or incomplete answer leaves the identity unresolved under that method.
The public snapshot also retrieved OpenAI’s search and training feeds. The parser found 39 prefixes in the search file and 18 in the training file, with six overlapping prefix pairs. It found no match for the Google example address in either OpenAI file. It also found no match for the control address in any of the three feeds. These are counts for the captured files, which may change. The overlapping pairs are comparisons between prefixes, with no implication about how many requests or individual IPs were observed. OpenAI search crawler ranges · OpenAI training crawler ranges
For a concrete overlap check, the address 4.227.36.1 was selected arithmetically from a shared prefix. Both captured files contained 4.227.36.0/25, and the parser matched the address to both. The test sent no traffic to that address. If a request had come from a shared range, that result alone could support network membership in both sets while leaving the claimed role unresolved. Use the documented identity signal appropriate to your rule, or leave the decision pending if the evidence cannot distinguish the requested purpose.
Save the applicable JSON feed locally and use an address parser rather than matching a text prefix. The following example reports all matching networks in the file. Substitute an address from a permitted record for a production investigation. Keep the raw feed or a verifiable snapshot with the result so a later reader can tell which range list you used.
import ipaddress
import json
from pathlib import Path
address = ipaddress.ip_address("66.249.66.1")
feed = json.loads(Path("common-crawlers.json").read_text(encoding="utf-8"))
for row in feed["prefixes"]:
for field in ("ipv4Prefix", "ipv6Prefix"):
if field in row:
network = ipaddress.ip_network(row[field])
if address in network:
print(network)
Give unresolved requests their own outcome
A request can pass a network check but still leave its role unclear. This is especially important when two feeds share a prefix or the log contains only a broad product label. Record which combinations are supported and which remain ambiguous. If you cannot get the original connecting address from a trusted layer, keep the identity unresolved and ask the infrastructure owner for the appropriate evidence. A valid-looking user-agent string cannot fill that gap.
The distinction also appears in public security education. Google India’s Instagram reel embedded in coverage of its online safety campaign provides context for checking who is behind a plausible claim. A crawler exception needs the same underlying discipline, with the exact procedure coming from the vendor’s technical documentation. Coverage with the official Google India reel · Google requester verification guide
If you automate feed refresh, define what happens when the source is unavailable or malformed. Validate the new data before it replaces a previously approved list. Preserve the last successful capture date and surface a stale-feed condition to the owner. The person on call needs to distinguish a definite nonmatch from a check that failed because the expected list could not be loaded. Record the refresh policy with the firewall rule.
Try the policy against a realistic proxy case before deployment. Suppose the origin records the same load balancer address for every request. The security owner needs a record from the edge, or a documented and authenticated route by which the edge supplies the actual connecting address. Testing the load balancer against a vendor list cannot answer the identity question. When an inspection tool is used, keep its requester category with the test result so a later reader can distinguish a diagnostic fetch from routine search crawling.
Document how the team will record disagreement. A confirmed reverse hostname and a missing result from a chosen range list may point to a category mismatch, a stale feed or a lookup error. Check the vendor’s accepted methods and the category rather than treating the two checks as interchangeable. Ask for the original timestamp and retain the evidence that led to the decision. The rule owner can then revisit the same case when an address feed or product changes.
Approve the smallest change the verification supports
Use a Crawler identity record with the host, path, request time, claimed agent, verified address source, vendor method, matched category, evidence date and policy decision. Give the exception a named owner. If the business intends to allow a verified search crawler to reach public product pages, scope the decision to the relevant path and enforcement layer. A verified identity can support the requested access decision while private paths and unrelated traffic keep their existing protections.
Retest with a legitimate observed request or an appropriate vendor tool where available. Record which requester the tool uses and which conditions it tests. Keep one public control and one restricted control in the test plan. Capture the resulting status and check the expected content if it is available. A successful ordinary render verifies the ordinary client route, while the verified crawler record establishes evidence for that requester.
Keep the approval linked to the exact evidence used at the time. During a later review, confirm the rule still matches the supported identity, expected paths and current vendor documentation.
Close the investigation with the confidence level the evidence supports. If a verified later request reached the intended content, record the request and time. If the only result is a DNS and feed check for an example address, record the verification method as tested and leave production traffic open. This keeps the approval tied to a request you can explain and repeat.
AI Visibility is published by Lunasol, which offers GEO and AEO services. The worked observations use public DNS and official address feeds. About AI Visibility



