A publisher wants its service pages available to search crawlers while limiting crawling for model training. The current robots.txt has one group for every crawler. Someone adds a named search group with a broad allowance. The service pages are still open under the proposed rules, but the named group now omits an exclusion that used to apply. This is the sort of change that passes a quick reading and fails a review against actual paths.
Write down the crawler purpose and the paths you intend to expose before changing the file. Then test how the file resolves each crawler and path pair. The existing AI Visibility access checklist covers reach, permission and readability across a website. This walkthrough takes the next step for a robots.txt change. It uses the live AI Visibility file as a baseline and runs a separate draft through a parser before discussing deployment. AI Visibility site access checklist
Define a policy by purpose and path
Start with the part of the site you want to make available. A public service page, an unfinished preview and an administrative interface serve different audiences. List which areas should be crawlable for search and which areas should have restricted collection for other documented purposes. Keep access to private resources under authentication and server-side authorisation. The Robots Exclusion Protocol specifies rules for participating crawlers and explains that publishing paths in robots.txt can make them discoverable. Robots Exclusion Protocol, RFC 9309
Next, connect the policy to the exact name in each vendor’s documentation. OpenAI lists OAI-SearchBot for search and GPTBot for crawling content that may be used for training. Anthropic documents Claude-SearchBot, ClaudeBot and Claude-User separately. Perplexity distinguishes its search crawler from its user-triggered fetcher. Treat this list as a set of documented examples and check the current source when editing the file. A token copied from an old template can refer to a different purpose or an outdated name. OpenAI crawler documentation · Anthropic crawler guidance · Perplexity crawler documentation
| Policy decision | Example requester or control | Evidence to use |
|---|---|---|
| Allow automated search discovery | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Vendor purpose and matching rules for the public path |
| Limit crawling for model training | GPTBot or ClaudeBot | Vendor scope and tested rule for the intended path |
| Handle a user-requested fetch | ChatGPT-User, Claude-User or Perplexity-User | Vendor-specific guidance and authorised response evidence |
| Keep Google Search crawlable | Googlebot | Google’s rule interpretation and the relevant property tools |
| Restrict private material | Application or network access controls | Authorisation tests by the owner |
User-triggered fetches deserve their own entry in the policy. OpenAI says robots.txt rules may not apply to ChatGPT-User. Perplexity says its user-requested fetcher generally ignores them. Consult each provider rather than treating the result for its search crawler as a prediction about its user fetcher. Google also publishes specific guidance for AI features in Search. It identifies ordinary Search eligibility as the starting point for supporting links in AI Overviews and AI Mode. Use Google's Search documentation to decide which crawler rule applies. OpenAI crawler documentation · Perplexity crawler documentation · Google Search AI features
Fetch the deployed file for the intended host
Use the exact scheme, hostname and port for the URLs in scope. The file at the root of one host governs that origin under the documented rules, so check separate subdomains where relevant. Take a copy of the file and the HTTP response headers. Look at the status and final address, then verify that the body is the intended rule file. A successful response containing an HTML error page gives you a deployment issue to investigate. Google’s robots.txt interpretation
robots_url='https://aivisibility.ae/robots.txt'
curl --silent --show-error --location --max-redirs 8 --max-time 20 \
--dump-header robots.headers --output robots.txt \
--write-out 'status=%{http_code} final=%{url_effective} redirects=%{num_redirects}\n' \
"$robots_url"
On 6 November 2026 at 01:16 UTC, this GET returned HTTP 200, no redirect and a plain-text content type. The file contained a wildcard group allowing the root and disallowing the prefix /admin. It also listed the site's sitemap. The captured response advertised a cache lifetime of 14,400 seconds. That header describes the observed response. A crawler may apply its own caching and refetch behaviour, so log when you checked the file and use current vendor guidance to interpret a lag. Live AI Visibility robots.txt · Google’s robots.txt interpretation
A sitemap entry helps identify a location where URLs are listed. Resolve crawler permissions through the relevant groups and path rules. Review the sitemap and the rules together if an important URL is listed but an applicable rule disallows it. This makes an unintended conflict easy to raise with the owner without treating the sitemap listing as an override.
Check the named group and the path boundary
Read the file as groups. A crawler with a matching named group uses the matching rules under the applicable parser. The wildcard group supplies fallback rules when a specific group does not match. Google documents that it uses the most specific matching user-agent group and combines applicable groups for the same agent. It does not merge the wildcard group into the specific group. Under the protocol, compare the matching path rules by specificity. Where two equally specific rules conflict, check the vendor interpretation. Google gives the less restrictive rule precedence in that tie. Robots Exclusion Protocol, RFC 9309 · Google’s robots.txt interpretation
The live example's Disallow: /admin applies to /admin and can also match a path beginning with the same prefix, such as /admin-guide/. If those are meant to be separate routes, make both explicit test cases. A proposed change to the slash boundary needs its own tests for the bare path, the directory form, descendants and any similarly named public routes. Keep the route variants that the site actually serves or redirects.
Here is the unpublished fixture used for comparison. It carries the common exclusions into the named search group, then gives the training crawler a separate sitewide rule. The public preview exception illustrates a more specific path. The paths are illustrative. Evaluate them offline.
User-agent: *
Disallow: /admin
Disallow: /preview/
Allow: /preview/public/
User-agent: OAI-SearchBot
Disallow: /admin
Disallow: /preview/
Allow: /preview/public/
User-agent: GPTBot
Disallow: /
The local parser allowed the checklist page for all five tested tokens under the live file and disallowed the two test paths beginning with /admin. Under the fixture, it still allowed the checklist page for OAI-SearchBot but disallowed it for GPTBot. The test also disallowed /preview/draft/ and allowed /preview/public/guide/ for OAI-SearchBot, illustrating the longer exception. The exercise evaluates text rules. It sends no requests to those example paths and supplies no evidence that a vendor attempted to crawl them.
A second local test reproduced the mistake from the opening example. The wildcard group disallowed /admin and the named OAI-SearchBot group contained only Allow: /. The parser disallowed the path for Googlebot through the wildcard group and allowed it for OAI-SearchBot through the named group. If both crawlers should be excluded there, repeat the restriction in the named group. Review every named group in the same way when adding one.
To reproduce the baseline locally, run the following with Python 3.14 and the captured file named robots.txt. Save the proposed fixture under another filename and run it again. These checks use the standard library parser in that version. Use vendor documentation for cases where its result and a platform tool differ.
from pathlib import Path
from urllib.robotparser import RobotFileParser
origin = "https://aivisibility.ae"
parser = RobotFileParser(origin + "/robots.txt")
parser.parse(Path("robots.txt").read_text(encoding="utf-8").splitlines())
for agent in ("Googlebot", "OAI-SearchBot", "GPTBot"):
for path in ("/", "/admin", "/admin-guide/", "/preview/public/guide/"):
print(agent, path, parser.can_fetch(agent, origin + path))
Include one test per policy boundary. Test a public URL that must stay available, a path that must remain excluded, a near match that should behave differently if intended and any specific exception. Where a relevant platform provides a parser or inspection tool, compare its result with your offline check. Different tools can implement edge cases differently. Use an isolated file and write down the parser version so the result is reproducible.
Separate a policy check from evidence of delivery
Review Google’s AI Mode documentation when deciding policy for its Search crawler. A public product announcement can introduce the feature, and vendor documentation can explain how site owners control access. Google’s Instagram reel included in coverage of AI Mode illustrates the product context. The crawler documentation is the source for operational decisions. Google AI Mode coverage with the official reel · Google Search AI features
If the rule permits a crawler but a request fails, check delivery separately. A firewall, rate limit or authentication layer can return its own response. Cloudflare documents how AI Crawl Control interacts with its firewall and bot features. On a site you administer, record the HTTP status and the edge action for a verified requester. Compare what the edge saw with what the origin received when the request was forwarded. If the only evidence is a user-agent string, report that limitation and verify identity using the relevant provider method where available. Cloudflare AI Crawl Control and bot configuration
A browser can show whether the body of a public page is visible in a clean session. The browser check and the rule check answer different questions. Run both when someone says a permitted page is still unusable. The same applies to indexing. Use platform-specific inspection where authorised and avoid treating an offline allow result as proof that a vendor indexed or selected the page.
Review the diff and record the retest
Use a Crawler path decision record with the origin, agent token, documented purpose, path, intended decision, effective group, winning rule and evidence date. Preserve a copy of the current file and the proposed revision. Compare the matrix before deployment, paying particular attention to any new named group and to rules that affect broad prefixes. Give the reviewer a public control path as well as the path under change.
After an approved deployment, fetch the live robots.txt again and compare the returned body to the reviewed file. Record the response code, redirect and caching headers. Re-run the matrix against the live copy. If a vendor has not yet reflected the change, check the vendor’s stated refetch behaviour and any permitted request evidence before making another edit. A precise handoff can say which path and requester changed, which paths stayed within policy and which external outcomes are still unobserved.
Assign one owner to the site’s published file and one owner to any separate edge policy if different teams manage them. When a new subdomain or preview area launches, add it to the review with its own root file and controls. When a vendor changes its crawler guidance, update the purpose entry first, then decide if an existing rule needs revision. That order keeps the file tied to the publisher's actual access decisions.
AI Visibility is published by Lunasol, which offers GEO and AEO services. The worked example uses a public robots.txt response and a local rule simulation. About AI Visibility



