A publisher wants people to find its reporting through search and wants to limit certain uses of that reporting for model development. Its technical team is asked to block “the AI bot.” The instruction leaves out which vendor, which purpose and which content the publisher wants covered. A team can implement exactly what the request seems to say and still make the wrong choice about search.
Start with a decision about use, then map it to the controls each provider actually documents. Some providers expose separate crawler identities. Others use a robots.txt token to express a preference about data their existing crawler collects. A user-requested visit can have its own rules. This guide helps editors and technical owners agree on what to approve, what to verify after release and which outcomes remain outside the site’s evidence. The existing AI Visibility checklist covers basic site access. AI Visibility site access checklist
Give search, training and other uses separate decisions
Write down the product outcomes you want to support. A public service description might be useful to people looking for a provider. A publisher might want its articles discoverable through ordinary search while taking a different position on model development. A paid subscriber area might require access controls regardless of the publisher’s preference about either use. Give each area and purpose its own decision. Where the vendor groups several uses under one control, record that tradeoff for the person approving the change.
A search crawler collects or examines content to support search. A model-training control expresses a preference about use for model development as described by the provider. A grounding control may affect use of retrieved material when an answer is generated. A user-triggered fetch is a request made in connection with a user action. These categories can overlap in a vendor's implementation. Use the vendor’s documentation to determine what the available control covers. OpenAI crawler reference · Google common crawler reference
| Provider | Search route or crawler | Other documented control | Decision to write down |
|---|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot | Search policy and training-crawl policy separately |
| Anthropic | Claude-SearchBot | ClaudeBot, with Claude-User documented separately | Search, collection and user-directed requests |
| Googlebot for Search | Google-Extended | Search and the Gemini-related uses described by Google | |
| Apple | Applebot | Applebot-Extended | Discovery and the documented foundation-model training use |
| Perplexity | PerplexityBot | Perplexity-User for user-directed access | Search and user-requested access as separate questions |
Read the controls as the provider describes them
OpenAI documents OAI-SearchBot and GPTBot settings as independent. It explicitly gives allowing the search crawler while disallowing the training crawler as an example. It also says that when a site allows both, a single crawl can supply both purposes. That matters to the interpretation of logs. A count of requests under one name cannot establish every downstream use of the response. Treat the rule as an expression of policy for the documented control and record any traffic separately. OpenAI crawler reference
Anthropic documents Claude-SearchBot for search optimisation, ClaudeBot for potential collection for model development and Claude-User for retrieval at a user’s request. Record all three decisions even when they happen to have the same outcome. If an editor approves a change to search discovery later, the technical team can then tell which entry needs reconsideration and which controls have different owners. Anthropic crawler guidance
Google-Extended has no separate HTTP user-agent string. Google describes it as a product token that manages whether crawled material can be used for training future Gemini models within the documented scope and for grounding in named Gemini and Vertex AI products. Google says it does not affect inclusion or ranking in Google Search. Review the described grounding effect alongside training before approving the setting. Google’s separate AI features documentation explains the relationship between ordinary Search eligibility and supporting links in AI Overviews and AI Mode. Google common crawler reference · Google Search AI features
Apple describes Applebot-Extended as a control for the specified use of data collected by Applebot to train foundation models. Apple says the extended token does not crawl pages and that disallowing it still permits Search inclusion. The practical check is to inspect the published directive and the scope Apple describes. Searching access logs for a separate Applebot-Extended visitor would test the wrong thing. Check Apple’s other published controls separately if the concern is how content can be used to provide context in a generated result. Applebot and Applebot-Extended documentation
Keep a separate note for user-directed requests. Perplexity says Perplexity-User generally ignores robots.txt because a user initiated the request. OpenAI says the file’s rules may not apply to ChatGPT-User. For sensitive content, use authentication and server-side authorisation. The vendor’s handling of a voluntary directive is a separate policy question. Perplexity crawler documentation · OpenAI crawler reference · Robots Exclusion Protocol
Compare real policies without copying them
On 6 November 2026, three public robots.txt endpoints were fetched from The Guardian, The New York Times and English Wikipedia. Each returned HTTP 200 and a plain-text content type. Their saved files were evaluated locally with Python’s urllib.robotparser against one example URL per origin and eight tokens. The test read the file and evaluated its rules. It did not send vendor requests or measure search visibility. Three origins and one URL per origin are enough to show contrasting configurations. The results describe these samples. The Guardian robots.txt · The New York Times robots.txt · English Wikipedia robots.txt
For The New York Times homepage, the sampled file permitted Googlebot and Applebot in the local evaluation. Separate directives disallowed Google-Extended and Applebot-Extended. The same file disallowed both OAI-SearchBot and GPTBot at that address. A publisher can make different choices across providers. The file supplies observable evidence of directives and does not explain the publisher’s reason for selecting them. The New York Times robots.txt
For The Guardian’s international path, the sampled file grouped ClaudeBot, Claude-SearchBot and Claude-User with a shared disallow rule. The local results disallowed both ClaudeBot and Claude-SearchBot on the selected path. The same evaluation permitted Applebot and disallowed Applebot-Extended. For the sampled English Wikipedia article path, the parser permitted all eight tested tokens. That result applies to the particular path and file captured. A different path could match a different rule. The Guardian robots.txt · English Wikipedia robots.txt
To repeat the comparison, fetch a public file at the root of the origin, save its final URL and headers and read the directives. Save the exact page path you evaluated and keep the result by token. Label results for Google-Extended and Applebot-Extended as evaluations of control tokens. Where a vendor-specific parser is available, use it to check any borderline group or path result before treating it as an operational finding. Google’s robots.txt interpretation
For the New York Times example, a minimal repeat uses Python 3.14 and the saved public file. Run the download as a normal client and evaluate the result locally. The parser’s output applies to the file you saved.
curl --silent --show-error --location --max-time 20 \
--dump-header nyt.headers --output nyt.robots \
'https://www.nytimes.com/robots.txt'
python3 - <<'PYTHON'
from pathlib import Path
from urllib.robotparser import RobotFileParser
origin = "https://www.nytimes.com"
parser = RobotFileParser(origin + "/robots.txt")
parser.parse(Path("nyt.robots").read_text(encoding="utf-8").splitlines())
for token in ("Googlebot", "Google-Extended", "Applebot", "Applebot-Extended"):
print(token, parser.can_fetch(token, origin + "/"))
PYTHON
Read the result against the decision you intend to make. In this example, checking only Googlebot would miss the distinct extended-use setting. Checking only Google-Extended would leave ordinary Search policy unanswered. Keep both entries in the record even if the values happen to match. When two different providers use similar token names, compare the product descriptions before applying the same decision to both.
Approve a change that the technical team can test
Use a Crawler use decision record with the content owner, origin, representative paths, provider, intended product effect, token, expected result and evidence date. Start with a sentence the owner can approve. For example, keep the public help centre eligible for documented search crawling and apply the selected training controls to that area. If a single control covers more uses than the owner intended, bring the tradeoff back to the decision owner before implementation.
For a brand with a public product site and a separate documentation host, start with one sample from each. A product policy might cover public pages only. The documentation host might also have a preview area whose access is restricted by the application. The technical team can evaluate the same search and training choices on both hosts while keeping separate evidence for application access. If a site-wide rule is requested, add examples that demonstrate what happens to ordinary search, public content and excluded sections after the change.
An Instagram announcement can help identify a product surface. For example, an official ChatGPT reel appears in public coverage of shopping and product discovery. Use such material for product context. Use the platform’s technical documentation for the allowed purpose and the exact control. This keeps the decision anchored to wording that a site owner can check when the interface or launch message changes. Coverage embedding the official ChatGPT reel · OpenAI crawler reference
Decide how the organisation will handle a provider for which no documented control maps cleanly to the request. Record the open question and route it to the owner instead of substituting a different vendor’s token. If the policy only covers new material, make the time boundary explicit in the approval. A current directive and a claim about material collected before the change require different evidence.
Verify the publication, then check the relevant outcome
Have the site owner compare the approved rules with the file returned by the public origin after deployment. Record the response status, final address and relevant caching information. Evaluate a public control URL and the paths affected by the change. If the site uses an edge block in addition to robots.txt, record the owner and scope of that separate enforcement rule. A crawler may encounter a network restriction even when the file permits the path. Google’s robots.txt interpretation · Cloudflare AI Crawl Control documentation
Where you are authorised to inspect logs, look for the specific requester and path over a defined time window. Verify identity using the provider’s documented method before attributing traffic. Inspect an ordinary clean render if the search path must remain readable, and use a platform’s own inspection for indexing questions where available. The three-file comparison used public responses and offline parsing. Private request logs and platform inspection were outside its evidence. Google requester verification guide
Close the review by giving the owner a result that describes what the team actually checked. A change record can say the public file contains the approved directive and the tested search token still permits the public path. An authorised log can add evidence of a particular later request. Claims about model training, an answer citation or a ranking change require evidence of those outcomes. Keep the record current when the vendor changes the meaning of a token or when a new content area needs its own policy.
AI Visibility is published by Lunasol, which offers GEO and AEO services. About AI Visibility



