Can AI Search Access Your Website? A Practical Checklist

Yes, almost certainly. Your robots.txt is the small text file that tells crawlers where they may go, and a public page on a valid certificate with a default one is already reachable by every AI crawler that matters. Across the thirty-four files we could read for this article, not one carried a blanket block. But access is really three questions, and most people check only one. Can the crawler reach your server? Is it allowed to fetch the page? Is what you sell written on the page as text? You can answer all three yourself with a browser, no paid tools, and an hour. Lunasol, the agency that owns AI Visibility, publishes a four-step process that begins with an audit (Lunasol); the access part of that work is written out in full below so you can run it yourself.

The short version: open your own site, then open your own robots.txt and read it. Find out which AI crawler names it mentions and whether each one is allowed or blocked. Check that no index or snippet switch is on by accident, and that your services, your city and your contact route exist as readable text rather than only inside an image. Access is where this checklist stops. Whether anyone mentions you is a different job.

What it means for AI to access your website

Access means three separate things: that a crawler can reach your server, that it is permitted to fetch the page, and that your facts are written on the page as text. Those three are what the Lunasol team checked on thirty-five UAE sites for this article, and they are what you are about to check on your own.

1. Reach means a crawler can get to your server. The certificate has to be valid, the page has to answer, and the redirect chain has to end somewhere sensible. A site nobody can open is not a site with an AI problem, it is a site with a hosting problem.
2. Permission is what your robots.txt grants. The file names crawlers and tells each one where it may go. The names matter, because one crawler governs whether you are eligible to appear in an assistant's search results and a different crawler governs whether your pages may be collected for model building. Those are separate decisions, made in different lines of the same file.
3. Readability is whether your facts are on the page as text. A crawler that reaches your page and is allowed to fetch it still has to find them. A price inside a JPEG, a service list inside a PDF, a phone number that only appears after a form submission: none of that is text an answer can quote.

Getting all three right earns you nothing in particular. Google states plainly that "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary" (Google Search Central), and OpenAI writes that "Any public website can appear in ChatGPT search" (OpenAI Help Center). Those three are the entry conditions, and almost every site in our own sample already cleared them.

One AI crawler decides search and another decides training

Every major AI company runs several separately named crawlers, and each name governs a different decision. One kind governs whether you are eligible to appear in an assistant's search results. A different kind governs whether your pages may be collected for model building. OAI-SearchBot and GPTBot, both OpenAI's, are the clearest pair. A third kind fetches a single page when a person asks a question, and the vendors say robots.txt may not apply to it at all. The name a crawler answers to in that file is called a token, and a common and costly mistake is treating one line as a decision about "AI".

The AI crawler tokens to look for in robots.txt

These are the tokens as they stand in September 2026, and each row is a separate decision.

GPTBot and OAI-SearchBot: what is the difference?

OAI-SearchBot governs whether your pages can appear in ChatGPT's search features. GPTBot governs whether your pages may be used to train OpenAI's models. On OpenAI's documented split, blocking GPTBot does not remove you from ChatGPT search, and blocking OAI-SearchBot is not what governs training. OpenAI's documentation says "OAI-SearchBot is used to surface websites in search results in ChatGPT's search features" and recommends the matching step: "To help ensure your site appears in search results, we recommend allowing OAI-SearchBot in your site's robots.txt file". Of the other one it says GPTBot "is used to crawl content that may be used in training our generative AI foundation models", so "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models" (OpenAI). The publisher FAQ puts the practical half in one line: "For your site content to be included in summaries and snippets in ChatGPT, make sure you aren't blocking OAI-SearchBot" (OpenAI Help Center).

Some fetches are not crawls at all. Of ChatGPT-User, OpenAI writes: "ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply" (OpenAI). Perplexity says the same of its own user-triggered fetcher: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules" (Perplexity). Robots.txt asks. It does not enforce.

Google's AI surfaces run on ordinary Search access. There is no separate token to allow for AI Overviews or AI Mode. Google puts Googlebot at the centre of it: "robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search". Eligibility then follows from ordinary indexing, because a page has to be "indexed and eligible to be shown in Google Search with a snippet" before it can appear as a supporting link in either surface (Google Search Central). Google-Extended is a different question, and Google is explicit that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search" (Google Search Central).

Anthropic documents three names, and Common Crawl one. ClaudeBot "helps enhance the utility and safety of our generative AI models by collecting web content", Claude-User "supports Claude AI users", and Claude-SearchBot "navigates the web to improve search result quality for users" (Anthropic). Common Crawl describes CCBot as "Common Crawl's Nutch-based web crawler", blocked with a plain "User-agent: CCBot" and "Disallow: /" (Common Crawl).

For Bing's chat answers Microsoft added controls that live in the page. Its guidance is that "Content with the NOCACHE tag may be included in Bing Chat answers. We will only display URL/Snippet/Title in the answer", while "Content tagged NOARCHIVE will not be included in Bing Chat answers, not be linked to in the answers", and content carrying either tag "will still appear in our search results" (Bing Webmaster Blog).

What we found when we ran this checklist on thirty-five sites

We ran these checks over thirty-five UAE accounting and tax firm websites in September 2026, twenty of them cited in AI answers and fifteen not, and crawler permissions did not separate the two groups at all.

Here is the method, because it limits what the numbers are worth. We asked two questions a Dubai buyer would ask, in ChatGPT and in Google's AI Mode: who the best accounting and bookkeeping firms in Dubai are for a small business, and which company to use for VAT and corporate tax filing. The first returned a full answer twice in ChatGPT. The second cut off part-way each of the three times we ran it, which we log as three failures rather than replace, so only its source links were usable. Google's AI Mode answered each question once, and on the first question it carried no source links in the prose at all: its only two outbound links came from the map card. From those five answer sets, the four that completed plus the source links under the one that cut off three times, we took twenty business domains, and separately fifteen firms in the same category that ordinary Google results returned and no answer had pointed to. The whole survey ran from a browser rather than from a server, on one ordinary connection, one attempt per file, in a single day. Fourteen of the fifteen uncited sites could be opened at all, and the uncited counts below are against those fourteen. Three domains without the www redirect to their www address, which made our own homepage fetch fail until we re-ran them there; those three failures were ours, not the sites'. We did not record platforms, response headers, or whether anything sat in front of these sites. Far too small to be a study, and we name no company.

Almost nobody in the thirty-five had made a decision about AI crawlers. Four of the twenty cited sites named any AI crawler in robots.txt, and two of the fourteen reachable uncited ones. Of the thirty-four files we could see, twenty-eight named none at all. None of them carried a blanket "User-agent: * / Disallow: /". So permission did not separate the two groups. With almost nobody on either side having set a rule, there was nothing to compare.

The most deliberate file in the sample was in the uncited group. It blocks the main training and archive crawlers and allows the search and user-triggered ones. That is close to the split described above, written line by line by somebody who had actually read the documentation, although one of the training tokens is left allowed. One site in the cited group had made a narrower version of the same decision. That is two files out of thirty-five where anybody had really thought about it.

One site could not be opened at all. On the day we checked, Chrome showed its certificate warning page, so we never saw it, and it sits in the uncited group. A second uncited site served an empty robots.txt, which imposes no restriction but records no decision either. One hard failure out of thirty-five is not a pattern and we are not claiming one, but it is a reminder that the quickest check here is the one that catches the worst fault.

The difference showed up in the page text. Every one of the twenty cited homepages carried more than 6,000 characters of readable text, median about 16,400; three of the fourteen reachable uncited homepages carried under 5,000. A contact number written as text appeared on nineteen of twenty cited homepages and ten of fourteen uncited. Structured data in JSON-LD, the machine-readable block that restates a page's facts in a fixed format, appeared on eighteen of twenty cited against eleven of fourteen uncited. The gaps are small and the sample is smaller, and none of it shows cause. But what differed was what the page said, not what the crawler was allowed to do.

How to check if AI can access your website, step by step

To check whether AI can access your website, work through these sixteen checks in order, starting with your own homepage and your own robots.txt. Steps 1 to 8 are browser work, about fifteen minutes if nothing is wrong, and every check with a clean verdict is marked PASS or FAIL. Two reach outside your own admin, so read them before you start: step 11, which asks whether any service sits in front of your site, and step 16, because Search Console access can take a day to arrange. Step 10 you can run yourself, though the fix, if there is one, belongs to your host. The keystrokes are for Chrome, and Edge behaves the same way.

1. Open your own website, and look at the address bar. Type your domain and press Enter, or Return on a Mac. PASS: the page loads with a padlock. FAIL: a warning page, an expired certificate, a parked page or nothing at all. A parked page is the holding screen a registrar shows for a domain with no site on it. Stop here and call whoever runs your hosting, because nothing further on this list matters until this works. The one site in our sample we could not read at all failed at exactly this point.
2. Try all four versions of your address. Type each of these and note where it settles: yourdomain.ae, www.yourdomain.ae, http://yourdomain.ae, http://www.yourdomain.ae. PASS: all four end at the same address, with a padlock, on your real homepage. FAIL: any of them lands on an error, a parked page or a different site. That is a redirect to fix, and it belongs with your hosting or your registrar rather than in your site's admin.
3. Now go and read your own robots.txt. Type your domain followed by /robots.txt. The file has to sit at the top level of the domain: "You must place the robots.txt file in the top-level directory of a site, on a supported protocol" (Google Search Central). PASS: a plain text page appears. FAIL: your styled 404 page appears instead, which means there is no file. A blank white page means the file is there but empty. Neither is an emergency, because no file and an empty file both mean nothing is restricted, so carry on down the list and come back to step 8 if you want to set anything.
4. Do this once for every address you own. robots.txt is per hostname, so ar.yourdomain.ae, shop.yourdomain.ae and yourdomain.com each have their own file and can each be set differently. If your Arabic site is a folder rather than a subdomain, at yourdomain.ae/ar, it shares one file with the English site, and a "Disallow: /ar/" line left over from launch hides the whole Arabic site. Check your staging address too, if you have one.
5. Is the whole site shut? Look for a group that reads "User-agent: *" followed by "Disallow: /". That closes the site to every well-behaved crawler. PASS: no such pair anywhere in the file. FAIL: you find one, in which case stop and treat it exactly like step 1, today, with whoever runs your site. We found none in our thirty-five. Where it does happen, it is usually a staging site that went live, or a migration that went wrong.
6. Learn to read one group, because only one applies. A crawler obeys exactly one group: the most specific one that names it, or in Google's phrasing "the group with the most specific user agent that matches the crawler's user agent" (Google Search Central). A permissive "User-agent: *" block does not rescue a crawler with its own stricter group further down. Inside a group, "Disallow:" with nothing after it means allowed everywhere, "Disallow: /" means blocked from the whole site, and "Disallow: /wp-admin/" means blocked from one folder, which is normal. An "Allow:" line re-opens something a Disallow above it closed. Paths start at a slash and are case sensitive, so a rule written for /Services does not cover /services.
7. Search the file for seven names. With your robots.txt still open on screen, press Ctrl+F, or Cmd+F on a Mac, and look for GPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended, PerplexityBot, ClaudeBot and CCBot. For each one write down two things: named or not named, and allowed or blocked. PASS: each name matches the decision you would make if asked. FAIL: a "Disallow: /" under OAI-SearchBot, PerplexityBot or Googlebot when you want to be eligible in search results, or an open GPTBot, ClaudeBot, Google-Extended or CCBot when you would rather not be collected for model building. If a name is absent, the "User-agent: *" group decides on your behalf.
8. Find out where your robots.txt comes from before you try to edit it. On many sites there is no such file on disk. The platform, or an SEO plugin, which is an add-on that manages how search engines see the site, generates one on request, so where you edit it, and whether you can edit it at all, depends on what you are running. Before you change a line, look up the current help page for your own platform, whether that is WordPress, Wix, Squarespace, Shopify or something else. Search its help site for two things: robots.txt, and search engine visibility. Then write down one of two answers. Either you can edit the file yourself and here is the screen, or you cannot and this one goes to whoever built the site. That single line is the difference between fixing step 7 this afternoon and sending it to somebody.
9. If somebody has ever switched off indexing, the evidence is in the page code. These are instructions to search engines. noindex says do not list this page. nosnippet says do not quote from it. noarchive and data-nosnippet are narrower versions of the same idea. On your homepage in Chrome press Ctrl+U, or Cmd+Option+U on a Mac, and the page's source code opens in a new tab. In Safari on a Mac, switch on the Develop menu first, under Settings then Advanced. Press Ctrl+F, or Cmd+F on a Mac, and search it for each of those four words. PASS: no results. FAIL: the word sits inside a tag that starts <meta name="robots", so look at the few characters just before your highlighted hit. A stray match buried in a block of script is almost always harmless and is worth a question rather than an alarm. Google's guidance is that to limit what is shown you "use nosnippet, data-nosnippet, max-snippet, or noindex controls" (Google Search Central), so if one is live by accident, the limiting is already happening.
10. Look at your own response headers. The same controls can arrive as an X-Robots-Tag header, which never appears in the page source. In Chrome on Windows press F12, holding Fn if your keyboard needs it, or Cmd+Option+I on a Mac, open the Network tab, reload the page, click the first row and read the Response Headers panel. PASS: no x-robots-tag line. FAIL: an x-robots-tag saying noindex or nosnippet, which is a server setting and the one item here you will almost certainly need your hosting provider to change.
11. Ask whether anything sits in front of your site. A content delivery network, a firewall or a bot-management service, any of which sits between your visitors and your server, runs above your website. Its settings are separate from your robots.txt and invisible in your site's own admin. You can collect the first evidence yourself. Go back to the Response Headers panel from step 10 and read the whole list, not just the x-robots-tag line. Copy it into a document, because these services usually name themselves in those headers. Send that list to whoever runs your hosting with one question: is anything sitting in front of our site, and are its bot or crawler rules blocking anything? Your job is to hand over the evidence, so they can answer in a sentence. A clean robots.txt can be overruled at this layer.
12. Know what blocking does and does not do. A disallowed page can still show up as a bare link. Google's own wording is that it "may still index the URL and show it in search results without a snippet" (Google Search Central). OpenAI says something similar of its browser: if it obtains the URL of a disallowed page from elsewhere and the page looks relevant, "we may surface just the link and page title in ChatGPT Atlas" (OpenAI Help Center). To keep a page out of an index the tool is noindex, not Disallow, and the two do not combine, because noindex only works on a page a crawler is still allowed to fetch. A noindexed page is also not eligible as a supporting link in Google's AI surfaces.
13. Read your homepage the way a machine does. Click once on a blank part of the page, then press Ctrl+A and Ctrl+C, or Cmd+A and Cmd+C on a Mac, and paste into any blank document. What lands there is roughly what an assistant can read. Look for six things: what you do, who you do it for, where you are, what it costs or how pricing works, what is included, and how to reach you. Anything missing from that pasted text is a fact no answer can quote back to a customer. This is a presence test only: you are asking whether each fact exists as text, not whether it is well written.
14. A price inside a JPEG is a price no answer can quote. Check the same six facts are not trapped in an image, a PDF or a contact form. In our sample, every cited homepage carried more than 6,000 characters of text and nineteen of twenty carried a phone number as text. Two small gaps in a small sample. Writing a fact out as text costs nothing either way.
15. Decide about llms.txt honestly. If you want one, publish it, then open the address and see which you get: a file, or your 404 page. Four sites in our sample served an ordinary web page there while reporting success, which is a broken address dressed up as a working one. Twelve of the twenty cited domains and seven of the fourteen reachable uncited ones served a real file. Do not report it as a result, though. Its own home page calls it "A proposal to standardise on using an /llms.txt file to provide information to help agents use a website" (llmstxt.org), and none of the crawler documentation quoted in this article mentions reading it.
16. Get into Search Console and find the Performance report. Google folds these surfaces into ordinary reporting: "Sites appearing in AI features (such as AI Overviews and AI Mode) are included in the overall search traffic in Search Console" (Google Search Central). If nobody at your company has access, start here rather than at step one. Proving ownership means adding a record at your domain registrar, uploading a file to your hosting, or pasting a verification tag into your site. Ten minutes for whoever holds those logins. Ask for Search Console access as a full user.

What to fix first, and what to ask someone else to do

Fix a certificate warning or a blanket Disallow first, because both make every other question irrelevant. Then any accidental noindex or nosnippet. Then your deliberate crawler decisions. Last, what your pages actually state as text.

1. Fix these today. A certificate warning or a blanket Disallow, with whoever runs your hosting.
2. Next, anything switched on by accident. An accidental noindex or nosnippet is often a single toggle in the same settings screen where somebody switched it on. On WordPress it is Settings, then Reading, then the box marked "Discourage search engines from indexing this site", and other platforms have their own version of the same switch. Check which site you are on before you clear it: on a staging or pre-launch address that box is usually ticked deliberately, and clearing it there puts unfinished pages into search results. We found no accidental noindex in our sample, which does not mean you will not.
3. Then the decisions only you can make. Decide whether you want your pages collected for model building. That is GPTBot for OpenAI's training, Google-Extended for Gemini training, ClaudeBot for web content collected for Anthropic's models, and CCBot for Common Crawl's own crawler. Then decide, separately, whether you want to be eligible in assistant search results. That means OAI-SearchBot, PerplexityBot and Claude-SearchBot, and for Google's AI surfaces it just means ordinary Googlebot access. These are business decisions rather than technical ones, and a publisher with a paywall will answer differently from an accounting firm that wants the phone to ring.
4. Last, and most useful, the page itself. Nothing in our own sample separated the cited sites from the uncited ones at the crawler layer. What differed, a little, was how much of the business was written down. Turning that text into something worth quoting is a different job from this one.

What you can change yourself, and what you have to ask for. Inside your site's own admin, usually: robots.txt lines on most platforms, the indexing toggle, and the page text. Needing hosting, registrar or developer access: a certificate, redirects, an X-Robots-Tag header, the settings of any service sitting in front of your site, a robots file held in your theme, and Search Console verification. One last item belongs to your developer. The name a crawler gives when it arrives in your server log can be faked, and Google says so plainly: "The HTTP user agent string can be spoofed" (Google Search Central). If anyone is blocking crawlers on the strength of log lines, ask them to check those visits against the published IP ranges first (OpenAI, Perplexity).

Common questions

Can AI see my website? Yes, if it is public, loads over a valid certificate and does not carry a blanket Disallow. That was true of thirty-four of the thirty-five sites we checked.

Will allowing every crawler get me mentioned? No. Nothing about access obliges an assistant to name you. It only makes you eligible.

Does ChatGPT crawl my site? Yes, but "crawl" covers four different jobs. OpenAI documents GPTBot as the crawler whose content may be used in training and OAI-SearchBot as the one that matters for appearing in ChatGPT's search features (OpenAI).

Do I need a special robots.txt rule to appear in AI Overviews? No. Google says there are no additional requirements to appear, and ordinary Googlebot access is what governs eligibility (Google Search Central).

A note from the Lunasol team

The check that finds the most in the least time is step seven, read against step six. Of the thirty-four robots.txt files we read for this article, twenty-eight named no AI crawler at all. Step eight is where you find out whether that file is yours to edit. The question "which of these names is in our file, and what does each one do" is one few site owners have ever been asked. A file that names nothing has decided nothing, so write the answer down once, a line per crawler, and you never have to guess again.

The Lunasol team, Dubai

If you would rather have it checked for you, Lunasol offers a free AI visibility check as a starting point. Its published process and its contact routes, WhatsApp and email, are at Lunasol.

Documentation quoted here. OpenAI's crawler overview and publisher FAQ, Google Search Central on AI features, common crawlers and robots.txt, Perplexity's bot guide, Anthropic's crawler support article, Microsoft's Bing Chat announcement, Common Crawl's CCBot page and llmstxt.org. Each is linked at the point it is quoted, and all were read on 21 September 2026.

This article is general technical guidance, current at September 2026, and it is not legal or contractual advice. It asks you to look at, and possibly change, live configuration files and page controls. Make those changes with whoever runs your site; neither this publication, its owner nor the author accepts liability for the consequences of a change made on the strength of it. The survey described above covered thirty-five sites in one sector, in one day, over one connection, with one attempt per file, and it illustrates a method rather than proving a rule; nothing in it is a finding about any particular company. Assistant answers are not reproducible: they vary by account, personalisation, location and time, and the answers behind this survey came from one account on one day. Crawler names, robots.txt behaviour and platform documentation change without notice, so check the current version of any page quoted here before acting on it. Allowing a crawler does not cause a mention, a citation or a ranking in any AI product, and nobody can promise one.

Product names are used here for identification only, and neither this publication nor its owner is affiliated with, endorsed by or sponsored by any of the companies named. ChatGPT, ChatGPT Atlas, GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot are products of OpenAI. Google Search, Search Console, AI Mode, AI Overviews, Gemini, Chrome, Googlebot, Google-Extended and Google-InspectionTool are products of Google. Perplexity, PerplexityBot and Perplexity-User are products of Perplexity AI. Bing, Bing Chat, Windows and Edge are products of Microsoft. Mac and Safari are products of Apple. Claude, ClaudeBot, Claude-User and Claude-SearchBot are products of Anthropic. CCBot is operated by the Common Crawl Foundation. WordPress is a product of the WordPress Foundation, Wix of Wix.com, Squarespace of Squarespace, Shopify of Shopify, and WhatsApp of Meta. All are trademarks of their respective owners.

AI Visibility is owned by Lunasol.