# How to Test Whether a Specific Page Can Be Retrieved By AI Visibility editorial team AI-assisted article. Your homepage opens, but a service page appears to be missing from a search result. Before changing a sitewide crawler rule, test the exact address and identify the evidence you need. A successful browser load says something about one visitor’s access. A search platform’s inspection report says something about that platform. A verified request in an edge log shows what one requester received at a particular time. Each can help narrow the problem. Pick one public URL, one nearby page that works, and one sentence that the target should expose. Collect the HTTP response, check the rendered content and consult the relevant platform or permitted logs if the question requires them. This walkthrough uses the existing AI Visibility access checklist as the test page. The checklist covers sitewide setup and crawler policy. Here, the job is to diagnose one address and make a retest repeatable. AI Visibility site access checklist (/article/can-ai-search-access-your-website-a-practical-checklist/) Name the URL and the requester Copy the exact address from the report, search result, link or error message. Keep the scheme, hostname, case, path, trailing slash and query string. Keep a fragment in your notes if it affects what the browser should show, although the fragment is not sent as part of the HTTP request. Record the URL you expect the request to reach. That gives you a way to tell an intentional redirect from a route that silently sends visitors to the homepage. Choose a control that is likely to share the same delivery path, such as another article or service page on the same host. A homepage is a useful coarse control if you have no closer match. It can show that the hostname responds while leaving template or path differences open. Pick a content marker from the body that is unlikely to appear in the site's header, menu or error template. Record the intended requester too. A customer browser, Google Search inspection and an OpenAI search crawler are distinct checks with different evidence. Check Record What the result supports HTTP response Requested URL, hop status, destination, final status and headers What this client received Response body Page title and unique main-content marker Whether the tested GET supplied the expected text Browser rendering Marker in visible content, clean session and relevant resource errors What the tested browser displayed Platform inspection Platform, live or indexed view and time What the platform reports within that tool Permitted logs Verified requester, path, timestamp, action and response An observed request at the recorded layer Capture the full GET before diagnosing it Use a public URL and request it without signing in or reusing a browser session. A GET records headers and a body together. It lets you check whether the final response is the page you expected. Keep a modest timeout and a redirect limit. Replace the example address with a public page you are authorised to check. Keep downloaded responses local if they include identifiers or sensitive headers. url='https://example.com/replace-with-the-exact-path' curl --silent --show-error --location --max-redirs 8 --max-time 20 \ --dump-header page.headers --output page.html \ --write-out 'status=%{http_code} final=%{url_effective} redirects=%{num_redirects}\n' \ "$url" grep -iE '^(HTTP/|location:|content-type:|x-robots-tag:)' page.headers Read the hops in order. If the final address differs, decide whether the redirect matches the route you intended. If you get a success status, check that the body contains the expected content. A sign-in screen, branded error message or generic landing page can be served with a success code. Google's crawling documentation gives the related example of a page that returns success while its content resembles an error and Google classifies it as a soft 404. Google Search crawling troubleshooting (https://developers.google.com/search/docs/crawling-indexing/troubleshoot-crawling-errors) On 6 November 2026 at about 01:12 UTC, the checklist URL with its trailing slash returned HTTP 200 with an HTML content type. The version without the slash returned HTTP 301 and then HTTP 200 at the trailing-slash address. The homepage returned HTTP 200. The tested article’s response body included its title, the heading “What it means for AI to access your website” and a canonical link pointing to the trailing-slash address. No X-Robots-Tag header appeared in the captured article response. These are observations from one public client. The redirect explains the difference between the two requested addresses in that run. Tested checklist page (/article/can-ai-search-access-your-website-a-practical-checklist/) If both the target and control fail, start with the shared hostname, network or delivery layer and record the error before retrying. If only the target fails, preserve the path and look for a route, redirect or edge rule that applies to it. A 403, 429 or 5xx is a finding about the response you received. The code alone rarely identifies the rule or component that generated it. Stop and capture a timestamp, relevant response headers and any request identifier that the owner can use for correlation. Compare the response body with a clean render Open the page in a new browser profile or private session. Check that the visible page contains the marker without a login, consent action or a particular scroll position. Inspect the main document and any request that supplies the missing content. Chrome DevTools can show response headers and bodies in its Network panel and can disable the browser cache during a repeat check. Use the Elements panel or the visible page to determine whether a string found in source is actually displayed. Chrome DevTools Network reference (https://developer.chrome.com/docs/devtools/network/reference) For the checklist, a fresh headless Chrome profile returned a rendered document containing the same article heading and content marker. The marker was already present in the raw GET. If your result differs, record the earliest step at which the text appears. For example, when a script calls an API to load service details, test whether that request succeeds in a clean session and whether the content is present at initial render. If the marker appears only after interaction, write down what you did. Google's JavaScript troubleshooting guide shows how to check rendered HTML and resources through its own inspection tools. Google JavaScript troubleshooting guide (https://developers.google.com/search/docs/crawling-indexing/javascript/fix-search-javascript) Third-party embeds offer a useful way to see the distinction. Meta's Rugby World Cup newsroom article includes an Instagram post attributed to Meta and also carries an ordinary text explanation. Treat the parent article, embed and fallback permalink as separate items to check. A readable parent page can still contain an embed that fails in the tested environment. Keep facts that readers need for a decision available as ordinary page text. Meta newsroom example with official Instagram embed (https://about.fb.com/news/2023/08/celebrate-the-rugby-world-cup-france-2023-by-sharing-your-passion-with-fans-worldwide/) Check the marker in its context. A heading from the article may also appear in the table of contents, so check for a phrase from a nearby paragraph before concluding the article body is available. A price in a JavaScript data object may be present in the response even if a visitor cannot see it. Conversely, a successful browser render may show a price added later by a separate request. Save the relevant excerpt from the document and note where the browser displays it. That distinction helps a developer identify whether the change belongs in the server template, the client application or the service that supplies the content. View the selected official Instagram post (https://www.instagram.com/p/CwhxY5PAqyG/) Official Meta Instagram post embedded in its Rugby World Cup newsroom article, shown here to illustrate a third-party dependency. Match platform and log evidence to the question If you have access to the relevant Google Search Console property, enter the exact URL in URL Inspection. Record whether you are reading the indexed view or running the live test. The indexed view can report Google's selected canonical. The live test can expose the tested HTML and resources and, where available, a rendered screenshot. Inspect the final destination separately when the requested address redirects. A live test is a check at the time it runs. The indexed result reflects the version in Google's index and can lag a repair. Google Search Console URL Inspection documentation (https://support.google.com/webmasters/answer/9012289?hl=en) For a documented AI search crawler, check the applicable policy for the exact host and path. OpenAI distinguishes OAI-SearchBot from GPTBot and describes ChatGPT-User as a user-triggered agent. Choose the documented requester that matches the question. If you change the user agent string in a local request, label the result as a simulation. It may help isolate a rule that reacts to the string, but it does not establish how a request from the vendor's infrastructure was treated. OpenAI crawler documentation (https://developers.openai.com/api/docs/bots) If you are authorised to see logs, search the edge first for the exact time and path. Record the layer, response code, rule action, source verification and request identifier where available. Check the origin when the edge forwarded the request. An edge block may leave no origin entry. A string saying Googlebot needs verification before it can support a claim about Google. Google's documentation lists supported verification approaches and ranges by requester category. Use the vendor's own method for the agent you are investigating. Google requester verification guide (https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) · Cloudflare AI Crawl Control documentation (https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/) The checklist example used public responses and a fresh Chrome render. It had no permissioned Search Console result, edge log or origin log. Those fields remain unknown. An empty search in a log can reflect the selected time window, retention or request path. Record the filters and coverage before treating it as evidence about a particular request. Decide what to fix and repeat the same test Use a Page retrieval record with the exact URL, expected destination, target marker, tested client, control, UTC time, response chain, rendered result and any verified platform or log evidence. Write one sentence describing the failure supported by the record. “The old service link redirects to the homepage” gives the owner something concrete to fix and retest. “AI cannot read the website” leaves the owner guessing which layer failed. Assign the repair to the layer that produced the finding. Fix a wrong redirect with the team that owns the route. If a clean browser loads an empty shell, investigate the dependencies that provide the article body. If a verified requester received an edge challenge, review the named rule with the site owner. Preserve intended access controls and check that the change applies to the public path you meant to expose. When the evidence reaches only your own client, record what you can say about that client and keep the vendor outcome open. If the page has intentionally moved, include both URLs in the handoff. The old address should lead to the intended destination, while the final address should return the intended content on its own. If a control unexpectedly changes as well, check whether the release touched shared routing or delivery before attributing the outcome to a single-page fix. Retest the original address and the control using the same steps. Verify the intended destination, marker and status, then record the date and what changed. If the underlying question was about search index inclusion, use the platform’s indexed view when it next reflects the change. A successful repair proves the specific failure cleared under the conditions you observed. Whether a platform later selects the page for a particular answer requires separate evidence. AI Visibility is published by Lunasol, which offers GEO and AEO services. The worked example uses public HTTP and browser evidence. About AI Visibility (https://aivisibility.ae/about/)