How to Verify a Crawler's Identity Before Allowing It
Verify crawler identity before allowing requests by checking trusted source addresses, DNS, vendor IP ranges and managed verification evidence.
Crawler access, structured data and internal links: the technical checks that decide whether AI search can read your site at all.
Verify crawler identity before allowing requests by checking trusted source addresses, DNS, vendor IP ranges and managed verification evidence.
Diagnose crawler firewall denials by verifying identity, correlating edge and origin events, and testing a narrowly scoped correction.
Decide when PDF menus, brochures and price lists need HTML by testing text extraction, mobile visibility and update ownership.
Compare original HTML, closed and open browser states, keyboard activation and requests to find content that depends on a click.
Test how JavaScript affects AI discovery by comparing raw HTML, visible browser text, interactions, and failed data requests.
Build an XML sitemap from maintained canonical business pages, test the live XML and its entries, and verify processing separately from indexing.
Verify migration canonicals by comparing the URL map, redirect chain, raw and rendered declarations, and Google’s selected canonical.
Find the source of an accidental noindex directive by comparing the final response headers, initial HTML and rendered document head.
Separate AI search and training crawler decisions, understand Google and Apple extended-use tokens, and compare three real publisher policies.
Review robots.txt rules by documented crawler purpose and test named groups, path prefixes and exceptions before changing the live file.
Use a public HTTP response, clean browser render and platform-specific evidence to diagnose retrieval of one exact page URL.
A clear checklist for headlines, authors, images and dates—without invented ratings or special “AI schema”.