A business owner sees a burst of requests labelled as an AI crawler and asks whether its website has started appearing in recommendations. The logs might show a request for robots.txt, an old URL redirecting or a successful response from the cache. Each event can answer a technical question about delivery. To answer the owner’s question, you need evidence from the answer surface or a report that measures citations.
Read the request, the response and the layer together. Separate claimed identity from verification, then decide which events count as page responses for the period you are reviewing. The AI Visibility access checklist introduces crawl access. This walkthrough focuses on counting what was actually observed and presenting those counts without implying an answer outcome. AI Visibility site access checklist
Define the time window and the log source
Start with the question you can answer. For example, did a verified search crawler request the public pricing page after Monday’s release, and what response did the edge send? Record the full host and path, the time window in UTC, the named requester and the layer you will inspect. Add the query string where it materially affects the response. If you plan to aggregate parameter variants later, retain the exact request separately so you can trace a discrepancy.
Use records you are authorised to see. Identify whether they come from an edge service, a reverse proxy or the origin application. Check which hosts and routes are covered, which fields are retained and whether you have sampled or complete data. NGINX documents configurable access log formats, so the fields available depend on the format the operator chose. Cloudflare’s HTTP request dataset includes distinct edge and origin response fields. Read the schema for your own records before giving a field a meaning. NGINX access log reference · Cloudflare HTTP request log fields
| Log evidence | Classification | What it supports |
|---|---|---|
| Request for robots.txt | Policy-file request | The requester asked for the crawler policy |
| GET with 301 and a Location | Redirect response | The old URL sent a redirect to the recorded destination |
| GET with 200 and expected body | Page response | The tested layer delivered content for the request |
| Conditional GET with 304 | Revalidation | The request did not receive a fresh body |
| HEAD with 200 | Metadata response | The request succeeded without a response body |
Before filtering, separate claimed identities from verified ones. OpenAI documents different requester purposes, including search crawling, training and user-triggered visits. Google publishes verification methods and ranges by category. Where available, use vendor-supported evidence and the connecting address at a trusted layer. Keep an unverified row under its claimed label. Counting it as a confirmed vendor visit would change the meaning of the report. OpenAI crawler documentation · Google requester verification guide
Reconstruct the request before counting rows
For one event, collect the method, path, response code, content type, request ID and bytes where your schema defines them. Check whether a cache status or origin status is present. If the request redirects, record the Location when available and look for a separate request to the destination. An old-address record tells you the redirect was returned. A separate destination response is needed to establish what was delivered there. Use timestamps and trusted correlation fields to connect related observations, and note when the relationship is only inferred.
A 200 on a GET gives you evidence about a response, but you may still need the response content or a control capture to establish whether it contained the intended page. A challenge, error template or login response can look different from the expected article. Conversely, a conditional GET with 304 can be normal when a client revalidates a stored representation. A HEAD request can return 200 with no response body by design. Keep the method and status together when categorising records. HTTP semantics, RFC 9110 · Google HTTP status guidance
Correlate edge and origin records using the request ID your stack forwards where supported. Preserve the identity of each layer. If the edge served a cached copy, there may be no origin request for that visit. If the edge revalidated against the origin, the statuses at the two layers can differ. Use the documented cache and origin fields instead of assuming a missing origin event means the page was blocked. If you have only sampled events or incomplete retention, describe that limit alongside your counts. Cloudflare HTTP request log fields
Count six requests in a controlled local log
On 6 November 2026, a local edge and origin served a test guide. Six real HTTP requests used the fictional user-agent label LabSearchCrawler. The fixture recorded the response seen by the client and wrote separate edge and origin rows with request IDs. It produced eleven log rows for six requests. The lab label was unverified and the experiment made no vendor request.
Request 1 fetched robots.txt. Request 2 received a 301 from the old route to the guide. Request 3 requested the guide directly and got HTTP 200 with the expected content. Request 4 requested the guide again and got the same content from the cache. That fourth request appeared only in the edge log. Request 5 sent a conditional GET with the matching ETag and got 304 with no body. Request 6 used HEAD and got 200 with no body. Two requests delivered the expected guide body to the client.
If you count all eleven layer records, you overstate request volume. If you count only origin rows, you miss the cached visit. If you count every HTTP 200, you include the policy file and the HEAD response. Count from the layer that answers the question, document your classification and use the other layer to explain delivery. The exact numbers are from this fixture. Your production reports should use the real provider’s field meanings.
For the supplied fixture record, the following filter shows one row per edge request and preserves the fields that distinguish the outcomes. It uses the fixture schema. Adapt the field names only after checking the documentation for the export you are using.
jq -r '.observations.fixture.logs[] | select(.layer == "edge")
| [.requestId, .method, .path, .status, .cache, .bodyBytes] | @tsv' \
09-crawler-log-evidence.json
A response size is a useful check when comparing similar requests. Its definition matters. Some logs report the body, while others may count transferred bytes or record the size at a different layer. In the fixture, the logged body size can be compared directly with the client read. In production, check the field definition and consider compression or partial responses before comparing sizes. For a high-value page, keep a sample of the expected content or a stable marker from a permitted response check. NGINX access log reference · Cloudflare HTTP request log fields
To reproduce the sequence on an authorised staging site with a known cacheable page and a test redirect, save the edge and origin records during a narrow time window. Request robots.txt, request the old address without automatically following the redirect, then fetch the final page twice. If the page supplies an ETag, send that exact validator in a conditional GET. Run HEAD as a separate check. Save the client headers and compare them with the logged outcomes. A production cache can behave differently from this fixture, so observe each result before assigning a category.
Report fetches, citations and recommendations separately
A fetch is an event your infrastructure recorded. An index or retrieval result is evidence from a platform. A citation is an observed source reference. A recommendation is a judgment expressed in an answer, which might use a citation or might be phrased without one. These categories call for different evidence. A log can support the statement that a verified requester fetched a page. It cannot reveal what an assistant said in a particular answer.
Microsoft’s AI Performance documentation describes a report for citations across supported AI surfaces and explains that citation counts do not show the placement or role of a source within a particular answer. Where an authorised report is available, record its surface, date range and coverage separately from server traffic. For a recommendation question, use a permitted answer observation that preserves the question, date, surface and wording. This guide used no live chatbot test. Bing Webmaster Tools AI Performance
Microsoft’s official Instagram reel in anniversary coverage presents Copilot as a product. It is useful context for the user-facing side of the distinction. For reporting on a specific website, use server records for delivery, a supported platform report for measured citations and a recorded answer for the actual recommendation. Coverage with the official Microsoft reel
You can also record a separate ordinary browser check. Open the exact page in a clean browser and confirm the expected content appears. Keep the browser’s response and rendering observation beside the log sample without assigning it to the crawler. That can help rule out a missing page for ordinary users while leaving the crawler-specific outcome open. When the issue is a particular blocked request, return to the matched records for that requester.
Make the log summary reproducible
Use a Crawler visit log record with the time window, log layer, coverage, identity method, path filters, classification rules and counts. Keep policy-file requests, page requests, redirects and errors as distinct categories. If you aggregate exact paths into content groups, document the rule and keep a sample of underlying records. Include successful body-bearing responses separately when that detail is available. For shareable reports, minimise request details that can identify a person or reveal credentials.
Compare like periods when assessing a change. If last week’s export covered only the origin and this week’s includes the edge, a higher count may reflect coverage. Note deployments, cache changes and traffic filters before attributing a change to crawl demand. If a verified request stops appearing, check the time range and retention before diagnosing access. A clearly stated observation window makes absence interpretable.
When you report unique pages requested, describe the grouping separately from total requests. Two fetches of the same URL count as two request events and one unique URL within a window. A redirect and its destination are separate requests, and may represent one navigation if the correlation supports that interpretation. Choose the denominator before putting the count in a chart.
If a record lacks the method or status, place it in an unknown category. Keep that count visible so the report shows how much of the traffic could be classified.
Give the owner a sentence that matches the evidence. For example, the edge returned the expected page to two verified requests during the defined window. If a separate report shows a citation, record that result under its own source and coverage. Keep a pending action for any recommendation claim that still needs an answer observation.
AI Visibility is published by Lunasol, which offers GEO and AEO services. The worked observations come from controlled local HTTP and server logs. About AI Visibility



