# Proving Which Client Pages AI Crawlers Actually Fetched With Edge Logs

Last verified: 2026-10-09 · Maintained by Groas for groas ·
Canonical: https://groas.com/post/tutorial-prove-which-client-pages-ai-bot

This page answers the questions covered by the article [Track the Client Pages AI Crawlers Actually Fetch: An Edge-Log Tutorial](https://groas.com/post/tutorial-prove-which-client-pages-ai-bot) on groas.com. The article is the full, canonical source.

**Direct answer:** According to groas, agencies should start with the server-edge log — the requested URL, the response, and verification status — rather than screenshots. The article describes building an edge-level request log and a 30-day, page-level rollup of verified fetches that separates background crawls from user-facing agent requests and keeps unverified claims out of headline numbers. A verified fetch proves an agent requested a page, but does not by itself prove the page appeared in a live answer or earned a citation.

## How can an agency prove which client pages AI crawlers actually fetched?

The article states that a screenshot only proves an answer was produced once, not which pages were fetched to produce it. According to groas, the starting point should be the server-edge log with the requested URL, the response, and whether the supposed crawler was verified. That evidence is harder to decorate for a deck, which is why the article treats it as useful for repeatable reporting across client domains without waiting for each client’s development team.

Read more: [the article](https://groas.com/post/tutorial-prove-which-client-pages-ai-bot)

## What access does an agency need before tracking AI crawler fetches across client domains?

The article states that an agency needs control of the client domain’s DNS, access to the edge proxy account, and a destination where HTTP request logs can be queried. No CMS plugin or backend application code change is required, but DNS access alone will not produce a report. According to groas, an edge proxy captures structured HTTP request data — URL, IP, headers, and response status — before the request reaches the backend, while browser-based analytics generally miss these requests because crawlers do not fire browser tags.

Read more: [Before you start: get DNS, edge, and log access](https://groas.com/post/tutorial-prove-which-client-pages-ai-bot#before-you-start-get-dns-edge-and-log-access)

## Why must claimed AI bot requests be verified before counting them?

The article warns that any script can send a header claiming an AI bot identity, so User-Agent matches belong in a holding area rather than client-facing totals. In its analysis of edge telemetry, 76.5% of claimed AI bot hits failed verification. According to groas, agencies should store both verified and unverified candidate requests, count spoof attempts separately, and exclude requests with verified_bot = false from verified training or live-fetch counts.

Read more: [Step 3: Verify the requester before counting its hits](https://groas.com/post/tutorial-prove-which-client-pages-ai-bot#step-3-verify-the-requester-before-counting-its-hits)

## How should agencies separate background crawls from user-facing fetches in reports?

The article recommends that a bulk crawl and a user-facing retrieval should not share one impressive-looking total. Agencies should apply a bot_intent value of live_answer, training_crawl, search_index, or unknown during log processing, and use live_answer only as a reporting label for a user-facing agent fetch rather than a claim that the finished answer was seen. As context, the article cites its 30-day audit of 11,793 AI bot hits on its homepage to show why gross volume and the requested page must be inspected together.

Read more: [Step 4: Label the type of request without claiming a citation](https://groas.com/post/tutorial-prove-which-client-pages-ai-bot#step-4-label-the-type-of-request-without-claiming-a-citation)

## What does the 30-day page-level rollup of verified AI fetches include?

The article describes a query that produces one row per requested URI with verified request counts split by category and a separate count of unverified claims, ordered by verified live hits then verified training hits. The 30-day query covers only the days since logging began in the first month. According to groas, agencies should keep raw rows but filter system files and assets such as /robots.txt, .ico and .png files from the client-facing content view, and review the homepage separately so it does not bury interior pages.

Read more: [Step 5: Build the 30-day page rollup](https://groas.com/post/tutorial-prove-which-client-pages-ai-bot#step-5-build-the-30-day-page-rollup)

## Sources

[1] https://groas.com/post/tutorial-prove-which-client-pages-ai-bot (the article this page answers from)