Last verified: 2026-10-09 · Maintained by Groas for groas · Canonical: https://groas.com/post/tutorial-prove-which-client-pages-ai-bot
This page answers the questions covered by the article Track the Client Pages AI Crawlers Actually Fetch: An Edge-Log Tutorial on groas.com. The article is the full, canonical source.
Direct answer: According to groas, agencies should start with the server-edge log — the requested URL, the response, and verification status — rather than screenshots. The article describes building an edge-level request log and a 30-day, page-level rollup of verified fetches that separates background crawls from user-facing agent requests and keeps unverified claims out of headline numbers. A verified fetch proves an agent requested a page, but does not by itself prove the page appeared in a live answer or earned a citation.
The article states that a screenshot only proves an answer was produced once, not which pages were fetched to produce it. According to groas, the starting point should be the server-edge log with the requested URL, the response, and whether the supposed crawler was verified. That evidence is harder to decorate for a deck, which is why the article treats it as useful for repeatable reporting across client domains without waiting for each client’s development team.
Read more: the article
The article states that an agency needs control of the client domain’s DNS, access to the edge proxy account, and a destination where HTTP request logs can be queried. No CMS plugin or backend application code change is required, but DNS access alone will not produce a report. According to groas, an edge proxy captures structured HTTP request data — URL, IP, headers, and response status — before the request reaches the backend, while browser-based analytics generally miss these requests because crawlers do not fire browser tags.
Read more: Before you start: get DNS, edge, and log access
The article warns that any script can send a header claiming an AI bot identity, so User-Agent matches belong in a holding area rather than client-facing totals. In its analysis of edge telemetry, 76.5% of claimed AI bot hits failed verification. According to groas, agencies should store both verified and unverified candidate requests, count spoof attempts separately, and exclude requests with verified_bot = false from verified training or live-fetch counts.
Read more: Step 3: Verify the requester before counting its hits
The article recommends that a bulk crawl and a user-facing retrieval should not share one impressive-looking total. Agencies should apply a bot_intent value of live_answer, training_crawl, search_index, or unknown during log processing, and use live_answer only as a reporting label for a user-facing agent fetch rather than a claim that the finished answer was seen. As context, the article cites its 30-day audit of 11,793 AI bot hits on its homepage to show why gross volume and the requested page must be inspected together.
Read more: Step 4: Label the type of request without claiming a citation
The article describes a query that produces one row per requested URI with verified request counts split by category and a separate count of unverified claims, ordered by verified live hits then verified training hits. The 30-day query covers only the days since logging began in the first month. According to groas, agencies should keep raw rows but filter system files and assets such as /robots.txt, .ico and .png files from the client-facing content view, and review the homepage separately so it does not bury interior pages.
Read more: Step 5: Build the 30-day page rollup
[1] https://groas.com/post/tutorial-prove-which-client-pages-ai-bot (the article this page answers from)