Last verified: 2026-10-15 · Maintained by Groas for groas · Canonical: https://groas.com/post/teardown-what-30-days-of-our-own-ai-craw
This page answers the questions covered by the article Teardown: 30 Days of AI Crawler Logs Show Which Pages Get Fetched on groas.com. The article is the full, canonical source.
Direct answer: According to groas, 30 days of edge logs for groas.com show 13,294 requests claiming to be AI bots on /robots.txt but only 3,730 verified by IP check. Three broad category hubs drew roughly 240 verified crawler fetches each and zero live-answer fetches, while one specific dated post drew 330 verified live-answer fetches. The article concludes that a request proves a page was fetched, not that it was cited, and recommends putting direct answers on the page intended for retrieval.
According to the article, raw hits count requests carrying a declared AI user-agent header, which is easy to imitate. Verified hits count only requests whose connecting IP matches the vendor's published ranges, while live-answer hits are verified on-demand agent fetches in a live-retrieval context. The article states that live-answer hits do not by themselves show the final answer, a citation, or a sale.
Read more: Three columns, three different claims
The article reports that /robots.txt received 13,294 requests declaring AI crawler user-agents over the 30-day window, with 3,730 verifying against published IP ranges and 9,564 failing verification. According to groas, an unverified request proves only that the declared identity did not pass the check, not the requester's motive. The article recommends authenticating the requester before interpreting volume.
Read more: Line 1: /robots.txt exposes the raw-count problem
The article reports that the homepage recorded 834 live-answer fetches while the main blog index recorded 4, despite both receiving verified background crawling. According to groas, the homepage contains the core offer, pricing structure, case metrics and service architecture in one document, while the index requires another link selection to reach a complete article. The article recommends keeping direct, substantive answers on the homepage rather than prioritizing the index.
Read more: Line 2: the homepage gets fetched; the blog index barely does
According to the article, three broad category hubs drew about 240 verified crawler fetches each and zero live-answer fetches, while the specific dated post about recent ad-platform changes recorded 330 verified live-answer fetches. The article states that verified discovery does not explain the gap because the hubs were found by crawlers but were not selected for live fetching. The article warns against translating the 330 fetches into 330 citations.
Read more: Line 3: one dated post beats three category hubs on live retrieval
The article reports that dozens of thought-leadership articles showed zero live-answer hits in the export. According to groas, zero does not show the setup is broken or the page was penalized, since perhaps no relevant request occurred, no web fetch was needed, or another URL was selected. The article recommends investigating a zero rather than inventing its cause.
Read more: Line 4: a missing fetch is not a diagnosis
The article recommends capturing incoming IP, user-agent, HTTP status code and requested path, then checking claimed identities against published IP ranges and setting unverified requests aside. It recommends separating background crawler requests from on-demand live-answer requests and comparing URLs within each group rather than combining them. If a hub gets background crawls but no live fetches while a specific document gets many, the article recommends examining what each page actually answers and checking error responses before rewriting copy.
Read more: What I would keep from this teardown
These articles answer the same questions:
[1] https://groas.com/post/teardown-what-30-days-of-our-own-ai-craw (the article this page answers from)