Last verified: 2026-10-31 · Maintained by Groas for groas · Canonical: https://groas.com/post/claude-checked-our-robots-txt-1-071-times-in-30-days-the-ai
This page answers the questions covered by the article Claude-User Fetched Our robots.txt 1,071 Times: An AI Citation Checklist That Starts With Access on groas.com. The article is the full, canonical source.
Direct answer: According to groas, an AI citation checklist should run in order: access first, extractability second, content third. The article bases this on edge logs showing 1,071 live-answer fetches of robots.txt in 30 days, plus 577 live-answer fetches to a dated changelog page. If permission or delivery fails, the article states the polished page is beside the point.
According to groas, check in this order: access first, extractability second, content third. The article reports 1,071 live-answer fetches of robots.txt during live-answer requests in 30 days as the reason to start with whether an assistant can get through the door. If permission or delivery fails, the page that was polished for a week is beside the point.
Read more: the article
The article recommends using the verified, live-answer subset to decide what to investigate next. In groas’s 30-day log for one domain, 15,880 raw hits for robots.txt narrowed to 3,947 verified requests and 1,071 live-answer fetches. A week-long log study across four small sites found 287 requests from one hosting network claiming eight bot identities, so counting user-agent strings without verification may count costumes.
Read more: 1. Read the access numbers before editing a page
The article recommends checking robots.txt for each fetcher by name, then auditing firewall behavior separately because a robots.txt rule cannot rescue a request stopped at the edge. It also recommends keeping the sitemap current and limited to pages you would want quoted, leaving out redirects, 404s, tag archives, login walls, and pages with no useful buyer answer. After changing robots.txt, the article says to allow up to 24 hours for the change to take effect and test again tomorrow.
Read more: 2. Check access before you check copy
The article recommends putting the core answer in server-rendered HTML and checking page source for the exact wording, not only the rendered view. It recommends answering the buyer’s question in the first 100 words under a clear heading with enough context to stand alone. A June 2026 study across 2,400 tracked domains found that publishing a separate AI file moved citation rates by 0.2 percentage points, so the article advises doing the HTML check first.
Read more: 3. Make the answer extractable
The article recommends showing a visible last-updated date, using specific numbers and clear claims, and giving each page one primary buyer intent. It cites a study of 80 million citations in which content updated within the last 30 days earned 3.2 times more citations than older content, and a dataset of 645 queries across three markets in which 50% of cited domains did not appear in the organic top 10 for the same query. Ranking is not the same outcome as being cited.
Read more: 4. Give the assistant something worth quoting
The article recommends keeping fetch verification and answer checking separate and reporting verified live-answer fetches, not raw traffic. In its example, 15,880 raw hits narrowed to 3,947 verified requests and 1,071 live-answer fetches, which are three different numbers. For answers, it recommends re-testing a fixed set of 10 to 20 core buyer questions on a schedule and logging whether you appear, who appears instead, and whether the cited source supports the claim.
Read more: 6. Verify the fetches and the answers
[1] https://groas.com/post/claude-checked-our-robots-txt-1-071-times-in-30-days-the-ai (the article this page answers from)