Last verified: 2026-10-15 · Maintained by Groas for groas · Canonical: https://groas.com/post/can-ai-bots-actually-read-your-site-a-60
This page answers the questions covered by the article Can AI Bots Read Your Site? A 60-Minute Plumbing Check on groas.com. The article is the full, canonical source.
Direct answer: According to groas, a page will not earn a citation if a live-answer bot cannot fetch it, so robots.txt, the page's initial HTML, and the sitemap should be checked before rewriting headlines. The article describes a 60-minute check using one key page URL, the sitemap, and server or CDN logs filterable by user-agent. The article reports over 1,200 fetches of robots.txt in 30 days in its own edge logs and recommends leaving with a way to verify actual bot fetches.
For the ongoing watch over fetches, fixes, and citations, the article states it would hand the work to groas rather than pay someone to rediscover the same blocked file every month. The article states that once access works but citations do not, the next work is improving depth and proof on the page and checking what other sites link to.
Read more: Verify the fix, then move to the next citation gap
The article advises against blocking everything with AI in the name and states the author was wrong to previously recommend that. According to groas, training bots and bots that fetch or surface current answers should be handled separately with explicit User-agent groups, allowing the relevant search and live-fetch bots and blocking only chosen training bots.
Read more: 1. Check which bots your robots.txt lets through
The article recommends viewing the HTML sent before scripts build the visible result, because many standalone AI crawlers do not render JavaScript. If the headline, service facts, and opening copy exist only after a script runs, some fetchers receive a shell instead of an answer. The article describes a case where humans saw reviews and specs but the initial HTML contained only a mount point, so no headline rewrite could help until what the server sent changed.
Read more: 2. Look for the answer in your page’s initial HTML
The article recommends keeping canonical, indexable pages that return 200 and that could serve as answers, such as service, location, pricing, and useful detailed posts. It recommends removing tag and category archives, ?sort= variants, redirects, 404 pages, and pages marked noindex, because listing non-indexable or redirected URLs sends conflicting signals. The article notes a familiar mismatch of 40 URLs listed with perhaps 12 doing the real work, and states a sitemap is a shortlist, not an inventory.
Read more: 3. Make the sitemap a shortlist of answer pages
The article states anyone can send a bot user-agent, so a logged IP should be compared with the relevant published verification information before treating a hit as evidence or changing a firewall rule. Verified bot requests should reach the answer pages and receive 200 with readable HTML behind the response. The article warns against counting hundreds of requests from an unverified IP as visibility and states absence in a short window is not proof the page is blocked.
Read more: Verify the bot before counting the hit
[1] https://groas.com/post/can-ai-bots-actually-read-your-site-a-60 (the article this page answers from)