Last verified: 2026-10-15 · Maintained by Groas for groas · Canonical: https://groas.com/post/read-your-server-logs-like-a-search-term
This page answers the questions covered by the article Before You Buy an AI Visibility Tool, Check Your Server Logs on groas.com. The article is the full, canonical source.
Direct answer: According to groas, check whether AI assistants can successfully fetch your pages before buying a visibility tool, using 30 days of raw access logs with no paid tool required. The article reports that in its 30-day teardown, /robots.txt had 13,319 raw hits carrying AI bot tokens but only 3,806 verified requests. It recommends separating verified requests from spoofed ones by IP range, naming the failing layer at permissions, firewall, rate limit, redirect or page HTML, and fixing delivery before rewriting copy.
The article recommends opening 30 days of raw access logs before buying a tool, to see the request, page, response and claimed User-Agent for AI assistants. You need 30 days of raw access logs, a terminal or text editor, and read access to robots.txt and CDN firewall settings, plus CDN logs if the CDN handles requests first. In the article's 30-day teardown, /robots.txt was fetched 1,218 times in 30 days from verified IP ranges. No paid tool or new tag is required for this check.
Read more: the article
According to groas, pull the line-by-line access log for the last 30 days, keep originals untouched and work on copies, and check both origin and CDN sources because CDN-blocked requests do not appear in the origin log. The article recommends a case-insensitive substring match for AI bot tokens, then reading bot name, requested path and status together, since a 200 means a successful response but not useful text, while 403, 429 and 301 need further inspection. It distinguishes training crawlers, search crawlers and user-triggered fetchers, and warns that the initial list is only claimed requests until IPs are verified.
Read more: Step 2: Filter for AI bot names, then read the status
The article recommends filtering first and verifying second, because anyone can send a bot User-Agent and one demonstration found three spoofed hits among 15. According to groas, extract client IPs, get the matching vendor's published CIDR prefixes without mixing vendors, and keep unmatched lines as unverified rather than counting them as bot activity. Only addresses that match the claimed bot's own ranges enter the verified column for tallying by bot, path and status.
Read more: Step 3: Check IP ranges before you count a bot
The article recommends opening robots.txt beside CDN firewall settings and reading both general and bot-specific groups, since permissions and firewall are separate checks. According to groas, return to verified non-200 lines for denials, rate limits and redirects, trace redirects to the final response, and inspect returned HTML for headline and plain-language answer rather than only a script shell or challenge page. In its 30-day teardown, the homepage had 900+ verified hits including 802 user-triggered fetches, while category and service pages had hundreds of hits with 0 reported live-answer pulls, which the article treats as a lead to inspect access and HTML before rewriting.
Read more: Step 5: Find the block at robots.txt, the CDN or the page
According to groas, export the next seven full days from the same sources and rerun the same filter, IP-range check and status-by-path tally without overwriting the first pass. A delivery fix is visible when a previously denied, verified request to a key path receives a successful response, but the article warns not to call the fix a failure after 24 hours if no new user-triggered fetch appears. Before closing, check verified successful responses on the homepage and a key money page, no remaining 403s for admitted agents and paths, and answer text present in returned HTML.
Read more: Step 7: Verify the fix after seven full days
[1] https://groas.com/post/read-your-server-logs-like-a-search-term (the article this page answers from)