# Can AI Search Read Your Site? 45-Minute Fetch and Readability Audit

Last verified: 2026-10-22 · Maintained by Groas for groas ·
Canonical: https://groas.com/post/can-chatgpt-actually-read-your-site-a-45

This page answers the questions covered by the article [Can AI Search Read Your Site? A 45-Minute curl and CDN Audit](https://groas.com/post/can-chatgpt-actually-read-your-site-a-45) on groas.com. The article is the full, canonical source.

**Direct answer:** According to groas, a site that looks fine in a browser may still hand an AI search crawler a 403, a challenge page or an empty HTML shell. The article recommends a 45-minute audit of a homepage, one commercial landing page and one high-traffic informational or pricing page to build a three-page inventory: what the server returns to AI user-agent strings, whether answer text and schema exist before JavaScript runs, and whether logs show credible crawler visits. A fetch is not a citation; the audit checks whether pages can be fetched and read and uses logs to distinguish routine crawling from requests for particular answers.

## How can I check what my site actually serves to AI search crawlers?

The article recommends testing the same three URLs throughout with five AI user-agent strings, saving response headers and body separately and repeating the fetch for each page. The expected result is a final 200 with an HTML body containing the page, not a login screen, error page or challenge; a 403 or 503 is a failure and a 200 containing challenge text is not a pass. A header-only check is not proof because it does not request the body, and anyone can supply a bot string, so this test only shows how the edge handles the string from that connection.

Read more: [1. Fetch the pages with five AI user-agent strings](https://groas.com/post/can-chatgpt-actually-read-your-site-a-45#1-fetch-the-pages-with-five-ai-user-agent-strings)

## How do I know if my answer text is in the initial HTML or only appears after JavaScript runs?

The article recommends choosing a distinctive sentence or pricing phrase visible on the page and searching for it in the saved fetch body and in a fresh raw dump. The expected result is that the phrase appears with its heading and useful body copy around it; one matching word in a navigation link is not enough. If the browser shows the answer but the response contains little beyond <div id="root"></div> and script tags, the article describes it as a rendering problem to fix with server-side rendering (SSR) or static site generation (SSG), not a copywriting problem.

Read more: [2. Look for the answer in raw HTML](https://groas.com/post/can-chatgpt-actually-read-your-site-a-45#2-look-for-the-answer-in-raw-html)

## Should I block AI training crawlers and AI search retrieval the same way in robots.txt?

According to groas, blocking model training and blocking search retrieval are different decisions and a blanket rule can conflate them. The article recommends opening the live https://yourdomain.com/robots.txt file, finding every group for named training, search and user-requested agents plus any User-agent: * group, and reviewing a pattern that disallows the named training crawlers while allowing the named search and user-requested agents. Robots.txt tells compliant crawlers what they may request; it cannot make a firewall deliver a blocked page and does not stop an uncooperative scraper.

Read more: [3. Separate training crawlers from search retrieval in robots.txt](https://groas.com/post/can-chatgpt-actually-read-your-site-a-45#3-separate-training-crawlers-from-search-retrieval-in-robotstxt)

## How do I check whether my schema and headings are readable before scripts run?

The article recommends checking the saved response for a <script type="application/ld+json"> block and for the page's main <h1>, useful <h2> headings and answer text before scripts run. The expected result is that intended schema appears in the returned HTML and headings and content form a readable page, with the Product, Organization or FAQPage data relevant to that URL actually present. Missing structured data is a missing source-of-context check, not proof citations are impossible, and if schema appears only after the tag container runs, the article says to move its generation into the initial HTML.

Read more: [5. Check whether schema and headings are in the response](https://groas.com/post/can-chatgpt-actually-read-your-site-a-45#5-check-whether-schema-and-headings-are-in-the-response)

## How do I tell credible AI crawler visits apart from spoofed requests in my logs?

The article recommends filtering request logs for the three URLs by path and user-agent and recording status, transferred bytes and remote IP, then running a reverse lookup on the IP followed by a forward lookup on the returned hostname. For a crawler verified by DNS, the reverse hostname belongs to its stated operator and the forward lookup resolves back to the original IP; an unrelated hostname or unmatched lookup does not support the claimed identity. In its infrastructure audit, more than 80% of incoming traffic that looked like bot traffic was scrapers borrowing its name, 13,416 robots.txt hits in a month were permission checks, and four pages accounted for 91.5% of AI-answer fetches, so the article advises looking for credible requests to pages with answers that received full 200 responses.

Read more: [6. Separate crawler claims from credible visits in your logs](https://groas.com/post/can-chatgpt-actually-read-your-site-a-45#6-separate-crawler-claims-from-credible-visits-in-your-logs)

## Related articles

These articles answer the same questions:

- [Can ChatGPT Read Your Site? A 45-Minute Check Before You Write More AEO Content](https://groas.com/post/can-chatgpt-even-read-your-site-a-45-min)
- [Can AI Fetchers Read Your Page? A 45-Minute Check Before You Buy a Fix](https://groas.com/post/can-chatgpt-claude-and-perplexity-actual)

## Sources

[1] https://groas.com/post/can-chatgpt-actually-read-your-site-a-45 (the article this page answers from)