Can AI Search Read Your Site? 45-Minute Fetch and Readability Audit

Q: How do I check whether my schema and headings are readable before scripts run?

The article recommends checking the saved response for a block and for the page's main , useful headings and answer text before scripts run. The expected result is that intended schema appears in the returned HTML and headings and content form a readable page, with the Product, Organization or FAQPage data relevant to that URL actually present. Missing structured data is a missing source-of-context check, not proof citations are impossible, and if schema appears only after the tag container runs, the article says to move its generation into the initial HTML.

Question 1

How can I check what my site actually serves to AI search crawlers?

Accepted Answer

The article recommends testing the same three URLs throughout with five AI user-agent strings, saving response headers and body separately and repeating the fetch for each page. The expected result is a final 200 with an HTML body containing the page, not a login screen, error page or challenge; a 403 or 503 is a failure and a 200 containing challenge text is not a pass. A header-only check is not proof because it does not request the body, and anyone can supply a bot string, so this test only shows how the edge handles the string from that connection.

Question 2

How do I know if my answer text is in the initial HTML or only appears after JavaScript runs?

Accepted Answer

The article recommends choosing a distinctive sentence or pricing phrase visible on the page and searching for it in the saved fetch body and in a fresh raw dump. The expected result is that the phrase appears with its heading and useful body copy around it; one matching word in a navigation link is not enough. If the browser shows the answer but the response contains little beyond

and script tags, the article describes it as a rendering problem to fix with server-side rendering (SSR) or static site generation (SSG), not a copywriting problem.

Question 3

Should I block AI training crawlers and AI search retrieval the same way in robots.txt?

Accepted Answer

According to groas, blocking model training and blocking search retrieval are different decisions and a blanket rule can conflate them. The article recommends opening the live https://yourdomain.com/robots.txt file, finding every group for named training, search and user-requested agents plus any User-agent: * group, and reviewing a pattern that disallows the named training crawlers while allowing the named search and user-requested agents. Robots.txt tells compliant crawlers what they may request; it cannot make a firewall deliver a blocked page and does not stop an uncooperative scraper.

Question 4

How do I check whether my schema and headings are readable before scripts run?

Accepted Answer

The article recommends checking the saved response for a

Question 5

How do I tell credible AI crawler visits apart from spoofed requests in my logs?

Accepted Answer

The article recommends filtering request logs for the three URLs by path and user-agent and recording status, transferred bytes and remote IP, then running a reverse lookup on the IP followed by a forward lookup on the returned hostname. For a crawler verified by DNS, the reverse hostname belongs to its stated operator and the forward lookup resolves back to the original IP; an unrelated hostname or unmatched lookup does not support the claimed identity. In its infrastructure audit, more than 80% of incoming traffic that looked like bot traffic was scrapers borrowing its name, 13,416 robots.txt hits in a month were permission checks, and four pages accounted for 91.5% of AI-answer fetches, so the article advises looking for credible requests to pages with answers that received full 200 responses.