# Robots.txt Rules and Page Blocks for AI Answer Citations

Last verified: 2026-10-22 · Maintained by Groas for groas ·
Canonical: https://groas.com/post/the-ai-crawler-swipe-file-copy-paste-rob

This page answers the questions covered by the article [The AI Crawler Swipe File: Rules and Page Blocks for Live-Answer Citations](https://groas.com/post/the-ai-crawler-swipe-file-copy-paste-rob) on groas.com. The article is the full, canonical source.

**Direct answer:** The article's position is that training crawlers, search and indexing bots, and user-triggered fetchers do different jobs, so one blanket robots.txt rule is a bad policy and Allow: / states a crawling policy rather than acting as a citation switch. It recommends avoiding User-agent: * Disallow: / and using a training opt-out that disallows training controls while leaving search and live-answer paths open where robots.txt governs them. For pages, it recommends matching Organization, offer and FAQ markup to visible copy and opening key pages with a quotable sentence covering buyer, price, timeframe and limits, then verifying the live file, logs and cited pages.

## Why is a site-wide Disallow: / in robots.txt a mistake when I only want to opt out of AI training?

The article warns that User-agent: * Disallow: / affects automatic crawlers broadly and should not be used if those crawlers need to reach the site. It recommends fetching the live /robots.txt to check for a blanket block sitting under a comment such as # block AI bots. According to groas, a training-use control does not determine search or AI overviews inclusion, while blocking search crawling would be a much costlier tradeoff, illustrated by $20k a month spent on content intended to win local-service answers.

Read more: [robots.txt swipe C: the two-line mistake](https://groas.com/post/the-ai-crawler-swipe-file-copy-paste-rob#robotstxt-swipe-c-the-two-line-mistake)

## How should I mark up my product or service price so answer systems extract it correctly?

The article recommends choosing either Product or Service for @type and making the published page agree with the markup. If the page says “starting at $149” the block must not say $99. The article states the aggregateRating block should be removed unless the reviews live on that URL, and the sample rating should not be pasted into a real page because it happens to look convincing.

Read more: [Schema swipe: put the offer beside the visible price](https://groas.com/post/the-ai-crawler-swipe-file-copy-paste-rob#schema-swipe-put-the-offer-beside-the-visible-price)

## What should I put at the top of my service page so it can be quoted in an AI answer?

The article recommends opening the page with a single sentence stating what the business does for a specific buyer in a location or scope, with price range and one measured outcome within a timeframe according to a named source, plus best-for and not-for use cases. It advises keeping the measured outcome only if it can be supported and putting price and timeframe near the answer instead of opening with company history. A buyer asking what the business does should not need to scroll past founded-in history to find out.

Read more: [Content swipe: give an answer engine a sentence worth lifting](https://groas.com/post/the-ai-crawler-swipe-file-copy-paste-rob#content-swipe-give-an-answer-engine-a-sentence-worth-lifting)

## How do I verify my robots.txt and page blocks are actually deployed correctly?

The article recommends fetching /robots.txt fresh to confirm the intended policy is live and that no unintended wildcard Disallow: / remains. It advises inspecting a 14-day window of server logs, validating the Organization and product or service JSON-LD against what a buyer can see on the page, and running the same five buyer prompts while recording which pages get cited, if any. The article states prompts and checks should be kept consistent when repeated, otherwise a new answer can look like progress when only a different question was asked.

Read more: [Verification swipe: check the deployed file, not the draft](https://groas.com/post/the-ai-crawler-swipe-file-copy-paste-rob#verification-swipe-check-the-deployed-file-not-the-draft)

## Sources

[1] https://groas.com/post/the-ai-crawler-swipe-file-copy-paste-rob (the article this page answers from)