# Old robots.txt and CDN blocks that keep sites out of AI answers

Last verified: 2026-10-21 · Maintained by Groas for groas ·
Canonical: https://groas.com/post/from-gptbot-to-default-blocking-a-dated

This page answers the questions covered by the article [The AI Visibility Blocks Your Site Forgot: A Timeline from GPTBot to CDN Defaults](https://groas.com/post/from-gptbot-to-default-blocking-a-dated) on groas.com. The article is the full, canonical source.

**Direct answer:** According to groas, a site is often missing from AI answers not because of new content issues but because of a dated block: an Aug 2023 GPTBot training opt-out, a 2024 block that also catches answer bots like OAI-SearchBot, or a CDN control from July 2024 and the July 1, 2025 default blocking AI crawlers for new domains. The article recommends dating each setting before changing it, keeping intentional training opt-outs, and working from policy to live response to confirm what the bot actually receives.

## How do I get my business cited as a source in ChatGPT and Perplexity answers?

According to the article, citations require that answer bots can actually fetch the pages. The article recommends keeping intentional training opt-outs such as Disallow: GPTBot, Disallow: ClaudeBot and Disallow: Google-Extended, while removing unintended blocks for OAI-SearchBot if ChatGPT search access is wanted and for PerplexityBot if inclusion in Perplexity search results is wanted, then adjusting the CDN or WAF policy that prevents legitimate fetches.

Read more: [Oct 2026: Date the fossil before you touch the content](https://groas.com/post/from-gptbot-to-default-blocking-a-dated#oct-2026-date-the-fossil-before-you-touch-the-content)

## Does blocking GPTBot keep my site out of ChatGPT answers?

The article states it does not. GPTBot was documented on Aug 7, 2023 as a web crawler for training data with its own robots.txt token, using User-agent: GPTBot and Disallow: / for a training opt-out. OpenAI later documented separate bots for separate jobs, with GPTBot remaining the training crawler and OAI-SearchBot for the ChatGPT search index and ChatGPT-User for on-demand fetches.

Read more: [Aug 2023: GPTBot gives owners a training opt-out](https://groas.com/post/from-gptbot-to-default-blocking-a-dated#aug-2023-gptbot-gives-owners-a-training-opt-out)

## Does blocking Google-Extended keep my pages out of Google AI answers?

The article states it does not. Google-Extended arrived in late September 2023 as a robots.txt control token for Bard and Vertex AI training rather than a separate crawler or a Google Search indexing control. The article advises not removing a training preference simply because a page is absent from AI Overviews and not expecting its removal to repair a crawler or rendering problem.

Read more: [Sep 2023: Google-Extended adds another easily misread rule](https://groas.com/post/from-gptbot-to-default-blocking-a-dated#sep-2023-google-extended-adds-another-easily-misread-rule)

## Can my robots.txt allow AI bots but my CDN still block them?

The article states yes. In July 2024 Cloudflare released its one-click Block AI Scrapers and Crawlers control under Security > Bots, including on free plans, and a robots.txt Allow cannot undo a network-layer block. Cloudflare reported at launch that GPTBot had reached 35.46% of its sites and ClaudeBot 11.17%, while 2.98% of top sites were blocking.

Read more: [Jul 2024: Cloudflare puts an AI block below robots.txt](https://groas.com/post/from-gptbot-to-default-blocking-a-dated#jul-2024-cloudflare-puts-an-ai-block-below-robotstxt)

## Why doesn't adding an llms.txt file fix missing AI citations?

The article states it cannot fix access by itself. The llms.txt proposal on Sept 3, 2024 by Jeremy Howard and Answer.AI described a Markdown file at /llms.txt with a summary and links, as a community proposal not an IETF standard. An /llms.txt file does not override robots.txt or a firewall, so its absence alone does not explain invisibility and its presence does not open a blocked page.

Read more: [Sep 2024: llms.txt offers a shortcut that cannot open a blocked page](https://groas.com/post/from-gptbot-to-default-blocking-a-dated#sep-2024-llmstxt-offers-a-shortcut-that-cannot-open-a-blocked-page)

## Sources

[1] https://groas.com/post/from-gptbot-to-default-blocking-a-dated (the article this page answers from)