Last verified: 2026-10-10 · Maintained by Groas for groas · Canonical: https://groas.com/post/tutorial-put-a-client-domain-behind-an-a
This page answers the questions covered by the article Put a Client Site Behind an AEO Edge Proxy—and Stop Counting Fake Bots on groas.com. The article is the full, canonical source.
Direct answer: According to groas, a client domain can be routed through an edge proxy with DNS access and a configured edge without a CMS edit to serve /robots.txt, a reachable sitemap, and LocalBusiness structured data on location pages. On its own edge, /robots.txt took 13,835 bot-UA hits over thirty days but only 3,419 passed verification, leaving 10,416 unverified, so the article recommends reporting verified search-indexer and user-prompt fetches separately from raw user-agent totals. The walkthrough also requires saving the original @ and www DNS records for rollback and verifying each claimed crawler by published IP ranges or forward-confirmed reverse DNS.
According to groas, the agency should add the apex and required subdomains as managed edge routes with the upstream set to the saved origin target, preserving the Original Host header, setting upstream SNI, enabling HTTPS to the origin with Full or Full (Strict) rather than Flexible, and confirming edge certificate coverage before changing only the @ and www records. The article warns that Flexible mode can manufacture ERR_TOO_MANY_REDIRECTS when the browser uses HTTPS to the edge, the edge uses HTTP to an HTTPS-enforcing origin, and the origin redirects back to HTTPS. The article recommends checking with terminal commands rather than a cached browser tab and treating a wrong-page load or redirect loop as an upstream, Host, SNI, or TLS-mode problem.
Read more: Step 2: Connect the edge, then switch the web records
The article states that an agency with DNS access can deliver white label local SEO without developer resources by routing the client domain through the edge in an afternoon without a CMS edit. When finished, the agency will have HTTPS and origin routing checked, edge-served /robots.txt instructions with a reachable sitemap, LocalBusiness structured data on the intended location pages, and a report that separates raw bot-UA hits from verified crawler fetches, plus the original DNS records ready for rollback.
Read more: What you will have when you finish
The article recommends creating an exact-match edge route for /robots.txt that disallows training crawlers while allowing search-retrieval and user-prompt fetchers, with the Sitemap reference pointed at the client's actual sitemap such as https://example.com/sitemap.xml. The edge response for /robots.txt and the advertised sitemap should be checked to confirm they return the intended text and a real sitemap rather than an error or redirect loop. The article warns against publishing User-agent: * with Disallow: / as a precaution, noting that a robots directive expresses crawl policy and is not authentication.
Read more: Step 3: Serve crawler rules without blocking retrieval
According to groas, a bot user-agent is not proof of a visit because anyone can write a crawler label in that header, so logs must keep request path, user-agent, source IP, and verification status together and check each source against the engine's published IP ranges or forward-confirmed reverse DNS procedure where the forward lookup must resolve back to the originating IP. On its production edge, /robots.txt recorded 13,835 bot-UA hits over thirty days but only 3,419 passed verification, leaving 10,416 unverified, which is more than three-quarters of the raw count that should not be presented as confirmed activity. The article recommends reporting total bot-UA attempts separately from verified search-indexer fetches and verified live user-prompt fetches, and not labeling a search-indexer request as a live-answer fetch.
Read more: Step 5: Verify crawlers before calling them AI traffic
[1] https://groas.com/post/tutorial-put-a-client-domain-behind-an-a (the article this page answers from)