Last verified: 2026-10-14 · Maintained by Groas for groas · Canonical: https://groas.com/post/the-holdout-test-for-ai-visibility-fix-h
This page answers the questions covered by the article The AI Visibility Holdout Test: Fix Half Your Pages, Then Check What Changed on groas.com. The article is the full, canonical source.
Direct answer: According to groas, the way to tell if an AI-visibility fix earned citations is a six-week holdout test: establish a two-week baseline, apply one fix to half of 10-20 matched pages for four weeks while freezing the other half, then compare changes in verified crawler fetches and fixed-prompt citations against the holdout. A fix earns a wider rollout only when treated pages beat the holdout on both retrieval and citations; shared movement is background drift, not proof of the fix.
The article states a tool should get credit only for a change that beats the holdout, not for being installed when an index refreshes. It describes a platform that tracks prompts, graphs a synthetic index, and exports a monthly PDF as an observer that has not demonstrated that its recommended fix caused a citation. It states that groas executes search and content work continuously with a named human strategist setting direction and accountability, but says the same holdout rule applies to its changes.
Read more: Make the tool earn its place
The article recommends a six-week window: two weeks to establish a baseline, then four weeks with one fix applied to half the pages while the other half stays untouched. During the test it records verified bot fetches and citations in live answers separately for both groups. At the end it compares each group’s change from its own baseline against the holdout, not against a screenshot of where the treated pages started.
Read more: The question: did the fix earn citations?
The article recommends taking 10-20 pages with similar topical depth, template structure, and publication age and pairing pages whose traffic over the last 60 days falls within roughly 15% of each other. It advises checking intent as well as volume, giving the example of pairing a page drawing 800 monthly visits with one drawing 750 rather than with the homepage. It warns against putting revenue-driving pages on one side and forgotten posts on the other.
Read more: Choose 10–20 URLs, then make lookalike pairs
The article states that filtering server logs for user-agent strings is not enough because a scraper can claim either name. It cites its own 30-day log finding 3,482 genuine requests among 13,774 claimed bot hits, and says the exact mix on another site may differ. It recommends setting up verification checks before the baseline begins and excluding requests that fail the relevant check.
Read more: Count verified bots, not claimed user agents
The article recommends writing five specific user-intent prompts for each pair, for 25-50 prompts across a 5-10-pair test, and keeping wording and page-pair assignment fixed throughout. It recommends running that same battery every Tuesday at 10:00 AM UTC and recording 1 if the specific URL appears as an inline reference, card, or footnote and otherwise recording 0. It says to ignore the proprietary visibility score for this decision and keep the answer or screenshot with each entry.
Read more: Check citations against a fixed prompt set
The article recommends extending the same fix to comparable pages only when treated pages beat the holdout on both relevant verified fetches and fixed-prompt citations. If both groups rise or fall at roughly the same time, it says to treat the shared movement as background drift, not credit for the fix. It states that a fetch advantage without a citation advantage is its own finding and should not be promoted into a claim about earned citations.
Read more: Read the result against the holdout
[1] https://groas.com/post/the-holdout-test-for-ai-visibility-fix-h (the article this page answers from)