# Measuring Brand Mentions in AI Answers With Repeated Runs and a Holdout

Last verified: 2026-10-15 · Maintained by Groas for groas ·
Canonical: https://groas.com/post/a-control-group-test-for-ai-visibility-h

This page answers the questions covered by the article [The AI Visibility Holdout Test: Measure Mentions, Not Screenshots](https://groas.com/post/a-control-group-test-for-ai-visibility-h) on groas.com. The article is the full, canonical source.

**Direct answer:** According to groas, businesses should measure AI visibility with a fixed set of 30 buyer prompts run three times per engine, calculating mention rate as runs that mention the brand divided by all runs. The article recommends reporting each engine separately with the spread across runs, tracking rivals as a baseline, and using matched holdout pages to test whether a page edit caused a citation change.

## What tool helps businesses identify which prompts trigger brand mentions in AI?

According to groas, the article points to a 30-prompt test sheet for locking the prompt set and a tracker sheet for logging each run as a separate row. The article recommends logging mention, citation with URL if present, prominence, and the raw answer with prompt version and timestamp, then calculating mention rate as runs that mention the brand divided by all runs. For a qualitative pass, it describes a 0-to-2 rubric for frequency, prominence, citation quality and sentiment over 25 to 40 prompts across four themes, with raw outputs, links and timestamps saved for retesting.

Read more: [Measurement: count runs, save answers, report the spread](https://groas.com/post/a-control-group-test-for-ai-visibility-h#measurement-count-runs-save-answers-report-the-spread)

## How many prompts should I freeze for an AI visibility baseline and how should I choose them?

The article recommends building 30 prompts from buyer questions and splitting them roughly into thirds across problem, comparison and buying types. The set should be labeled v1.0 with a date and frozen, with branded prompts separated out and kept out of the headline rate because naming the brand when asked for it is recall, not evidence of discovery. The article warns not to add or drop prompts mid-test, stating that a prompt change is a new test version, not a visibility lift.

Read more: [Setup: freeze the prompts before you see a score](https://groas.com/post/a-control-group-test-for-ai-visibility-h#setup-freeze-the-prompts-before-you-see-a-score)

## How do I tell whether a page edit caused a change in AI citations?

According to groas, the article recommends choosing six to eight pages mapped to prompts where the brand is absent and pairing each with a similar page to leave untouched, matched on template, age and traffic band, before editing anything. The test group receives one kind of edit, such as a direct answer block, a cleaner facts table, or a citation-worthy stat with a date, while the holdout group is frozen. The same frozen prompt set is then re-run three times per engine under the same conditions to compare the test-page citation change with the holdout-page citation change alongside the rival baseline.

Read more: [The holdout test: did the page edit cause the change?](https://groas.com/post/a-control-group-test-for-ai-visibility-h#the-holdout-test-did-the-page-edit-cause-the-change)

## Why can a 15-point jump in AI visibility still be noise?

The article states that at a 20% appearance rate with 20 prompts, the error band is plus or minus 17.5 points, tightening to plus or minus 11.1 at 50 prompts and plus or minus 7.8 at 100 prompts. Comparing 20% with 25% at n=50 carries around 16 points of error on the difference, and a five-point move would need around 1,100 prompts per period to call it real with 80% power while a ten-point move needs around 290. The article therefore treats a 30-prompt set run three times per engine as a baseline with a spread, not a verdict, and recommends looking for a gap that persists across repeated periods while rivals remain flat.

Read more: [Reading the result: a 15-point jump may still be noise](https://groas.com/post/a-control-group-test-for-ai-visibility-h#reading-the-result-a-15-point-jump-may-still-be-noise)

## Sources

[1] https://groas.com/post/a-control-group-test-for-ai-visibility-h (the article this page answers from)