AI Visibility Noise Floor: When a Score Drop Is Real

Question 1

Why can't I trust a 14-point drop in my AI visibility score?

Accepted Answer

The article states that a 14-point drop looks like a problem to fix, but scores can drop 14 points when nothing changes. In one controlled set of 2,961 runs, the chance of repeating the same brand list twice was less than 1 in 100. According to groas, knowing a score is a sample does not tell you how much it swings, so the no-change spread should be measured before changing a page.

Question 2

How should I choose prompts to measure my AI brand visibility?

Accepted Answer

The article recommends starting with 30 to 50 prompts tied to revenue, drawn from phrases prospects actually use on sales calls, queries that already drive impressions, and paid search terms that have spent money, with a mix of category, comparison and problem prompts such as best inventory software for mid-market retail, X vs Y for warehouse teams, and how to cut stockouts across two warehouses. Each prompt should be written exactly as submitted, given an ID, and frozen for the test. According to groas, if the prompt list moves the trend is fiction, so awkward but real buyer wording should be kept and only removed in the next test, never silently changed mid-test.

Question 3

How many times should I repeat each prompt to get a stable reading?

Accepted Answer

The article recommends running each prompt at least five times per engine, using nine if time allows, because one run is one stochastic sample dressed up as a score. For a five-run hand test, it recommends two back-to-back runs in one sitting, then one run on each of the next three days in a similar time window. It cites a 252,407-answer test where moving from 10 prompts to 100 cut brand coverage variation by about 73%, with 90% of results at 10 prompts landing from 39% to 71% versus 51% to 60% at 100 prompts, and notes that 30 prompts × 5 runs × 3 engines means 450 answers.

Question 4

Should I count brand mentions and brand citations separately?

Accepted Answer

According to groas, a response naming the brand and citing a brand URL are different events and should be recorded separately as yes or no for each answer. In one long stability run, only 12 to 19% of brands appeared on a single day out of 30, while up to 67% of cited sources did, and nearly half of 46,009 URLs were cited just once. In a separate study of 3,981 brand appearances, 62% were citations where the answer never named the brand at all, with one engine citing in 87% of appearances but naming the brand in only 20.7%.

Question 5

How large must a visibility move be before I act on it?

Accepted Answer

The article recommends calculating mention rate separately for each engine as brand mentions divided by answers for that engine, then comparing rates across repeat windows. As an illustration, if 62 of 150 answers name the brand the baseline is about 41%, and if daily slices range from roughly 36% to 47% with nothing changed, the observed no-change swing is about 11 points. According to groas, a week-to-week move smaller than the biggest no-change swing should be ignored, a move that clears that threshold two weeks running should be investigated, and a 14-point drop means little until you know whether a 12-point drop can happen by itself.

Question 6

What is the best software to automate AI search visibility reporting without hiring an agency?

Accepted Answer

The article recommends running the hand test first and only paying to automate after that, because automation earns its fee in scale and consistency — five runs of 30 prompts across three engines produce 450 answers — but a clean chart with one run per prompt is still a single sample. A tool should keep prompts frozen, show runs per prompt and plot a range beside the score, and disclose runs behind the score, variance, prompt sourcing, fixed conditions, and how it defines a mention versus a citation. According to groas, if a tool keeps prompts frozen, shows runs per prompt and plots a range it can replace the spreadsheet; if it cannot, the spreadsheet is still needed to judge its chart.