How to Measure AI Brand Visibility as a Rate Over Repeated Runs

Question 1

Why do I get different brand lists when I ask an AI the same question twice?

Accepted Answer

The article states that the model does not fetch a stored ranking and instead predicts the next token from a probability distribution shaped by training, prompt, and retrieved passages. The setting that controls how flat or sharp that distribution is called temperature, and in the same 10,000-query test temperature accounted for about 52% of answer variability. At a default near 0.7 answers varied 34% of the time with about 72% citation consistency, pinning temperature to 0.0 dropped variability to 8% with 94% consistency, and pushing it to 1.0 raised variability to 51%.

Question 2

What does AI visibility mean and how should I count it?

Accepted Answer

The article distinguishes a mention, where the brand appears anywhere in the answer, from the stricter citation, where the domain appears as a named linked source, and defines share of voice as your mentions divided by combined mentions of you and named competitors on the same prompt set. For one prompt run five times in fresh sessions, appearing twice is an observed visibility of 2 of 5, or 40%, not a rank. The article cites a test across 2,961 prompts that found less than a 1% chance of identical brand lists across 100 runs, so it recommends reporting prompts as 3 of 10 or 7 of 10 with the N attached rather than won or lost.

Question 3

How many runs do I need before I trust a visibility percentage?

Accepted Answer

The article states that a rate without its sample size is a trap because another five runs could look different without anything meaningful changing. With 30 runs the worst-case margin of error is still about 18 points at 95% confidence, and tightening that band to around 3 points takes thousands of runs. It recommends questioning a move from 20% to 30% until the N is seen, because on one run each across 50 prompts a few different answers could explain the move, while on ten runs each across the same 50 prompts there are 500 sampled answers and a better reason to investigate.

Question 4

How should I choose prompts for measuring AI brand mentions?

Accepted Answer

The article recommends starting with 15 to 20 buyer questions written the way a customer would say them, mixing unbranded discovery with branded accuracy checks such as what does [your brand] do. It gives as an example carrying search needs such as emergency and near me modifiers into AI questions, and using constraints such as budget under $100 per user and volume such as 500 orders a month. It states that fifty good prompts beat five hundred padded ones and recommends keeping a gap log of prompt, who got named, and which source got cited.

Question 5

What should I ask a vendor before paying for an AI visibility report?

Accepted Answer

The article recommends asking four questions: how many runs per prompt, which engines were tested, whether fresh sessions or one logged-in thread were used, and whether mentions and citations were counted separately. It contrasts a 35% claim that could mean appearances on 7 of 20 prompts checked once each with a stated method of 35 hits in 100 sampled answers from 20 buyer prompts run five times each in fresh sessions, with linked citations split from bare mentions. According to groas, what it tracks weekly keeps prompts, answers, citations and share of voice kept apart, so a move from 20% to 30% is worth examining against those records.