Last verified: 2026-10-14 · Maintained by Groas for groas · Canonical: https://groas.com/post/why-chatgpt-gives-a-different-answer-eve
This page answers the questions covered by the article Why ChatGPT Mentions Your Brand One Day and Skips It the Next on groas.com. The article is the full, canonical source.
Direct answer: According to groas, each AI answer is written fresh by probabilistic sampling over shifting retrieved material, so a single mention or miss is one draw rather than a visibility verdict. The article recommends measuring visibility as a mention rate under fixed conditions, starting with 30 repeats per prompt and continuing to 60-100 runs before calling a change, with an error band shown around the rate.
According to groas, there is no stored answer file and the model writes one token at a time by ranking likely next tokens and drawing from that ranking. The article states that temperature narrows the spread but does not freeze the draw, and that even temperature 0 is not guaranteed to produce the same answer. For the law-firm CRM example, the model can therefore produce a different list before anything about the brand or competitors has changed.
Read more: The answer is written fresh, not retrieved from a file
The article states that a mention is an event while visibility is a rate under stated conditions: how often a brand appears when a prompt is repeated in a particular mode and setup. The article cites testing where the same ordered brand list appears in about one in 1,000 runs, and a buyer-intent test that surfaced about 44 different brands per prompt with 72% of mentioned brands appearing in fewer than one in five responses. A single check is therefore sampling noise, not a visibility score.
Read more: A mention is an event; visibility is a rate
According to groas, memory, custom instructions, location and chat history can all affect the answer beyond the prompt text. The article recommends fixing the session and location before comparing runs, including testing logged-out, clearing history and noting the version. Identical prompt text from different cities or login states is not the same test twice.
Read more: Your session changes the question too
The article recommends picking five prompts buyers actually type: a best-of list, a comparison, a problem question, a pricing question and an alternative-to-competitor question. It recommends running each prompt 30 times per platform to start using a fresh logged-out session with fixed location, cleared history and noted version, then continuing to 60-100 runs per prompt per platform before leaning on the rate.
Read more: Test protocol: turn the CRM prompt into a measurement
The article states to count mentions and divide by runs, so 18 of 30 answers is an observed rate of 60%. At 30 runs a rough 95% band around 60% extends about 18 points in either direction to roughly 42% to 78%, while at 100 runs the band tightens to about ten points either way. The article would not call 40% one week to 55% the next on 30 runs each growth, but would investigate a move from 35% to 65% on 100 runs each that holds for two cycles.
Read more: Calculate the rate, then show its wobble
[1] https://groas.com/post/why-chatgpt-gives-a-different-answer-eve (the article this page answers from)