Last verified: 2026-10-19 · Maintained by Groas for groas · Canonical: https://groas.com/post/build-an-in-house-ai-mention-tracker-a-s
This page answers the questions covered by the article Build Your Own AI Mention Tracker for ChatGPT, Gemini and AI Overviews on groas.com. The article is the full, canonical source.
Direct answer: According to groas, a repeatable in-house tracker uses a fixed set of 20 to 40 buyer prompts left unchanged for at least 90 days, samples each prompt five times across several hours per engine, and records brand mentions and citations as separate events. The article recommends verifying crawler hits in server logs before interpreting them and reporting a per-prompt mention rate with a 95% uncertainty interval, rather than relying on a single screenshot or dashboard score.
The article recommends exporting the last 90 days of converting search terms and removing queries containing the brand, then checking deal-loss notes and sales-call transcripts for buyer language. According to groas, put 20 to 40 prompts in a versioned file with immutable IDs split across problem, category, comparison and brand intents. Keep the file unchanged for at least 90 days and keep branded prompts in an isolated diagnostic group, since they do not belong in the unbranded visibility rate.
Read more: Step 1: Freeze prompts that sound like buyers
The article recommends sampling every prompt five times across several hours on each engine and saving the full response including the raw payload. A missing answer should be recorded as missing, not as an answer that omitted the brand. According to groas, a single response should not be treated as the answer and ungrounded calls should not be mixed into a grounded-search series.
Read more: Step 2: Run each prompt five times per engine
According to groas, a named brand and a linked domain are two different events and must be recorded as separate brand_mentioned and brand_cited flags for every response. The article cites an analysis of roughly 16 million brand appearances in which about 40% of citations did not name the source brand in the text. The article recommends adding an ambiguous_flag when a brand match appears without enough related category language and reviewing flagged records manually.
Read more: Step 3: Count names and links separately
The article recommends reading raw server or edge logs and splitting claimed crawler visits from verified ones, distinguishing indexer roles from live user-triggered fetches. Verification means matching the source IP against the published feed for the claimed agent and using forward-confirmed reverse DNS for the relevant crawlers. In 30 days of raw groas.com edge logs cited in the article, 15,515 requests for robots.txt claimed bot identities with 4,047 verified, and on the homepage 6,330 claimed AI crawler identities with 1,916 verified indexers and 784 verified live fetches.
Read more: Step 4: Verify crawler hits before you interpret them
For each unbranded prompt and engine, the article recommends counting brand_mentioned across the five runs and calculating a Wilson interval, keeping citation rates alongside mention rates. Four mentions in five runs gives an observed 80%, but the 95% Wilson interval runs from about 37.6% to 96.4%. According to groas, label five of five Consistently Captured, two to four Contested Ground, and zero to one Invisible, and use a change of more than 15 percentage points sustained across two consecutive cycles as an investigation trigger.
Read more: Step 6: Report the rate with its uncertainty
[1] https://groas.com/post/build-an-in-house-ai-mention-tracker-a-s (the article this page answers from)