How do I measure AI mentions as odds with error bars?
For marketers and owners who want to know whether GEO work is actually working. After reading, you will be able to build a simple tracking routine and tell a real improvement from random noise.
Measure AI mentions as a frequency: run each important question many times across assistants and count how often your brand is named, then report that share with a margin of error. One check proves little, because answers change from run to run. Say an apparel brand is named in 20 of 50 runs: that is 40%, with a 95% margin of roughly 14 points either way, so a later 45% is not yet a real gain.
AI answers are samples, so treat each one like a coin flip
Every AI answer is one draw from a range of possible answers, so a single check tells you very little. Assistants build in deliberate randomness, often controlled by a setting called temperature, so the same question can produce different wording and different brands each time. Answers also shift with location, chat history, memory features, whether web search ran and silent model updates.
That is why a screenshot of one answer, good or bad, is not evidence. The useful question is not 'Are we named?' but 'How often are we named?'
Build a prompt set that mirrors real buying questions
Your measurement is only as good as the questions you test, so write them the way real customers ask. Aim for 20 to 50 prompts that cover several kinds of question, then freeze the exact wording so results stay comparable over time.
- Category questions: 'best linen bedding for hot sleepers'.
- Problem questions: 'how do I stop overheating at night'.
- Comparison questions: 'Brand A vs Brand B sheets'.
- Local questions: 'best florist for weddings in Denver'.
- Brand questions: 'is [your brand] good quality' and 'what does [your brand] sell'.
Run each question many times and record the same details
A simple, repeatable routine beats an elaborate one you run once. Use the same setup each time so changes in the numbers reflect changes in the world, not in your method.
- 01
Choose the assistants your customers use, for example ChatGPT, Gemini, Perplexity and Copilot.
- 02
Run every prompt at least 10 times in each assistant, using fresh chats with memory and personalization switched off.
- 03
Record for each answer: named or not, position in the list, whether the description was accurate, and which URLs were cited.
- 04
Do the same for your top three competitors in the same answers.
- 05
Calculate your mention rate: answers that name you divided by total answers.
- 06
Repeat monthly with the identical prompts and settings.
Add error bars before you celebrate
A mention rate always comes with uncertainty, and the margin of error tells you how much. A handy rule of thumb for a 95% margin is 1 divided by the square root of the number of runs. With 50 runs that is about 14 percentage points, with 200 runs about 7 points, and with 900 runs about 3 points. The rule is most accurate for rates near 50% and slightly generous for rates near 0% or 100%.
Treat a change as real only when it is clearly larger than the margin. Moving from 40% to 45% on 50 runs is noise. Moving from 40% to 65% on 50 runs is very likely real. Repeated runs of the same prompt are not fully independent, so treat the margin as a minimum, and trust trends that hold across several months.
Track accuracy, citations and traffic, not just mentions
Being named wrongly can be worse than not being named, so score accuracy alongside frequency. Note any wrong prices, outdated services or confusion with another business, and fix the source.
Also track the visits AI assistants send you. In your analytics, look for referral traffic from domains such as chatgpt.com, perplexity.ai, gemini.google.com and copilot.microsoft.com. ChatGPT often adds utm_source=chatgpt.com to the links it shows, which makes those visits easier to spot. This traffic is usually small but tends to arrive ready to buy.
Do this next
Questions people ask
How many runs do I need?
At least 10 per prompt per assistant, and enough in total to bring your margin under the size of change you care about. For a margin of about 5 points, you need roughly 400 answers.
Can tools do this for me?
Yes, several tracking tools now run prompts on a schedule and report mention rates. Check that they run each prompt many times and show sample sizes, not just a single snapshot.
How often should I measure?
Monthly is enough for most businesses. Measure more often only around big changes, such as a site relaunch or a major model release, so you can see the effect.