How DoneHat measures AI visibility
DoneHat shows the questions, answer sources, usable trials and uncertainty behind its sampled mention rates. Here is how to interpret those results and where the method has limits.
The problem
One AI answer does not describe a whole market.
Ask the same buyer question twice and an AI answer may change. We sample repeated answers where available, keep failures separate, and show the text behind a reported mention. That lets you judge the evidence behind a change: compare the samples, then inspect the answers.
The pipeline
Five steps from question to reviewable evidence
1. Ask what real buyers ask
DoneHat builds buyer questions from the pages it can read and the business details you provide. Questions that do not name your brand measure unaided discovery, or comparison inclusion when buyers are comparing options; any question that names your brand, including one asking for alternatives to you, counts only as branded recall. Review the question set before treating it as representative of your market.
2. Sample, don't spot-check
A run can repeat questions across its configured answer sources, within plan and cost limits. The report shows attempted and usable trials. Failed calls are not treated as a missing brand mention.
3. Judge with evidence, not vibes
An AI classifier proposes mentions and comparisons. Deterministic checks reject unsupported claims; the quoted answers and sources let you inspect the decision. The classifier is not a substitute for human review.
4. Report rates with confidence intervals
For sampled questions, DoneHat shows counts such as “mentioned in 9 of 15 usable answers” with a 95% Wilson interval. Small samples produce wide intervals. Composite scores are directional summaries, so inspect the underlying counts and answers.
5. Prove lift, or say so
After you record a published change, DoneHat can repeat comparable questions with the recorded model and compare the samples. A result can be inconclusive. A measured change does not establish that the published edit caused it.
The refusals
What DoneHat will not do
A measurement product is defined by what it refuses to fabricate. These rules are enforced in code, not policy documents:
- No mock or fabricated reports — a failed audit fails loudly instead of inventing a score.
- Failed engine calls are never counted as “brand not mentioned”.
- Judge verdicts without verifiable evidence in the answer are discarded.
- Cached results are labeled as cached, with their age — never passed off as fresh.
- Receipts can only be published for statistically verified improvements.
- Checks we can't evidence (speculative AI-discovery files with no engine adoption) stay out of your score.
The research
Built on published, measured methods
These references explain the statistical method and broader research context. Results in other studies are not a forecast for your business:
- GEO: Generative Engine Optimization (KDD 2024) — research into how content changes can affect answer visibility. We treat any suggested edit as a hypothesis for your team to review and test.
- AutoGEO research preprint — research on automated content preferences. It does not establish an outcome for your site.
- Wilson score interval (1927) — the interval shown for sampled mention rates. A small sample can leave substantial uncertainty.
FAQ
Methodology questions, answered
Why does DoneHat ask the same question multiple times?
AI answers can vary between runs. DoneHat repeats sampled buyer questions where the plan and run allow it, records the usable trials, and shows a Wilson interval for mention rates. A change in the sample does not by itself prove that the market changed or that your fix caused it.
What is an evidence-guarded judge?
An AI classifier helps interpret each answer. DoneHat checks claimed mentions against the answer text and supporting sources, and rejects unsupported mentions. Classification can still be wrong, so the underlying answers remain available for review.
What happens when an AI engine fails to answer?
Failed calls are excluded from the denominator. A timeout is an infrastructure problem, not evidence that your brand is invisible — counting failures as absences would systematically deflate scores, so we never do it.
How are Google AI Overviews and AI Mode observed?
AI Overviews and AI Mode are observed through a search-results data provider from a data-center location, so results can differ from what a signed-in person sees. When Google shows no AI answer for a question, the report says so and the check is left out of every rate.
What is a verified-lift receipt?
A public proof page for one shipped fix: the buyer question monitored, the before and after mention rates with confidence intervals, and the verbatim AI answers. A receipt can only be published when the after-interval sits strictly above the before-interval. This is a conservative publication rule for separation in the observed samples; it does not establish that the fix caused the change or eliminate chance and other explanations.
Which research does the methodology build on?
Mention rates use a Wilson interval to show sampling uncertainty. Content suggestions are based on observed gaps and public evidence; they are proposals for human review, not a promise that a particular page edit will raise visibility.
See the methodology on your own brand
Run a free audit to inspect the sampled answers, usable trial counts and uncertainty on mention rates.
Run a free AI visibility audit