From first question to clear priorities.

An audit, step by step, behind the scenes

The same journey from two sides. The middle column is what you experience. The right column is the work done for you.

Choosing the questions

Everything depends on asking the right questions. If the questions are unrealistic, the results say nothing about real customers. People also phrase the same need in very different ways: in one study, 142 people wrote a prompt for the same intent and the wording was almost never alike. So we do not rely on a handful of questions typed by hand.

How the question set is built

1. Describe who is asking and why. We combine types of customer (for example a homeowner with an urgent problem versus one planning ahead), stages of buying (noticing a problem, looking at categories, building a shortlist, comparing options, checking a specific business, after the purchase) and what they want to know.
2. Add place and detail. For local businesses, questions mention a city or neighbourhood and real constraints such as budget and timing, because that is how people actually ask.
3. Write them the way a person would. Each question is 15 to 40 words, in natural chat language, and gets several reworded versions so the test does not depend on one phrasing.
4. Remove repeats. Questions that are nearly the same in meaning are dropped, and the rest are grouped into topics.
5. Limit questions that name your business. A question like “Is Joe’s Heating any good?” will mention Joe’s almost by definition, which would inflate the score. These are capped at a small share of the set.
6. Read a sample and freeze the set. You read a sample of the questions and confirm they sound real. The final list is saved as a numbered version. A changed list starts a new series, so a movement in the numbers can never be blamed on different questions.

Illustrative example (made up): “My air conditioning stopped cooling and it is 95 degrees. Who is a reliable HVAC company near Holly Springs that can come out today?”

Collecting the answers

Why every question is asked many times

AI assistants do not give the same answer twice. In a study by SparkToro and Gumshoe (600 volunteers, 12 questions, nearly 3,000 answers from ChatGPT, Claude and Google’s AI), the same list of brands came back in two answers less than 1 time in 100, and in the same order about 1 time in 1,000. Yet when the answers were pooled, the leading brands appeared at fairly steady rates, often in 60% to 90% of answers for a given need. A single answer is noise. The pattern across many answers is the signal. That is the same reason a pollster asks hundreds of people rather than one.

A University of St. Gallen preprint found that per-brand estimates became reasonably stable at about 7 runs of the same question per day, and tighter still when results were pooled over roughly 10 days (error under 10 percentage points) or 24 days (under 5 points). That is the basis for repeating questions and, later, repeating audits over time.

The SparkToro and Gumshoe study has a commercial tie and was not peer reviewed, and the St. Gallen work is a preprint. We cite both as supporting evidence for the approach, not as settled science.

Which assistants

Your scope names the assistants measured, for example ChatGPT, Gemini, Perplexity and Claude. Where an assistant offers a web search feature, we use it switched on, because that is how assistants find current local businesses. The report states which systems were measured, which versions, and the dates of sampling.

What is recorded with every answer

• Which assistant and the exact model version, since models change without notice.
• The date and time, the location setting and the search mode.
• The full answer, saved unchanged, including the web pages it cited.
• A fresh conversation each time, with no memory of earlier questions, so one answer cannot influence the next.

Smart sampling

Every question is asked a base number of times. More runs are then added only for the questions whose results are still uncertain, so effort goes where it narrows the margin of error most. The scope sets a fixed ceiling, and collection stops at it.

Reading the answers

A saved answer is just text. To count anything, we have to find every business named in it, work out how the assistant described each one, and note which websites were quoted. This is done in two passes.

Pass 1: name matching

Software looks for your business and each competitor by every name they go by, matching whole words so that “Ace” does not match “place”.

Pass 2: a structured reading

For every business mentioned, tracked or not, the reading records:

• How it was treated: recommended, listed neutrally, spoken of negatively, or mentioned only in a comparison.
• Tone, on a scale from negative to positive.
• Any claims made about the business, and the web page each claim came from, if any.
• An exact quote from the answer that justifies the label.

Cited websites

Assistants that search the web show which pages they used. Each link is cleaned (tracking codes removed, redirects followed, reduced to the website) and sorted by type: your own site, a competitor’s site, a review site, a directory, news or media, a forum, a marketplace, or a reference site. This shows which outside sites the assistants trust when they recommend a competitor, which is often the most useful finding for deciding what to do.

Protections against a reader that makes things up

• The quote check. After the reading, code confirms that every quote appears word for word in the original answer. Any quote that does not is dropped or flagged. The reader cannot invent evidence.
• A fixed form. The reader can only return a structured form. It cannot browse, click or run anything.
• Answers are untrusted. Text inside an answer, or inside a crawled web page, could contain instructions aimed at an AI reader. All of it is treated as data to read, never as orders.
• A pinned model. The reader is locked to one dated model version, and that version is saved with every result, so a silent upgrade cannot quietly change the scores.
• A memory of past readings. Reading the same answer again returns the saved result, so reruns are consistent.

The numbers explained

Once the answers are read, we count. Each measure below is calculated for each assistant and each group of related questions, and each comes with a margin of error. Not every report includes every measure; the scope states which are in yours.

How to read a range

A result such as “named in 40% of answers” is only half the story. The other half is how sure we are. Roughly, with 25 independent samples the margin is about plus or minus 20 points, with 100 it is about 10 points, and with 400 about 5 points. We use a standard method for this, called Wilson intervals, that behaves well when counts are small.

Repeats are not all independent

Asking the same question 8 times gives 8 related answers, not 8 separate pieces of evidence. So we count the number of different questions and different days as the real sample size, and we widen the margin when most of the data comes from one day or a few questions. This is why a single-day result carries wide ranges.

Too little data

If a group of questions has fewer samples than a set minimum, the report says “insufficient data” instead of showing a number that looks precise and is not.

Comparing two results

A change over time is called real only when the ranges stop overlapping. A rise from 38% to 41% with overlapping ranges is reported as no clear change.

Why ranges, not a single rank

The easy thing to sell in this space is a single number such as “your AI rank is #3”. It is appealing and misleading. If the same question produces a different list of businesses almost every time, then “#3” describes one answer on one afternoon, not your business. Reporting it as a fixed position would be like announcing a sports team’s ranking from one game.

Our position is that visibility in AI answers is a probability, not a place in a list. The honest statement is “this business is named in about four of every ten answers to these questions, give or take seven points, and its closest competitor is named in six of ten.” That statement stays true when the sample is repeated, which a single rank does not. It also means nobody can disprove the report by asking an assistant once and getting a different answer.

Wording that holds up

The percentages in this table illustrate format. They are not real results.

How we know the reading is right

The reading could still make mistakes, such as missing a mention or labelling a lukewarm mention as a recommendation. If it did, every number downstream would be wrong. So accuracy is tested before anyone relies on it.

The answer key

• A sample of 200 saved answers, spread across assistants and questions, is set aside.
• A person reads each one and records the correct result: was the business mentioned, which business was recommended, and how it was described. They do this without seeing what the software decided, so the answer key is independent.
• The 200 are split into 100 for practice and 100 for a final exam. The reading is improved using the practice set only. The final-exam answers are kept locked and used once.

The grades

• Mention accuracy (F1 score). Combines two questions: of the mentions found, how many were real, and of the real mentions, how many were found? A score of 1.0 is perfect. Our target is 0.95 or higher.
• Agreement on how a business was described (Cohen’s kappa). Measures how closely the labels (recommended, neutral list, negative, comparison) match the person’s, after removing agreement that would happen by luck. Our target is 0.7 or higher, usually described as substantial agreement.

These are targets, not guarantees. We do not publish accuracy figures for a report until they have been measured.

Website readiness checks

Measuring answers tells you what is happening. The website checks help explain why. If an AI system cannot reach, read or trust your website, it has little reason to mention you. These checks do not need paid AI calls, so they are cheap and repeatable.

How findings are ranked

Findings are ranked by how strong the evidence is. Hard blockers (the site refuses AI crawlers, or the page cannot be indexed) come first because they stop everything else. Content and consistency issues come next. Items such as structured data and special AI text files are treated as low-weight hygiene, since the evidence that they change AI behaviour is weak. Google has said it does not require special markup or AI files for its AI features.

The GEO Readiness Score

The checks roll up into a 0 to 100 score with plain bands: 0 to 39 Critical, 40 to 59 Weak, 60 to 79 Competitive, 80 to 100 Strong. It is deliberately kept separate from the AI Visibility Index. Readiness measures what you control. Visibility measures what actually happened in AI answers. The weights behind the readiness score are our judgment, informed by evidence, and the report says so. Every finding carries a rule code, the evidence, and a strength-of-evidence label, so you or your developer can check it by hand.

What you receive

A report you can read without help, as a web page and as a downloadable PDF.

• Two headline numbers, kept apart: the AI Visibility Index (what happened in AI answers) and the GEO Readiness Score (how well your site is set up). They answer different questions, so they are never blended into one score.
• Charts with ranges: mention rate, citation rate and share of voice for each assistant, drawn with their margins of error.
• The lost-questions list: the exact questions where a competitor was named and you were not.
• Who the assistants trust: the outside websites cited when competitors are named.
• A ranked action list, each item with who should do it (developer, content writer, PR or operations), how big the job is (small, medium or large), and how to tell whether it worked.
• An appendix listing every finding with its rule code and the evidence behind it.

How actions are ranked

Each is scored by expected impact, multiplied by how confident the evidence is and how many customer questions it affects, then divided by effort. High-impact, well-supported, low-effort fixes rise to the top, and appear as P0, P1 or P2.

Illustrative action (made up): “Allow the ChatGPT search crawler in the site’s access file. Owner: developer. Effort: small. Measure: mention rate on the 12 ‘near me’ questions over the next 4 weeks.”

How the written explanations are kept honest

An AI helps draft the plain-language explanation for each action, but it is limited to the evidence supplied and may not invent claims about impact. After it writes, code checks the output: every number in a narrative must appear in the measured results, or the text is rejected and retried. Any suggested code, such as a structured-data snippet or a change to the access file, must pass the same validity checks as the audit’s own tests. In short, the AI writes the words and software supplies every number.

Follow-up

If a follow-up is included in your service agreement, we re-measure with the same questions and compare. A change is flagged only when the new range no longer overlaps the old one, so ordinary noise does not set off false alarms.

Safeguards

• Evidence is kept, untouched. Every answer from every assistant is saved exactly as received, before anything is read from it. Scores trace back to the original words, and if the reading improves later, old answers can be re-read.
• Reports can be rebuilt. Calculated results can be rebuilt from the saved raw answers, and the new report checked against the old one. This shows the numbers come from the evidence and not from anyone’s memory or hand edits.
• Everything is versioned. The model that read each answer, the version of the question set, and the version of the scoring rules are all recorded, so results from different periods are compared fairly.
• Scope and cost are agreed before collection. The number of questions, assistants and repeats is set in writing before any answers are collected.
• Hostile text cannot take control. Web pages and AI answers are treated as untrusted. The steps that read them have no tools and can only return a fixed form, so text hidden in a page or an answer cannot trick the process into acting.

What this cannot do

We would rather name the limits before you have to ask.

• It samples; it does not see everything. It reports what assistants say to a large, realistic set of questions. It cannot see every conversation, or what any individual user is shown after personalisation.
• Developer connections can differ from the consumer apps. Answers collected this way can differ somewhat from what a person sees in a chat app.
• Not every AI surface is covered. Your scope lists the assistants measured. Google’s AI Overviews and AI Mode, and similar search-integrated AI features, are not assumed to be included.
• It cannot promise results. Fixes are ranked by evidence, but nobody can guarantee an assistant will start recommending a business. The report says what to change and how to measure whether it worked.
• Changes take weeks to confirm. Because answers vary, a real change in visibility needs several days or weeks of repeated measurement to show up clearly.
• Assistants change without notice. A model update can shift results overnight. We record the model version and flag sudden changes, but the shift itself is outside anyone’s control.
• Readiness weights are judgment. They are informed by evidence, not by measured outcomes.
• Scope is narrow. Audits are for US, English-language service businesses unless your scope says otherwise.
• A person is still needed in two places: confirming your competitor list, and hand-labelling the answers that prove the reading is accurate.

Sources

• SparkToro and Gumshoe study on AI recommendation lists (reported by Search Engine Land): 600 volunteers, 12 prompts, nearly 3,000 runs. The authors are vendors and the study was not peer reviewed.
• University of St. Gallen preprint, “Don’t Measure Once: Measuring Visibility in AI Search (GEO)”.
• “GEO: Generative Engine Optimization” (arXiv 2311.09735): its visibility claims came from a benchmark, not live assistants, so we treat them as directional.
• Google, “AI features and your website”: no special optimization, markup or AI text files required; a page must be indexed and snippet-eligible.
• Vercel, “The rise of the AI crawler”: OpenAI, Anthropic and Perplexity crawlers do not execute JavaScript; Google’s Gemini does.

Question-set rules, reading rules and accuracy targets are our own design, not outside sources.