
A marketing team asks ChatGPT, “What’s the best CRM for a 20-person sales team?” Their brand appears in the answer. Someone takes a screenshot, drops it into a slide deck and reports a win in AI search.
It is a useful observation. It is not a benchmark.
The distinction matters because companies are starting to make budget decisions based on what AI assistants say about them. If the measurement comes from a handful of hand-picked questions, the result can look precise while telling us very little about what prospective buyers actually encounter.
The first question is rarely the whole question
A person looking for a CRM might begin with a broad question, then mention that their team is in France, uses Microsoft 365 and cannot spend more than €30 per seat. The assistant now has a different problem to solve. The shortlist may change completely.
That is not an edge case invented for a test. A 2025 study by researchers at OpenAI and Harvard, published through the National Bureau of Economic Research, analysed patterns in consumer ChatGPT conversations and found that information seeking and practical guidance account for a substantial share of use. These are situations where context and follow-up questions can matter more than a tidy, keyword-style opening prompt (How People Use ChatGPT).
A benchmark built only from first-turn prompts misses much of that context. It can also overstate the importance of broad questions such as “best project management software” when the actual purchasing decision depends on a particular integration, budget or regulatory requirement.
The unit of analysis should be a buying situation, not just a string of words.
Start with the decisions customers are making
The most defensible way to build a test set is to collect questions from real customer interactions, with appropriate permission. Sales calls, support tickets, on-site searches and customer interviews can help identify the constraints that repeatedly come up.
Consider a business selling accounting software. “What is the best accounting software?” is one possible question, but it is not much use on its own. A more informative set might include a freelancer who invoices international clients, a small company moving away from spreadsheets, and a finance director comparing solutions with an existing payroll system.
Each represents a different buying situation. Each may produce a different answer.
If real conversations are not available, synthetic prompts can still be useful. They should be labelled as test scenarios rather than presented as evidence of what the market is asking. A team that invents 50 prompts has learned how its brand performs against those 50 prompts, not how frequently buyers encounter it.
That difference ought to appear in the reporting.
Repeat the tests, then keep the variation
AI responses can change with wording, context, model behaviour and the information available at the time of the query. A brand that appears once may disappear on the next run, and a slight change in the question may bring in different competitors.
Instead of storing only the most flattering screenshot, record repeated responses to a stable set of scenarios. Keep the wording, date, assistant, location or language setting where relevant, conversation context, and whether web search was used. Record the full answer, not only a yes-or-no brand mention.
There is no universal number of repetitions that makes a benchmark valid. Five or ten runs per scenario may reveal obvious instability, but a small convenience sample does not justify claims about population-level market share.
Suppose a team tests six buying situations ten times each. Its brand is recommended in 15 of the 60 answers. It can reasonably report that the brand was recommended in 25% of those tested responses, with the scenarios and method disclosed. It cannot conclude that one quarter of all people shopping in that category will see the brand.
Keep the less favourable results in the dataset. They are often the most useful ones.
A mention, a recommendation and a citation are different outcomes
A brand may appear because the assistant recommends it, uses it as a comparison, or advises against it. A simple mention counter assigns all three the same value.
Take these answers: “Consider Product A for this team,” “Product A is popular but above your budget,” and “Unlike Product A, this option supports your required integration.” Each contains the brand name. Only the first is a positive recommendation.
Citations are another separate signal. ChatGPT Search can show links to sources used in an answer, but those links do not by themselves indicate that a source’s product was recommended. OpenAI also warns that search results and citations can be incomplete or incorrect, which is a reason to verify them rather than treat them as automatic endorsements (Searching the web with ChatGPT).
A practical report should separate brand mentions, positive recommendations, negative or qualified assessments, and linked citations. It should also note which competitors appeared in the same answer. Otherwise, a company could celebrate increased visibility even as the assistant increasingly explains why buyers should choose somebody else.
Do not confuse visibility with traffic or revenue
AI visibility is an exposure measure. Website visits, qualified leads and completed purchases are different measures, with different collection methods.
A prospective customer may learn about three providers in a conversation and later visit one of them directly. Another may click a cited link immediately. Both journeys involve AI, but they leave different traces in analytics. That makes attribution difficult, not impossible, and it certainly does not justify converting mention counts into revenue estimates.
Measurement tools are evolving. In 2026, Google introduced dedicated Search Console performance insights for generative AI features such as AI Overviews and AI Mode (Google’s Search Generative AI performance reports). Those reports are useful for understanding Google’s own surfaces, but they are not interchangeable with observations from a separate assistant or with conversion data in an analytics system.
Keep these layers separate in a dashboard. Otherwise, a rise in mentions can be mistaken for a rise in sales, while a useful influence earlier in the buying journey may be missed because it produced no immediate click.
What a credible benchmark should show
An honest report does not need an elaborate scoring formula. It needs enough detail that another person can understand how the results were obtained and what they can support.
At a minimum, publish the list or categories of buying scenarios, how those scenarios were selected, the assistants tested, the test dates, the number of repeated runs and the method used to classify each answer. Preserve examples of both successful and unsuccessful recommendations. If a query was edited after seeing an undesirable result, record that too.
Segment results by meaningful context. A travel company may be frequently recommended for weekend trips but almost never for business travel with a strict cancellation policy. One blended visibility score would hide the distinction that matters to the teams responsible for those products.
Then compare the next measurement period against the same baseline. Add new scenarios when buying behaviour changes, but do not quietly replace the old ones. You need a stable reference point to tell whether the answers have changed.
The purpose is not to find the prompt that makes your company look best. It is to learn where an assistant includes you in a real decision, where it leaves you out, and what information a customer would encounter along the way.
When the next AI-search screenshot lands in the marketing channel, ask for the full conversation and the repeated results behind it. That is where the useful evidence begins.


