Why Does AI Give A Different Answer To The Same Question, And How Consistent Are Its Recommendations?
A coffee machine retailer asks ChatGPT in the morning which espresso machines it would recommend for home use, sees their own brand in third place and sends the screenshot to the team. In the afternoon a colleague asks the same question; the list is different and the brand is gone. Neither of them saw it wrong. This piece draws on research to explain why AI assistants give different answers to the same question, how consistent their brand recommendations are, and how often the facts and sources they present turn out to be wrong. It is also one of the deepest differences between Search Engine Optimisation (SEO) and Generative Engine Optimisation (GEO): you can see a search result by looking once, but you can only get to know an AI answer by asking again and again.
An Answer Is Not Written Once, It Is Generated Each Time
A search engine results page is a ranked list pulled from an index at a given moment; two people running the same search at the same hour see largely the same page. A language model does not work that way. It builds the answer piece by piece, computes the probability of the next piece at each step and, in most products, chooses among those probabilities. The same question is a text rewritten every time it is asked. A sentence that starts with a slightly different word can end up somewhere else entirely a few lines later.
The first objection is obvious: if we switch off the randomness, won’t the answer settle? Research shows that this does not fully hold either. In a study where five models were run ten times on each of eight tasks with settings meant to be deterministic, accuracy moved by up to 15 percent between runs, and the gap between the best and worst possible result reached up to 70 percent1. In a 2025 experiment by Thinking Machines Lab, an open model answered the same prompt a thousand times with randomness set to zero and produced 80 different texts; the answers stayed identical for the first 102 pieces and diverged at the 103rd2.
The authors trace the main cause to the server’s load at that moment: as the number of requests processed together changes, the calculations come out with tiny numerical differences, and those differences are sometimes enough for a different word to be chosen2. For the user the meaning is simple. You cannot know how many other people are asking at the moment you ask, and your answer partly depends on it. Add chat history, personal settings, silent model updates and whether the assistant searches the web at that moment, and two users seeing the same screen becomes the exception.
How Often Do Brand Lists Repeat
For a business, this technical detail is felt most in brand recommendations. Rand Fishkin of SparkToro and Patrick O’Donnell of Gumshoe.ai asked around 600 volunteers to run 12 different recommendation prompts through ChatGPT, Claude and Google’s AI answers over the last two months of 2025, collecting 2,961 responses in total3. The prompts ranged from chef’s knives to headphones, from cancer care hospitals to digital marketing consultants.
The finding is plain: ask ChatGPT or Google’s AI the same question a hundred times and the chance of seeing the same list of brands twice is under one in a hundred, while seeing the same list in the same order is closer to one in a thousand3. Claude was slightly more likely to repeat a list, but the difference was small3. Even the length of the lists changed from answer to answer.
The less discussed side of the study is more instructive. While lists and order shifted, some brands kept appearing in most answers: the four leading brands in the headphone prompts showed up in 55 to 77 percent of responses, and one hospital appeared in 69 of ChatGPT’s 71 answers but was named first in only 25 of them3. It is also worth keeping in mind that the study is not peer-reviewed academic work, that the research partner builds a tool for tracking AI visibility, and that the authors call for repeat studies with larger samples3.
Position Is Noise, Visibility Rate Is Signal
This changes what the screenshot in your hand means. A furniture manufacturer’s marketing manager who sees ChatGPT recommend their brand first will want to present it to management as a win. Yet given the data above, a customer who asks the same question an hour later and never sees the brand is not surprising at all. A single answer is not a survey; it is one question put to one person.
This is where Fishkin lands too: he considers reporting ranking positions in AI meaningless, while he treats the visibility rate, the share of answers a brand appears in, as a reasonable measure, but only when dozens or even hundreds of prompts are run multiple times3. That runs against a habit inherited from classic search. In SEO, the question “what position are we in” had a clear answer. In GEO, the question to ask is “in how many of a hundred attempts are we named, alongside which competitors, and what is said about us”.
Consider a software company. A small business owner looking for bookkeeping software can put the same need to an assistant in three ways: the best program, one that suits their size, or one with e-invoicing support. Each question produces its own distribution of answers. The company’s real position emerges from looking at all of those distributions together, not at one answer to one question.
Can It Be Verified Even When It Cites A Source
Inconsistency is one side of the matter; the other is whether the answer is right. Assistants that provide source links look more trustworthy at first glance, because a footnote sits next to the claim. In 2023 Nelson Liu, Tianyi Zhang and Percy Liang of Stanford had human annotators examine answers from four generative search engines. Only 51.5 percent of sentences were fully supported by the sources given, and only 74.5 percent of citations actually supported the sentence they were attached to4.
A 2025 test by Klaudia Jaźwińska and Aisvarya Chandrasekar of the Tow Center, published in the Columbia Journalism Review, paints a more recent picture. They gave eight AI search tools excerpts from news articles and asked for the source; more than 60 percent of the 1,600 queries were answered incorrectly5. Error rates ranged from 37 to 94 percent depending on the tool, and 154 of one tool’s 200 source links led to error pages5.
For a business the meaning is concrete. Even when the assistant names your brand, a product feature, the founding year or the area you serve can be reported wrongly, and the footnote can make it look as if the error came from your own site. A law firm whose practice areas get mixed up, or a manufacturer named alongside a product it dropped years ago, is held responsible for it in the customer’s mind.
Why Fabrication Is Not An Exception
The literature calls these errors hallucination. The comprehensive survey by Ji and colleagues, published in ACM Computing Surveys, documents the tendency of language generation systems to produce fluent text that is not grounded in the source or in fact, task by task, across summarisation, question answering, dialogue, translation and more6. The problem is not a flaw of one particular product but a known property of this family of technology.
A 2025 paper by Kalai and colleagues offers a plain explanation of why: in training and evaluation, models are encouraged to guess rather than say “I don’t know”, like a student who marks an option instead of leaving it blank on an exam where a wrong answer costs no points7. Seen this way, it is not surprising that a model facing scarce and scattered information about your brand makes up a plausible-looking detail.
Put the answer that is generated afresh each time next to the detail that is occasionally fabricated, and the picture becomes clear. A wrong sentence about your brand does not appear in every answer; it may show up in two of ten attempts and vanish in the other eight. A single check will not catch it, and even if it does, it cannot tell you how often it happens.
Why One Measurement Is Not Enough
What everything above has in common is this: visibility in AI is not a photograph but a distribution. And you can only get to know a distribution through many observations. Unless the same question is asked several times, in different wordings, on different assistants and on different days, the data in your hands does not go beyond an impression.
Repeated measurement separates three things. First, how often the brand is actually named: a rate instead of a single lucky or unlucky answer. Second, what is said when it is named: right, incomplete or wrong, and if wrong, whether occasionally or every time. Third, change over time. Models are updated, web-based answers are fed by new pages, and last month’s picture may not hold this month.
The head of an education provider can look once at how the institution is described by assistants and either relax or panic. Both reactions come from the same mistake: treating a single answer as the truth itself. The shared message of the research cited here is that these systems’ answers should be read not as a one-off verdict but as the results of a repeated experiment135.
References
- Berk Atil, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, Breck Baldwin (2024). Non-Determinism of “Deterministic” LLM Settings. arXiv preprint 2408.04667.
- Horace He, Thinking Machines Lab (2025). Defeating Nondeterminism in LLM Inference. Thinking Machines Lab, Connectionism.
- Rand Fishkin (SparkToro), with Patrick O’Donnell (Gumshoe.ai) (2026). NEW Research: AIs are highly inconsistent when recommending brands or products; marketers should take care when tracking AI visibility. SparkToro Blog.
- Nelson F. Liu, Tianyi Zhang, Percy Liang (2023). Evaluating Verifiability in Generative Search Engines. Findings of EMNLP 2023 (arXiv 2304.09848).
- Klaudia Jaźwińska, Aisvarya Chandrasekar (2025). AI Search Has a Citation Problem. Columbia Journalism Review, Tow Center for Digital Journalism.
- Ziwei Ji, Nayeon Lee, Rita Frieske, et al. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys 55(12), Article 248.
- Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, Edwin Zhang (2025). Why Language Models Hallucinate. arXiv preprint 2509.04664.
How Is Your Brand Named In AI
In a free pre-scan, let’s look together at how your brand is named across a few assistants and repeated questions.