When you ask ChatGPT or Gemini to recommend the best CRM for a startup, how much should you trust that the same question asked five minutes later will produce the same answer? A new study from Pressfront suggests the answer is: trust the top picks, but not the full list.
The research team asked ten open-ended "best X for a small business" questions to both ChatGPT and Google Gemini, three times each, on the same day. That produced 60 total responses across categories like accounting software, email marketing services, and website builders. They then extracted every named business from each response and measured how much the recommendations overlapped between repeated identical queries.
The Top Picks Are Stable. Everything Else Rotates.
The headline number is 69.5% overall consistency, measured as mean pairwise Jaccard overlap. In plain terms, about one in three recommended businesses changed when the same question was asked again minutes later. But the average hides the real story.
The first one to three businesses recommended for a given question were remarkably stable. They appeared across every repetition. The fourth, fifth, and sixth positions, however, shuffled freely. This creates a structural disadvantage for smaller and newer companies. The spots they could realistically reach in an AI recommendation are exactly the ones that change the most between runs.
Fifty-six percent of all named business slots appeared in every repetition for their question. Twenty-seven percent appeared in only one of three. The tail is volatile. The head is not.
ChatGPT and Gemini Behave Very Differently
The two assistants showed a large gap in consistency. ChatGPT with web search enabled averaged 87.2% consistency across questions, with a standard deviation of 17.5 and a range from 59% to 100%. Gemini with search grounding averaged only 51.9%, with a tighter standard deviation of 11.8 but a range from 33% to 67%.
That is not a trivial difference. A business recommended by ChatGPT is likely to stay recommended across repeated asks. The same business recommended by Gemini might appear once and vanish the next time. For anyone trying to understand how AI-powered search affects their visibility, these numbers suggest the platform choice matters as much as the content.
The Methodology Is the Point
What makes this study worth paying attention to is not the sample size. Ten questions and two assistants is small by design. The value is in the transparency. The full dataset, including every response's complete text, is released as a single JSON file. The analysis script is a single Node.js file with no dependencies that regenerates every published number. The data dictionary defines every column. Everything is reproducible.
The team also made deliberate methodological choices. They used open-ended "best X" questions rather than head-to-head comparisons, because comparison prompts can only return the two names you give them, inflating apparent consistency. They excluded responses that named fewer than two businesses. They validated that all 290 extracted business names appear verbatim in the source responses, eliminating the risk of hallucinated entries.
The study was conducted by Pressfront, a commercial entity with a stated interest in the subject. The authors acknowledge this directly and argue that releasing the full method and data is the appropriate response. If you disagree with their numbers, you can rerun the analysis yourself or extend the study with more questions, more repetitions, and more assistants.
What This Means for Businesses
If your business depends on being recommended by AI assistants in search, the consistency data has practical implications. The stable core suggests that established, well-known brands have a durable advantage. They appear reliably across repeated queries. The volatile tail suggests that for less prominent businesses, AI recommendations are essentially random at the individual query level.
This also raises questions about how much weight to put on any single AI recommendation. If the same question asked twice produces different answers thirty percent of the time, a user's experience of AI search is partly a function of luck. That is not how search engines have traditionally worked, and it is worth understanding as these tools become more central to how people find products and services.
The data and code are released under CC BY 4.0. The preprint is available on Zenodo with DOI 10.5281/zenodo.22738861. For anyone building products that interact with AI-powered search or advising clients on visibility, this is a useful baseline to build on.