[ Answers · AI search ]

Why does ChatGPT give different answers to the same question?

By Ben Tabas, Intrica Research · Updated

Answer

ChatGPT gives different answers because it generates each one fresh. SparkToro's study, published Jan. 28, 2026, says "These tools are probability engines: they're designed to generate unique answers every time." It ran 12 prompts through 3 tools 2,961 times and found a less than 1 in 100 chance of the same list.

What the studies measured

  • SparkToro reports it is more like 1 in 1,000 runs before two lists come back in the same order.
  • Ahrefs tracked over 43,000 keywords, each at least 16 times over about a month. It reports a 70% chance that AI Overview content changed between consecutive observations.
  • Ahrefs also reports that 45.5% of cited URLs were new from one AI Overview response to the next. Because its checks were not daily, it says the real citation change rate is likely much higher.

How to measure anything useful

  1. Fix the question set and keep the wording identical each time.
  2. Run each engine separately.
  3. Repeat every question several times and count how often your company appears.
  4. Report a range or a bucket such as often, sometimes or never, not a single run.
  5. Compare the same buckets month to month.

How Intrica measures AI answers

Intrica uses fixed questions, runs each engine separately, repeats the runs and reports results in buckets.

See the data →

Check whether ChatGPT and Google AI name or cite your company, then repeat it to see the spread.

Run the free Answer ShareBook a call

Sources

  1. NEW Research: AIs are highly inconsistent when recommending brands or products; marketers should take care when tracking AI visibility · SparkToro, Jan. 28, 2026Supports: 2,961 runs of 12 prompts on 3 tools, the less than 1 in 100 chance of the same list and about 1 in 1,000 for the same order.
  2. 9 AI Search Myths, Debunked by 15 Million Data Points · Search Engine Journal (Ahrefs), Aug. 18, 2026Supports: Ahrefs summary of the 43,000 keyword study: AI Overview content changed 70% of the time and cited sources swapped 45.5%.
  3. AI Overviews Change Every 2 Days (But Never Change Their Mind) · Ahrefs, Nov. 11, 2025Supports: 43,000 keywords, a 70% chance of content change between observations, 45.5% of cited URLs new, and the note that the citation change rate is likely higher.

Related questions

Intrica gets crypto, fintech and AI companies named in AI answers. AI search optimization →