Your AI Tool Is Hallucinating Your Data Analysis — Here’s Why Default Mode Is the Problem

When mathematician Adam Kucharski fed Microsoft Copilot two identical datasets — one labeled “UK,” one labeled “US” — the AI returned a detailed breakdown of how the two groups differed. The problem? The data was exactly the same. Copilot had simply invented cultural stereotypes and reported them as findings.

This isn’t a fringe edge case. It’s a direct consequence of using AI tools on their default settings.

What the Experiment Revealed

Kucharski created 2,000 simulated responses about emotions and duplicated them for both countries. Copilot, running in “Auto” mode, confidently described tonal and stylistic differences between UK and US respondents — differences that couldn’t possibly exist because the data was identical. In a follow-up experiment using career goal statements duplicated across five countries, Copilot claimed Italians were three times more likely than Brits to be interested in arts careers. Again, the underlying data was the same for all five groups.

When Copilot ran a simple keyword count and got identical results, it ignored its own finding and substituted fabricated percentages instead.

Key Takeaways

• “Auto” mode doesn’t mean “best mode.” Microsoft’s Auto setting is supposed to select the optimal model for each task. In data analysis tasks, it consistently chose fast models that hallucinate rather than reasoning models that actually process the data.
• Thinking models handle this correctly — but only if you switch to them. When Copilot and Gemini were manually switched to their reasoning models, they caught the duplicate data. ChatGPT and Claude automatically triggered extended reasoning and wrote Python code to verify the data.
• Most users never switch models. The majority of enterprise Copilot users are running the default version that produced these errors — meaning fabricated analyses may be flowing into real business decisions right now.

The fix is straightforward: know your tools well enough to choose the right model for the task. Write down your expected result before running analysis. Run basic sanity checks. Never trust AI-generated data analysis without verifying it yourself.

Read the full article on The Decoder