Last updated: October 11, 2026
A popular shortcut for testing a side hustle idea: ask an AI to play your customers. A growing number of tools now sell it as market research.
On 30 September, the Pew Research Center published the most thorough test of it I’ve seen. It built AI stand-ins for thousands of people on its survey panel, then reran three recent surveys on them: nearly 300 questions the real people had already answered.
Each stand-in got that person’s profile, from age and education to income and religion, plus 71 of their answers from an earlier Pew survey. That’s more than you could tell a chatbot about a customer you haven’t met.
The AI poll’s results were still off by an average of 12 percentage points.
The unusual answers disappear
The average hides a bigger problem. In the three real surveys, every answer option on every question was chosen by someone. In the AI version, 47% of questions had at least one option that no stand-in chose.
A travel question shows how. The AI got most answers roughly right, then said only 1% of Americans have visited 10 or more countries abroad. In the real survey, 13% said they had: about one person in eight, not one in a hundred.
Sleep went the same way. The AI said nobody sleeps four hours a night or less; 5% of the real people said they do.
That fits how these tools work. Each stand-in tends to give the most typical answer for someone like the person it copies, so rarer answers get squeezed out.
Now think about who a side hustle sells to. Rarely the majority — usually a small group with an unusual problem. That’s the kind of answer this method shrinks, sometimes to nothing.
Why other research sounds more hopeful
Researchers at Harvard Business School and Microsoft have tested AI on a business version of this job since 2023: estimating what shoppers will pay. The latest version of their paper finds its estimates “sometimes comparable” to studies of real people, but often inaccurate — in some cases saying a feature would make people pay more when the human studies said less, or the reverse.
Feeding the AI earlier survey answers from real people helped, even for new features, as long as the product and the buyers were the same kind as before. It didn’t help with a new kind of product, or with differences between groups of customers.
So it works best with a familiar product, earlier answers from real people, and a small change. A new side hustle aimed at a small group usually has none of the three. The authors suggest AI’s place is as “a supplement to — not a substitute for — human studies.
What I’d use it for instead
Pew found two more habits that matter here. Its stand-ins answered factual questions correctly about 80% of the time; the real people managed about half. And on opinion questions that offered “not sure”, people picked it 16% of the time, the AI just 4%.
So a pretend customer probably knows more than a real one, and is surer of itself. With real customers, “not sure” is often the most useful answer: it shows what your pitch hasn’t made clear.
Use AI for the jobs that don’t need real opinions: drafting your questions, listing likely objections, tightening the wording. Then ask real people, and pay closest attention to the answers only a few of them give.
If you have an email list, start there. I’ve written about the signals worth waiting for before you sell to a list. None of them can come from a pretend customer.
