Last month, in a post about running a company where everyone is a builder, I mentioned something one of my cousins told me about the company where they work. People there have built a set of Claude skills to handle product discovery so when someone has an idea, the agent pulls in competitor activity, internal analytics, the voice-of-customer database and a set of synthetic user profiles. It then judges whether the idea is viable, and produces clickable wireframes of the proposed feature in a matter of minutes. I haven't seen the tool myself, so I'm passing it along the way it was told to me. The thing that made me laugh out loud (lol in the parlance of our times) was that the agent always finds the idea viable.
I used that as a throwaway line at the time, but it kept coming back to me, partly because a version of the same concern came up in the Q&A of our September webinar on what product management will look like in 2027. A UX research veteran in the audience asked whether the people pushing AI adoption inside their companies realize that synthetic users can't validate anything for real humans. My answer then was that none of these tools read anything directly from a human being, and that is still my answer. So let's look at why the synthetic user always says yes, where it does earn a place in your process, and what should inform a release decision instead.
Why the synthetic user always says yes
In 2024, Maria Rosala and Kate Moran at Nielsen Norman Group took three studies they had already run with real participants and ran them again with synthetic participants using a commercial synthetic-user tool and ChatGPT. In a study of self-paced online learning, most of the real participants said they stayed away from course discussion forums because the interactions felt contrived and unhelpful. Meanwhile the synthetic participants described forums as an important part of learning online and spoke about them warmly. When the synthetic learners were asked whether they had finished their courses they said they had, which is a lovely answer. It also bears no resemblance to the real participant who had finished three courses out of seven. The researchers put the gap down to the models' tendency to please. Their summary of the whole exercise was that "synthetic users seem to care about everything," which makes them close to useless for the one thing a product team needs most from research, a sense of what matters more than everything else.
None of this should surprise anyone who has spent more than an afternoon with these tools. A language model is built to be helpful and agreeable. A synthetic persona is that same model wearing a fake name tag, assembled from what has already been written about people like your customers. This means it can tell you what is already known about them and nothing about the things they do that nobody wrote down in those publications the LLM used to train. The forum finding is the perfect example, because the gap between what people say they value and what they actually do is exactly the surprise that real research exists to find. A synthetic user can't surprise you in that way by design. Put that persona inside an agent whose job is to assess viability, feed it an idea someone on the team is already excited about, and you get my cousin's company, where every idea passes and the wireframes arrive a few minutes later to make everything AI look like progress.
A synthetic yes deserves a prototype at most
In the builder post I argued that companies need to separate the right to build from the right to ship. I argued that anyone can make a prototype while exposure to real customers grows only as the builder brings evidence for further customer exposure. Synthetic users fit comfortably into the cheapest part of that path. The Nielsen Norman Group team lists the same kinds of supporting work I would use synthetic users for. For example , you can use them for getting up to speed in an unfamiliar domain, drafting a rough persona you fully intend to replace with real data, deciding which topics an interview should cover, and piloting your interview guide before you spend a real customer's hour on it. If a synthetic user helps a builder walk into a real conversation with a sharper question, it has done its job well.
Every gate after that first one is a decision about what real people will experience. The only evidence that should move a feature through those gates is what real people actually did when they used it. Because they tend to agree with every idea, I would go a step further and keep synthetic findings out of the release conversation entirely unless they are clearly labeled as the hypotheses they are.
This matters more now than it did two years ago because building got dramatically faster and getting feedback did not. A team that can produce a working feature in an afternoon will lament (or at least be pressed to hurry) every day of the week it takes to recruit and talk to five customers. That pressure is precisely what makes a tool that answers in minutes so tempting. The way out is to make real feedback faster, which is something AI happens to be good at. At the same September webinar, a product leader pointed out that voice of the customer used to be a game of telephone that had diluted the signal by the time it reached the product manager. AI can now pull that real signal together without the PM having to be in the right room at the right time. That is the version of AI-assisted discovery I'm excited about, because real people are still the source and the machine handles the gathering, sorting and summarizing.
What to change before your next release decision
Three small changes will keep synthetic evidence where it belongs without slowing your builders down.
Label where every piece of evidence came from. In any hypothesis, review or release document, mark each finding as either observed, meaning a real person did or said it, or generated, meaning a model produced it. Put that label on the slide where everyone can see it instead of in an appendix nobody reads. Once a leader can see that four of the five reasons to ship came from synthetic profiles, the conversation tends to change on its own (hopefully).
Write down who you actually asked. Add one line to your release checklist that names the real people whose behavior supports the decision, by segment and by number, along with what each of them did. If that line is empty, the feature stays at its current stage of exposure until someone fills it in, no matter how good the wireframes look.
Point your AI tools at real customers. Use the same agent skills your teams are building to recruit participants, draft and pilot interview guides, transcribe and synthesize sessions, and mine the support tickets and sales calls you already have. This helps the time between building something and hearing from a real customer shrinks from weeks to days. When real feedback is nearly as fast as the synthetic kind, most of the argument for letting the synthetic user make the call goes away.
If your company has its own version of my cousin's agent, try the labeling exercise on your next three release decisions and count how many of the reasons to ship were generated by a model and how many came from people you watched use the product. I'd love to hear what you find, especially if the number surprises you.
One more thing. If the question of what AI changes about research (and what it can't replace) is on your mind, Josh Seiden and I are hosting Jen Cardello, who has spent more than two decades building and leading UX research teams at companies including Fidelity and Capital One, for a free webinar called From UX Research to UX Intelligence on Tuesday, November 17 at noon Eastern (17:00 UTC). If you can't make it live, register anyway and we'll send you the recording. Save your seat at webinar.senseandrespond.co.