Understanding the Opportunities and Risks with Synthetic Users in User Research

Chances are you’ve heard of synthetic users by now and are curious when they might be useful. Synthetic users are AI-generated simulations that can imitate user groups through methods like interviews, surveys, and prototype testing. They can offer benefits that traditional user research with humans can’t match, such as speed, scalability, and a potentially budget-friendly way to gain user knowledge. As appealing as those benefits are, we believe synthetic users work best alongside human research, not in place of it.

Key limitations to be aware of include:

• People-pleasing tendencies: it’s been well-documented that LLMs have sycophantic tendencies; they like to give you the answers you want, which can be detrimental to user research and reinforce biases.
• Reduced critical nuance or context: AI is a great averaging machine, which means it misses the depth, complexity, and context you will receive from actual humans. For example, synthetic users may “worry about monthly expenses” vs. actual humans who admit to avoiding opening their bills.
• Attitudinal, over behavioral: a lot of current generative AI is trained on the data that’s available online, which means it often relies on pre-existing verbal and attitudinal data. It cannot predict or interpret nonverbal human behavior. This may change over time, but you should consistently stay aware of your model's training data.

✅ When to Use Synthetic Users

Knowing their limitations, when does it actually make sense to bring synthetic users into a project? 

Forming background understanding of an audience

Let’s imagine you and your team are building out a website for an audience that is brand new to you. Utilizing synthetic users or data can help form initial hypotheses, or gain better understanding of a new subject matter, before conducting any user research with actual humans. 

✅ A good scenario could look like utilizing synthetic users to build a baseline knowledge of the agricultural space, like a farmer’s workflow, tools, and financing. This way, you and your team can walk into human interviews with better informed questions.

Narrowing the scope of upcoming human research

Synthetic users can also help narrow any initial questions, so that any upcoming human research is more focused. If you and your team have so many questions about users, but only so much time with participants, you can ask synthetic users first, then refine what you want to ask actual humans. 

✅ A good scenario could look like using synthetic users to map which parts of an insurance benefits quote flow might be most confusing, then honing in on those key moments for upcoming human research. 

Vetting questions and language

You can run your interview questions, discussion flow, survey questions, and more by synthetic users to see what wording might be awkward, unclear, leading, or sensitive. This can be handy especially when you’re working in a space that is jargon-heavy.

✅ A good scenario could look like pilot-testing survey questions with synthetic users for a hospital's patient-portal redesign. Synthetic users could flag terms like "in-network referral" that might not be understood by actual patients.

Generating strong hypotheses before concept testing

Imagine you and your team have 10-15 ideas for a new feature. Testing with synthetic users may help you weed out ideas, clarifying the best ones and giving you better hypotheses to test through human research. 

✅ A good scenario could look like using synthetic users to anticipate objections to a new subscription-based meal kit service (i.e., “what if I’m traveling?”) Those responses can turn into hypotheses to probe for in future human interviews.

Conducting usability testing or heuristic evaluations

A newer category of AI tools that specialize in usability and heuristic audits with synthetic users is emerging. These tools could be used as a precursor to testing with humans to catch more obvious problems, so human research can focus on more complex issues. Be aware of what tools to use here; the tools for this type of testing will be trained on different types of data (i.e., eye-tracking, previous usability tests). 

✅ A good scenario could look like putting a checkout flow prototype through an AI usability tool to catch inconsistencies and violations (i.e., poor contrast) first, then reserving human research for more nuanced questions, like why people abandon their carts.

⚠️ When It’s Risky

Leaning heavily on synthetic users can be risky for your product or experience. When is it important to pause and consider other research approaches?

Learning about a brand new audience 

You don't know much about your audience to begin with and you’re utilizing synthetic users as your only touchpoint. If you don’t know what you don’t know, how will you be able to detect hallucinations or sycophantic behavior? You could be building on a skewed foundation, one that could have risky consequences later on for your product or website.

⚠️ A risky scenario could look like designing a fintech app without any background knowledge of how retail investors navigate the stock market today. Synthetic users can give you initial background information, but your team needs to learn from humans as you’re building out the app.

Understanding how users actually behave, not what they think 

You and your team may need to understand how people will behave when designing for a future product and experience. If you ask synthetic users about their “behavior,” the responses might be dangerously enthusiastic, overly positive, or reduced in critical nuance or context. This goes back to the type of data that most LLMs are trained on today. You need to know what people will actually do, but the data you’re leveraging is attitudinal.

⚠️ A risky scenario could look like designing a new feature in a wellness app about building habits over time. Synthetic users may say daily reminders are enough, while actual human behavior is more complex.

Vetting a new-to-world or blue sky concept

You and your team have an idea that is disruptive, something that doesn’t quite exist yet. You need to know how this new idea fits into people’s lives. You could run your idea by synthetic users, but synthetic users can’t be surprised, confused, or frustrated. They can only be influenced by what already exists in their training data.

⚠️ A risky scenario could look like asking synthetic users about their adoption of a genuinely novel wearable device. Synthetic users can only draw from what already exists, so there’s likely no precedent for synthetic users to draw from.

Designing for a niche audience or on a taboo subject 

You and your team may be looking to design for an audience that is small or marginalized, like people managing a rare disease. Or perhaps your subject area is sensitive or stigmatized, like drug addiction. Your audience may be hard to reach, which makes it tempting to rely on synthetic users. Chances are there isn’t enough publicly available data on your audience. What will come back won’t be lived experience, but problematic biases that will reinforce stereotypes. The harder an audience is to reach, the more their perspective tends to be missing from the products built for them, and reaching for a synthetic user widens that gap rather than closing it. If a group is worth designing for, it's worth designing with.

⚠️ A risky scenario could look like designing a website around gun safety. Utilizing synthetic users will likely not give the depth, tension, or underrepresented perspectives that humans would.

Synthetic users are just one way to bring AI into a user research practice. They can help form background understanding, test out research questions, or generate hypotheses. Here at Viget, their their true value is as a supplement to research with humans, not as a replacement for it. In this way, they can make our human research sharper, more focused, and more meaningful.