When and How to Conduct User Testing Throughout the Web Design Process

Test at every stage, with the method that fits the question you have at that stage: interviews and observation before design, paper and wireframe tests while the structure is forming, task-based sessions on interactive prototypes, and analytics, recordings, and A/B tests after launch. Qualitative tests need only a handful of people per user group. Anything you want to report as a number, such as a success rate or a usability score, needs many more.

Which test, at which stage

Stage	Question you are answering	Methods	Participants
Discovery, before design	What problem are users trying to solve, and in what context?	Interviews, contextual inquiry (watching people in their own environment), surveys, reading reviews of existing products	A handful of interviews per user group; surveys need larger numbers to show how widespread something is
Concept, while structure is forming	Does the structure match how users think?	Paper prototypes, wireframes, think-aloud sessions	About five per distinct user group, then fix and repeat
Interactive prototype	Can people complete key tasks, and where do they struggle?	Task-based usability testing; task success and time; the System Usability Scale	Five or so to find problems; around 40 or more to report reliable numbers
After launch	What do real users do at scale?	Analytics, heatmaps, session recordings, feedback prompts, A/B tests	Real traffic; A/B tests need enough of it to reach a planned sample

A/B testing belongs in the last row, not in prototype testing. Nielsen Norman Group’s A/B Testing 101 describes creating variations “in a live product” and notes that the need for large numbers of users “makes A/B testing a bad fit for pages with little traffic.” How to plan, stop, and read an A/B test is covered in how A/B testing should guide web design decisions, and what heatmaps can and cannot show in what heatmap analysis reveals about user behavior. Why testing structure before visual design pays off is the subject of what happens when the wireframing stage is skipped.

Five users, and when five is not enough

The five-user guideline is set out in Jakob Nielsen’s March 2000 article Why You Only Need to Test with 5 Users. Its model assumes that a single user reveals a share of the problems, and “The typical value of L is 31%, averaged across a large number of projects we studied.” On that assumption, a first study with five participants finds about 85% of the problems. The same article adds two conditions that are easy to drop:

Test again. Nielsen argues that three studies with five users each improve a design more than one study with fifteen, because each round fixes what the last one found.
Test each user group. “You need to test additional users when a website has several highly distinct groups of users.”

Results also vary from one group of five to the next. In Laura Faulkner’s 2003 study, published in Behavior Research Methods, Instruments, & Computers, 60 users were tested and random sets were drawn from them: “Some of the randomly selected sets of 5 participants found 99% of the problems; other sets found only 55%.” With 10 users, the lowest share found by any set rose to 80%, and with 20 users to 95%.

Numbers need larger samples still. Raluca Budiu’s 2021 article on qualitative versus quantitative testing states that “the 5-user guideline only applies to qualitative, not to quantitative studies” and that confidence intervals “usually are narrow only if you include a large number of participants (40 or more).” The same applies to the System Usability Scale: MeasuringU’s guide to SUS reports an average score of 68 across 500 studies, but warns that “small sample sizes generate imprecise estimates of the unknown user-population SUS score” and that a SUS score, although it ranges from 0 to 100, is not a percentage. With five to eight participants, report task success and SUS as directional findings, not as a benchmark against competitors.

Running a test session

Set one goal for the study, such as “can first-time visitors book an appointment?”
Write five to eight tasks as realistic scenarios. Kate Moran’s Usability Testing 101 describes tasks as “realistic activities that the participant might perform in real life.” Write “You need to change the delivery address for tomorrow’s order,” not “Click Account, then Addresses.”
Facilitate neutrally. Ask participants to think aloud, and do not explain or help when they struggle; the struggle is the finding.
Record where each person fails, hesitates, or takes a wrong path, and how long each task takes.
Rate every issue by severity before reporting. Jakob Nielsen’s severity scale weighs how frequent a problem is, how hard it is to overcome, and whether it persists, and runs from 0 (“I don’t agree that this is a usability problem at all”) to 4, a usability catastrophe that must be fixed before release.
Fix the most severe issues and test again with a new set of participants.

Recruiting the right participants

Testing with the wrong people can produce misleading results. Define the characteristics that matter for your product, such as experience with similar products, domain knowledge, or the device they use, and write screening questions that filter for them. Include people with disabilities and people from the different groups you serve; if you test with one group, you learn about one group. Set incentives with care: too low and you may hear only from the most motivated, too high and you may attract people more interested in payment than in the task.

Testing on a small budget

Discount methods. Jakob Nielsen’s discount usability approach, launched in 1989, combines simplified user testing with a handful of participants, a focus on qualitative findings, and thinking aloud.
Remote unmoderated tools. Platforms such as UserTesting, Maze, and Lyssna, which replaced the UsabilityHub name in October 2023, recruit participants and record sessions without a researcher present for each one. You lose the chance to ask follow-up questions, so use them when you know what you need to learn.
Short in-person sessions. Asking people in a public place for five minutes is informal and uncontrolled, but it catches obvious problems.
Colleagues. They are not your users, so use them only for early sanity checks.

If you can run only one thing this week, run three short sessions on the task that matters most to the business.

Communicating findings

Research that nobody acts on is wasted. Turn findings into a short list ranked by severity, with a recommendation for each. Short video clips of real participants struggling can persuade stakeholders more directly than a written description. Connect problems to business measures where you can, for example, “the checkout issue affected 4 of 6 participants and sits on the page where we lose the most orders.” Then follow up: check whether the changes were made and whether the next round shows improvement.

Mistakes to avoid

Testing too late, when fixes require rebuilding rather than redrawing.
Testing only with friends and family, who may want to be kind and may not match your users.
Leading questions such as “Don’t you think this button is hard to find?” Ask what they expected instead.
Defending the design during the session.
Testing once. A single round tells you what is wrong; the next round tells you whether your fix worked.

Questions teams ask

We don’t have budget for professional recruiting. What can we do?

Use the channels you have: customer lists, social media, and community forums. Offer a small incentive, be clear about the time commitment, and screen carefully. A few well-matched participants can tell you more than many mismatched ones.

Is it possible to test before the product exists?

Yes. Test concepts with whatever fidelity you have: paper sketches for concepts, clickable wireframes for flow. Even a description can be tested: “Imagine a product that does X. How would you expect it to work?”

Our stakeholders don’t believe in user testing. How do we convince them?

Invite them to watch a session. Seeing a real user struggle with something the stakeholder thought was obvious can change minds faster than any report.

Is remote unmoderated testing worth it?

For the right questions, yes. It scales, avoids scheduling, and reaches a wider geographic range, but you cannot probe or ask follow-up questions. Use it to check specific tasks, and use moderated sessions for exploratory research.

Match the method to the stage, test about five people per user group to find problems and around 40 or more when you need numbers, write realistic task scenarios, rate what you find by severity, and test again after every fix.