Answer 33 of 50
How A/B Testing Should Guide Web Design Decisions and Iteration
A/B testing should settle design questions that are measurable, matter to the business, and get enough traffic to answer. It guides decisions only when the test is designed and read correctly: a written hypothesis, one primary metric, a sample size fixed before launch (or a tool built for checking early), a check that each variant received the share of traffic you configured, and a clear understanding of what a “significant” result does and does not mean. Without those, a test produces confident-looking noise.
What “95% confidence” means
The usual explanation, that 95% confidence means there is only a 5% chance the difference happened by random chance, is a misreading of the p-value that statisticians have addressed directly. The American Statistical Association’s 2016 statement on p-values is explicit: “P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.” A p-value “is a statement about data in relation to a specified hypothetical explanation, and is not a statement about the explanation itself.”
In test terms: a result at p < 0.05 means that if the two versions truly performed the same, a difference this large would show up less than 5% of the time. It does not mean the winner has a 95% chance of being better. That distinction matters because most ideas do not win. Ron Kohavi and Stefan Thomke, writing in Harvard Business Review in 2017, report that at Google and Bing “only about 10% to 20% of experiments generate positive results” and that at Microsoft as a whole about a third prove effective, a third are neutral, and a third are negative. When real wins are that rare, a share of “significant” results will be false positives.
How many visitors a test needs
Decide the sample size before you start. Evan Miller’s guide to how not to run an A/B test gives a rule of thumb: n = 16 σ² / δ², where δ is the smallest effect you want to detect and, for a conversion rate p, σ² = p × (1 − p). Worked through for a page converting at 5%:
| Smallest change you want to detect | Visitors needed per variant (rule of thumb) |
|---|---|
| 1 percentage point (5% to 6%) | About 7,600 |
| 0.5 percentage points (5% to 5.5%) | About 30,400 |
Halving the effect you want to detect quadruples the visitors you need. For an exact figure with your own power and significance settings, use a calculator such as Evan Miller’s sample size calculator, then convert the result into days of traffic and round up to whole weeks so weekday and weekend behavior are both covered.
Do not stop at the first significant result
Checking results daily and stopping when they look significant inflates false positives. Miller puts it directly: “Repeated significance testing always increases the rate of false positives,” and “The more you peek, the more your significance levels will be off.” Stopping early on a lucky streak is what “peeking” refers to; the apparent win then shrinks or disappears with more data.
Some tools are built for looking early. Optimizely describes its Stats Engine as combining “sequential testing and false discovery rate controls to deliver results that are valid regardless of sample size” in its account of how the engine works. Know which method your tool uses: with a classic fixed-sample test, decide the sample size in advance and wait; with a sequential method, follow the tool’s own stopping rules.
Check the traffic split before reading the result
Before you look at the primary metric, compare the number of users each variant received with the split you configured. Microsoft’s Experimentation Platform team, in a 2020 article on diagnosing sample ratio mismatch, defines a sample ratio mismatch (SRM) as “a statistically significant difference between the ratio of users counted in variants A and B from what was configured before the experiment began” and runs the check on user counts “before the ship-decision analysis is started.” Their advice: “Don’t trust the results of A/B tests with an SRM until you diagnose the root cause.” In the article’s MSN example, a test first showed a drop in engagement because users of the more engaging variant were filtered out of the analysis by a bot detection algorithm; once that was fixed, the result flipped. The causes the team lists range from incorrect bucketing of users to redirects in one variant and incorrect data joins.
Low traffic: test bigger changes, or don’t test
Testing one small element at a time is a slow route on a low-traffic site: a small change that produces a small effect needs a large sample to detect. Two better options:
- Test a bolder variant. Change several elements toward one hypothesis, such as a page rebuilt around a clearer value proposition, and accept that you learn which package works rather than which element.
- Use research that does not need volume. Moderated usability testing, surveys, and session recordings explain why people struggle; they do not need thousands of visitors. Which method fits each stage is covered in when and how to conduct user testing.
On high-traffic pages, testing one change after another (headline, then call to action, then image) can work, but it misses interactions between elements; Kohavi and Thomke caution that including too many variables also makes causality hard to learn, so each test should still answer one question.
Write the hypothesis and the metrics first
A hypothesis states the change, the expected effect, and the reason: “Making the shipping cost visible on the product page will raise checkout starts, because visitors currently discover it late.” Then fix three things before launch:
| Item | What to decide | Example |
|---|---|---|
| Primary metric | The one number that decides the test | Checkout starts per product page visitor |
| Guardrail metrics | Numbers that must not get worse | Revenue per visitor, refund rate, page speed |
| Segments | The few you will examine, chosen in advance | Mobile versus desktop |
Pre-choosing segments matters. If you check enough segments after the fact, one will look significant by chance; treat any surprising segment result as a hypothesis for a follow-up test, not a conclusion.
Beyond conversion rate
Conversion rate is not always the right primary metric. Revenue per visitor captures changes that lower conversion but raise order value, and retention or repeat purchase matters more than first conversion for subscriptions. Leading indicators such as email signups give faster feedback, but only if you have checked from past data that they move with the outcome you care about; otherwise a test can win on the proxy and lose on the business.
What not to test
- Fixes. A broken feature or a mislabeled button should be fixed, not tested.
- Pages without the traffic. If the required sample takes months, use qualitative research instead.
- Ethics and accessibility. Whether to meet accessibility requirements or use truthful copy is a decision, not an experiment. A test that finds a misleading message converts better has found a problem, not an answer; the legal limits on false urgency are covered in CTA button placement and design.
- Strategy. A change of business model or audience needs a different kind of evaluation.
Keep the learning
Record every test, including the losers: hypothesis, dates, sample, result, and what you will do next. A documented negative result stops the same idea from being retested next year, and a run of losses on one hypothesis tells you the underlying assumption may be wrong. Prioritize the backlog with a simple scoring method, such as rating each idea’s impact, confidence, and ease, so the order is argued on criteria rather than seniority.
Write the hypothesis and primary metric first, compute the sample size, wait for it (or use a sequential method properly), check that the traffic split matches what you configured, expect most ideas to lose, and record every result so the next decision starts from evidence.