How to run an A/B test that means something

Start with a hypothesis tied to evidence

Good tests begin with a reason, not a hunch. A hypothesis is a short, testable statement that links what you saw to what you will change and what you expect to happen. It keeps you honest about what success looks like.

A useful format is: because we saw this evidence, we believe changing this thing for these visitors will improve this metric. For example: because 40% of checkout surveys mention surprise shipping costs, we believe showing the free-shipping threshold on product pages will increase orders per visitor on mobile.

Work out the sample size before you start

Sample size tells you how many visitors each version needs before the result is reliable. Free online A/B test calculators do the math; you supply a few numbers. Smaller expected lifts need far more visitors, which is why tiny changes rarely give clear answers on smaller sites.

1. 01

   Find your baseline conversion rate for the page and metric you will test.

2. 02

   Choose the minimum detectable effect: the smallest lift worth acting on, such as 20% relative.

3. 03

   Use the common defaults of 95% confidence and 80% power unless you have a reason not to.

4. 04

   Read the visitors needed per version, then divide by your weekly traffic to get the run time.

5. 05

   Round up to whole weeks so every day of the week is included equally.

Run the test cleanly

Small mistakes in setup can make a result meaningless. Split traffic evenly and randomly, keep each visitor in the same version on return visits, and run for at least two full weeks so weekday and weekend behavior both count. Do not change the page, the offer or the ad budget mid-test.

Check that each version received roughly the share of traffic you set. A large imbalance, called a sample ratio mismatch, usually signals a technical problem. Google Optimize shut down in September 2023, so most teams now use tools such as VWO, Optimizely, AB Tasty or Convert, and Shopify stores often use testing apps built for themes, prices or shipping offers.

Read the result honestly

Decide the primary metric before launch and judge the test on that metric alone. For stores, revenue or orders per visitor is often better than conversion rate alone, because a change can raise orders while lowering order value. Add a guardrail metric, such as refund rate or lead quality, to catch wins that cause harm later.

Look at the confidence interval, the likely range of the true effect, not just whether the tool calls a winner. Splitting results by device or channel after the fact is useful for new ideas but not for declaring victory. A flat result is still a result: it tells you that change did not matter much, so move on to a bigger idea.

When you do not have enough traffic

If the calculator says your test would take six months, change the approach rather than the rules. Test bolder changes that could produce bigger lifts, or measure a step earlier in the funnel, such as add-to-cart, where there are more events.

You can also learn without a split test. Watching five people try to complete a task, known as usability testing, often uncovers the biggest problems. Careful before-and-after comparisons can work too, as long as you note seasonality, promotions and tracking changes and treat the result as a strong hint rather than proof.