How do you run an A/B test?
To run an A/B test, split visitors at random between your current page (A) and one changed version (B). Decide in advance which metric decides the winner and how many visitors you need, then run the test for at least two full weeks without stopping early. A Shopify apparel store with a 2 percent conversion rate needs roughly 21,000 visitors per version to reliably detect a lift to 2.4 percent.
Start with a hypothesis drawn from evidence
Good tests come from something you saw, not a hunch. Look at analytics for drop-off points, watch session recordings, and read customer questions and reviews. Then write the idea as a sentence: because we saw this, changing that will improve this metric. That sentence keeps the test focused and makes the result easy to learn from, whichever way it goes.
Work out the sample size before you start
Use a free sample size calculator. Enter your current conversion rate, the smallest lift worth detecting, and the usual settings of 95 percent confidence and 80 percent power. The example above comes from exactly that math.
If the number is far beyond your monthly traffic, an A/B test is the wrong tool. Make research-backed changes instead and compare before and after over several weeks, knowing the result is less certain.
Run the test cleanly
Most bad results come from how a test was run, not from the idea.
- 01
Change one thing, or one clearly defined bundle of changes.
- 02
Split traffic 50/50 and check both versions on mobile and desktop.
- 03
Run in full weeks so weekday and weekend behavior are both included.
- 04
Do not stop the moment one version pulls ahead; wait for your planned sample size.
- 05
Check that each version received roughly the share of visitors you set; a big imbalance means something is broken.
Read the result honestly
A test has three outcomes: B wins, B loses, or there is no clear difference. No difference is a real result, and it tells you that change was not what was holding people back. Record every test, its hypothesis and its outcome so you do not repeat it. Google Optimize shut down in September 2023, so most teams now use tools such as VWO, Optimizely, Convert or AB Tasty, or a testing app built for Shopify.
Do this next
Follow-up questions
How long should an A/B test run?
Until it reaches the sample size you planned, and for at least two full weeks. Stopping early because one version looks ahead is the most common way tests mislead.
Can I test several changes at once?
You can test a bundle of changes as one version, but you will not know which part worked. Testing many combinations separately, called multivariate testing, needs far more traffic than most sites have.