An A/B test should run until it reaches the sample size you calculated before launch, counted in full weeks and never less than two. The length follows from four inputs: your current conversion rate, the smallest effect worth detecting, your traffic and the statistical standards you set. A calendar rule such as "one week" or "until it looks significant" leaves the answer to chance.
What decides how long a test needs?
The number of visitors a test needs depends on four things, and the run time is that number divided by your daily traffic to the tested page.
- Baseline rate: the current conversion rate of the page or step you test. Rare events need more visitors to measure.
- Minimum detectable effect: the smallest change that would be worth shipping. Halving it roughly quadruples the sample you need.
- Significance level: the risk of calling a winner that is not real, commonly set at 5 percent.
- Power: the chance of detecting the effect if it is real, commonly set at 80 percent.
All four are fixed before launch. Changing any of them after the data arrives turns the test into a search for a convenient answer.
How do you calculate the sample size?
A widely used rule of thumb, given in Trustworthy Online Controlled Experiments by Kohavi, Tang and Xu (Cambridge University Press, 2020), sets the sample per variant at about 16 times the variance divided by the square of the effect you want to detect. It assumes a 5 percent significance level and 80 percent power. For a conversion rate p, the variance is p times (1 minus p).
A worked example, with numbers chosen for illustration only. A product page converts at 3 percent and you want to detect a relative lift of 10 percent, so from 3.0 to 3.3 percent. The absolute effect is 0.003. The variance is 0.03 times 0.97, which is 0.0291. Sixteen times 0.0291, divided by 0.003 squared, gives about 52,000 visitors per variant, so roughly 104,000 in total.
If that page receives 4,000 visitors a day, the test needs 26 days. Rounded up to full weeks, it runs for 28 days. Any sample size calculator in a testing tool uses a more exact version of the same formula, and its result lands close to this one.
Why should a test run in full weeks?
Shoppers behave differently on a Monday morning than on a Saturday evening. Email sends, paydays and weekend browsing all shift who arrives and how ready they are to buy. A test that covers four weekdays and no weekend measures a slice of your audience and applies the result to all of it.
Start and stop the test on the same weekday, and run it for at least two full weeks even when the sample size is reached sooner. A second week also dampens the novelty effect, where returning visitors click on something simply because it is new.
Why is it a mistake to stop when the result looks significant?
A significance level only holds if you look at the result once, at the planned end. Checking every day and stopping at the first significant reading gives chance many opportunities to cross the line. Evan Miller shows in How Not To Run an A/B Test that a test with a nominal 5 percent significance level can produce false positives at 26.1 percent or more, depending on how often it is checked.
Watching a running test is fine for spotting broken tracking or a variant that fails on one browser. Deciding on it is the problem. If you need to stop early for business reasons, use a sequential testing method built for repeated looks, which some testing tools offer, and choose it before launch.
Can a test run too long?
Yes. The longer a test runs, the more visitors lose the cookie that assigns them to a group. Safari, for example, caps cookies set by scripts at seven days, as documented in WebKit's Intelligent Tracking Prevention 2.1 notes. A returning visitor may then see the other variant, which blurs the difference you are trying to measure.
Long tests also collect more outside noise: a sale, a new ad campaign, a stock problem. And every extra week spent on one idea is a week the next idea waits. If the calculated run time stretches past a few months, the plan needs to change rather than the patience.
What can you do if your traffic is too low?
- Test where traffic is highest, such as sitewide elements or the cart, instead of a single product page.
- Test bolder changes. A larger expected effect needs a far smaller sample.
- Measure a metric closer to the change, such as add to cart instead of completed orders, and check revenue as a guardrail.
- Skip the test and use research and a monitored release, as described in When is an A/B test the wrong tool?
What should be written down before the test starts?
The hypothesis, the primary metric, the audience, the minimum detectable effect, the significance level and power, the sample size, the planned end date, and what happens for each outcome. Volition fixes all of these in a test plan before the first visitor sees the variant, so the end of the test is a lookup and not a debate.