An A/B test is the wrong tool when the page cannot reach a reliable sample size in reasonable time, when the change fixes something that is broken, and when the question is why shoppers behave as they do rather than which version performs better. In those cases, user research, a heuristic review and a carefully monitored release give better answers for less traffic and less time.
How do you know your traffic is too low for a test?
Calculate the sample size before you build anything, using your current conversion rate and the smallest effect worth shipping. The method is explained in How long should an A/B test run? Divide the result by the daily visitors to the tested page. The free readiness check runs the same calculation for four weeks of your whole store.
If the answer is several months, the test is not practical. Visitors lose their group assignment as cookies expire, seasons and campaigns change the audience, and the result arrives after the decision was needed. Testing a bolder change or a page with more traffic can bring the number down. If neither works, choose another method.
Should you A/B test a bug fix?
No. An add to cart button that fails on one browser, a checkout field that rejects valid addresses or a page that takes too long to load should be fixed as soon as the problem is confirmed. Running half of your visitors through a known error for weeks costs revenue to prove something you already know.
The same applies to changes you must make anyway, such as legal requirements, payment provider updates or accurate delivery information. Release them, and monitor the numbers afterwards.
A test does become useful when a problem has more than one reasonable fix. If shoppers abandon a long checkout form, removing a field and splitting the form into steps are two different answers. Testing those answers against each other is sound. Testing the broken version against a fix is not.
What can research show that a test cannot?
A test tells you which version performed better. It does not tell you why, and it cannot test an idea nobody has had. Research fills both gaps. Customer surveys and interviews reveal the doubts and questions that stop a purchase. Session recordings show where visitors hesitate. Usability tests show a real person struggling with a real task.
Usability testing needs few participants. Jakob Nielsen's article Why You Only Need to Test with 5 Users (Nielsen Norman Group, 2000) argues that in his model five participants uncover about 85 percent of the usability problems in a design, and that several small rounds beat one large study. That makes research possible on any store, whatever its traffic.
What is a heuristic review?
A structured walkthrough of the store against established usability principles. The best-known set is Jakob Nielsen's 10 usability heuristics, which include visibility of system status, error prevention and consistency. For e-commerce, published checkout and product page research from sources such as the Baymard Institute adds more specific guidance.
A heuristic review is fast and needs no traffic. Its limit is that it finds likely problems, not measured ones. It is the right way to build a list of candidates and to fix clear errors, and the wrong way to decide between two reasonable designs.
How do you change a page safely without a test?
- Write down the change, the reason and the metric you expect to move, exactly as you would for a test.
- Record the metric for several full weeks before the release, split by device and traffic source.
- Release the change on a known date and avoid launching other changes or campaigns at the same time.
- Compare the same weekdays before and after, and look for movement beyond the normal week-to-week variation.
- Decide in advance what result would make you roll the change back.
A before-and-after comparison cannot separate the change from everything else that happened in the same weeks. It is weaker evidence than a controlled test, and it should be treated that way in the learning log.
Which other changes are hard to A/B test?
- Full redesigns and platform migrations, which change too many things at once to trace a result to any of them.
- Changes whose effect takes months to show, such as effects on repeat purchases or subscription retention.
- Price tests shown to different customers at the same time, which can raise legal and trust questions and need to be checked before launch.
- Changes that affect search engine rankings, which respond to the page Google sees and not to a visitor split.
Does skipping a test mean deciding without evidence?
No. It means using the strongest evidence the situation allows. Volition runs research on every store, uses split tests where a page has the traffic to give a clear answer, and bases the remaining changes on research and usability principles, with close monitoring after release.