How does Volition decide what changes on your store?
Every change starts as a hypothesis and ends as a decision. In between sits a test against a control group, run by rules that are written down before the first visitor arrives.
The six steps
- 01
Research
Find where shoppers hesitate, and why.
Analytics, funnel data, session recordings, heuristic reviews and customer surveys show where visitors drop off. Each finding is written down with its evidence, so later decisions can be traced back to something real.
- 02
Prioritise
Rank ideas by expected value and effort.
Every idea is scored on how much revenue it could move, how strong the evidence is and how much it costs to build. The pages with the most traffic and the clearest problems go first.
- 03
Hypothesis
Write the change, the metric and the reason.
Each test starts as one sentence: if this changes, then this metric improves, because shoppers will behave differently in this way. The reason is the important part. Without it a test can win and still teach nothing.
- 04
Test
Run the change against a control group.
Half of the visitors see the current page and half see the change, at the same time. The main metric, the audience, the confidence level and the minimum run time are fixed before launch, and the variant is checked for errors on every device.
- 05
Decide
Ship, keep or rethink, by the rules set in advance.
When the planned run time is over, the result is read against the rules written before launch. A clear winner ships. An unclear result keeps the current version, because shipping a change that only looked better is the expensive mistake.
- 06
Learn
Record what the result says about your shoppers.
Every test goes into a learning log with its hypothesis, its numbers and its decision. Over time the log becomes a map of what your customers respond to, and it feeds the next round of research.
How a hypothesis is written
Three parts, always in the same order. The third part is the one that makes a result useful after the test ends.
- If
- one specific thing on the store changes,
- then
- one named metric moves in a stated direction,
- because
- this is how the shopper's behaviour changes, backed by what the research found.
A filled-in example
This is how one test looks on paper before it goes live. The metric, the audience and the stopping rule are fixed first.
Test plan
Written before launch
If the delivery date appears next to the add to cart button,
then more mobile shoppers start checkout,
because they no longer leave the page to look up shipping times.
- Primary metric
- Checkout starts per product page session
- Audience
- All mobile visitors
- Confidence level
- 95 percent
- Minimum run
- Two full weeks
- Stopping rule
- Fixed before launch
The rules every test follows
They are agreed with you and written into the test plan before launch. Once the test runs, they do not move.
- One main metric
- Each test is judged on one metric chosen before launch. If the variant loses on it, a win on some other number does not rescue it.
- One audience
- The group of visitors the result applies to is fixed in advance. Slicing the data afterwards until some segment looks like a winner produces results that are pure chance.
- Confidence and power set up front
- The confidence level and the statistical power are agreed before launch and not lowered later because a result came close.
- A minimum run time
- Tests run for full weeks, at least two, so weekday and weekend behaviour are both included and early swings settle.
- No early stopping
- Results are not called on a good-looking day. Checking a running test and stopping at the first promising number makes false winners far more likely.
- One variant against the control
- Most stores reach a clear answer much faster with one challenger than with several. Extra variants need a stricter threshold and a lot more traffic.
- Quality checks during the run
- The variant is checked on every major device and browser at launch and again while it runs, because a broken variant quietly wastes traffic.
Three questions a test can answer
Choosing the right question saves weeks of traffic, so it is decided per change, before launch.
Is the change better?
The standard test, used for most ideas.
Is the change at least as good?
For changes that save money or effort, such as removing an app or simplifying a page. Proving that nothing got worse needs far less traffic than proving an improvement.
Is the change better by a clear margin?
For changes that are expensive to build, hard to undo or tied to a contract. The result has to clear the margin, not just zero, before it ships.
When a result is unclear
The current version stays. Keeping it when the change was slightly better costs a gain you never had. Shipping a change that only looked better can lower revenue for months without anyone noticing, so unclear results always fall on the side of what already works.
When testing is the wrong tool
Some pages do not get enough traffic for a test to reach a clear answer in reasonable time. There, changes are based on research, customer feedback and established usability principles, and they are watched closely after release. A broken checkout field gets fixed, not tested.
Find out where your store loses buyers.
Book a call