A practical framework for landing page testing

← Back to Field Notes

The goal of landing page testing is simple: to replace opinions about what works with evidence. The process for getting reliable evidence is more involved than most people realise. Here is the framework that consistently produces useful results across the accounts I manage.

Start with a hypothesis, not just a variant

A test without a hypothesis is an experiment without a purpose. Before changing anything, write down: what you are changing, why you expect it to improve conversion rate, and what mechanism you think will drive the improvement. "Changed the headline from X to Y because the new version addresses the specific fear about cost that our chat transcripts show is the main objection" is a hypothesis. "Changed the headline to see what happens" is not. The hypothesis disciplines you to think through the change properly and makes the test result interpretable regardless of outcome.

One change at a time

Multivariate testing - changing multiple elements simultaneously - requires significantly more traffic to reach statistical significance than A/B testing a single change. For most landing pages, sequential A/B testing of individual elements is more practical and produces clearer learning. The sequence should move from highest-impact changes to lower-impact ones. Headline, main CTA, social proof placement, and offer framing are higher-impact than button colour, font choice, or image selection. Start where the effect size is likely to be largest.

Sample size calculation before you start

This is the step most testers skip and it is the step that most often invalidates results. Use a sample size calculator before launching any test. Input your current conversion rate, the minimum detectable effect you care about - the smallest improvement that would be commercially meaningful - and the confidence level you require (95 percent standard). The calculator tells you how many visitors each variant needs. If your traffic level means you cannot reach that sample size in four to six weeks, the test is not viable. Do not run it and do not interpret the results as meaningful if you do.

Running time and weekly completion

Even if you reach your required sample size in ten days, run the test for a minimum of two complete weeks to account for day-of-week variation. Conversion rates on Tuesday are different from Saturday. A test that happens to run Monday to Wednesday and reaches sample size will produce biased results that do not reflect the full weekly pattern.

Documenting and applying learnings

A test result - win or loss - is only valuable if it is documented and learned from. Create a testing log that records the hypothesis, the variant, the sample size, the result, and your interpretation of why the result occurred. This institutional knowledge compounds over time. After 20 tests you have a clear picture of what your specific audience responds to. That picture informs not just your landing pages but your ad copy, your email messaging, and your sales conversations.