Sample Size

Why this matters for your decision

A test that is too small cannot show a real difference and ends up "inconclusive". A test that is too large costs more time and budget than necessary. The right sample size depends on three things: how frequent the measured behaviour is, how precise the result needs to be and how many variants are compared.

For decision makers, this means: precision is a deliberate choice. A pricing decision that is hard to reverse requires a narrower margin than a first demand estimate.

How to calculate it

For a proportion, the usual approximation applies: visitors per variant n = (1.96 / margin of error)² × p × (1 - p). Here p is the expected share on the primary metric and the margin of error is the desired precision at 95% confidence. The value is always rounded up.

The budget follows from variants × visitors per variant × cost per visitor, and the duration from the budget and a sensible daily spend. If markets are compared as well, the number of cells multiplies, and with it the requirement.

Example

A test compares three price points for a cordless food processor. It is expected that around 5% of visitors will click "Add to basket".

With a margin of error of 2 percentage points, each variant needs (1.96 / 0.02)² × 0.05 × 0.95, i.e. around 457 visitors. At 1.5 percentage points it is around 812, at 1 percentage point around 1,825. For three price variants with a 1 percentage point margin, that is just under 5,500 visitors in total.

Halving the margin of error quadruples the requirement. That is exactly why precision is set per test type before the start, narrower for price tests than for a first demand estimate.

How Horizon plans the sample

Horizon plans on the primary metric of the chosen set-up, with baseline values and cost per visitor from comparable tests in the same channel and the same industry, never from a guess. Before the start, it is checked whether the target group or the search volume can deliver the required visitors within the duration. If that does not work, the options are laid out openly: fewer variants, a wider margin or more budget. The measurement period should ideally cover a full week including the weekend.

How it differs

Sample size is not the same as statistical power, i.e. the probability of detecting an existing difference of a certain size; the two are, however, closely linked. Ad impressions or reach are not a sample: what is counted are the unique visitors to the offer page.

Limitations

The formula assumes an expected share. If the actual value is clearly lower, the planned sample is not enough. A larger sample also does not fix a bias: a measurement period of one and a half working days remains skewed, no matter how many visitors it contains. And with very rare signals, such as 0 versus 7 clicks, the base is too small for a conclusion, even if the intervals do not overlap.