Test and Learn

Why this matters for your decision

Many organisations already test, above all in digital marketing and in the live product: ad visuals, subject lines, checkout steps. This optimises what already exists. The big bets, however, such as a new product, a new pricing model or a new tariff, are often made without a prior behavioural test, because offers that are not available are hard to test in the live system.

Test and Learn delivers its greatest value when it moves ahead of the expensive decisions. A Painted Door Test makes this possible: it checks offers that do not yet exist, outside your own systems, with real people. In this way, Test and Learn for optimisation also becomes Test and Learn for direction.

For learning to happen, every test needs a question formulated in advance, a primary metric and a stopping rule. And the results need a fixed place in the decision, otherwise Test and Learn remains a collection of interesting individual findings.

A sequence helps: first the question with the greatest uncertainty and the largest budget behind it, for example whether an offer is chosen at all, then the question of price, then of promises and details. Each test reduces the uncertainty for the next. If the tests are set up comparably, over time you also build your own benchmark against which new results can be classified.

Example

A retailer is considering launching a private label for outdoor clothing in two price ranges. Instead of going straight into production, the team checks three price points for a weatherproof jacket in a first test. Around 2,000 visitors per variant reach the offer page. At €89, measured purchase intent is 2.8 per cent, at €109 it is 2.6 per cent, at €129 it is 1.4 per cent. The second test checks two benefit promises at €109. Only then is the decision on range and production made.

How it differs

An A/B test is a tool within Test and Learn, usually in the live system and for what already exists. Pretotyping and the Painted Door Test are tools for offers before they are built. Evidence-based innovation describes the overarching principle of basing decisions on evidence, and the Behavioural Gate its fixed anchoring in the approval process.

Limitations

Many small tests do not automatically add up to a good strategy. If you only optimise what already exists, you rarely find the fundamentally better offer. If you change too many variables at once, you learn little, because it remains unclear what had an effect. And if you stop tests as soon as you like a result, you collect false winners.

Horizon supports Test and Learn before the investment with Painted Door Tests, around four weeks from the question to the data analysis, with up to six variants.