Setting up a painted door test: 8 steps from the decision to a robust result.

A painted door test shows how people decide when faced with a realistic offer: in an ad, on an offer page, with a clear purchase or sign-up click. Whether the result can support a decision is mostly settled in the planning. This is how we proceed.

1. Write down the decision

Phrase the decision so that every result leads to an action: “If the price of €29 does not measurably reduce demand, we introduce it. If it does, we stay with the current price.” If you do not know beforehand what you will do with the result, you are testing blindly.

2. Fix the hypothesis and primary metric

The hypothesis describes a difference in behaviour, not a revenue forecast: “Variant B achieves at least as high a purchase decision rate as A.” The primary metric is the last committed click before the resolution, usually the purchase or sign-up button.

3. Choose the test type and variable

Price, concept, value proposition, feature, offer model, brand or target group: the test type determines what may differ between the variants. Exactly one variable per test. More questions are answered in a sequence of tests, not in one.

4. Define variants and a benchmark

Up to six variants per test. Often a benchmark variant belongs in the set, such as the current price or the existing offer. It shows whether a new variant is not only better than the other new ones, but better than the status quo.

5. Build pages from one template

Build and check variant A completely first: ad, offer page and, if needed, a product page. All other variants are created from this template by swapping only the variable. Before the start, a dedicated step checks that the variants really differ only in that.

6. Set the target group and ads

All variants reach the same target group through the same channel, randomly assigned. The ads are identical except for the variable; in price tests, the respective price appears in the ad text. Whether Google or Meta fits better depends on whether people actively search or discover the offer in everyday life.

7. Plan sample size, budget and stopping rule

How many visitors each variant needs follows from the required precision and the expected rate. Budget and runtime follow from that, typically about a week, always including a weekend. It is fixed beforehand from which certainty a variant counts as ahead; for pricing decisions we set a higher threshold than for other tests.

8. Resolve and analyse

Anyone who decides learns right afterwards that no purchase has taken place and can sign up for updates voluntarily. The primary metric is analysed per variant, with statistical validation and compared with similar tests. The result is a data analysis as the basis for the decision from step 1.

How Horizon supports this

The Horizon team takes on all eight steps together with you: from the decision question through setup and execution to the data analysis. From question to result takes around four weeks, of which about one week in the field.