Painted door test: 9 mistakes that make your result unusable.

A painted door test measures how people decide when faced with a realistic offer. That only works if every difference in behaviour comes from what you want to decide. We see nine mistakes again and again, and all of them can be avoided before the start.

1. No clear decision question

“We want to see how the product goes down” is not a question that leads to a decision. Phrase it beforehand: if variant A is ahead, we do X; if B is ahead, we do Y. Only then can you say what needs to be measured.

2. More than one variable

If two pages differ in price and headline, you will not know in the end what made the difference. Exactly one variable may differ. Everything else stays identical: target group, layout, imagery, number and wording of buttons, page length.

3. Advertising language instead of a realistic offer

“Interested? Click here to learn more” measures curiosity, not willingness to buy. The page should look like a real offer, with scope of service, price and a clear purchase or sign-up decision.

4. Ad and page do not match

Anyone who sees something different in the ad than on the page drops out or decides under false assumptions. In price tests, the price therefore already appears in the ad, and the image from the ad reappears on the page.

5. Variants look different

If variants are built one by one, spacing, colours and wording shift unnoticed. Build one template, agree on it and derive all variants from it by swapping only the variable.

6. The wrong measurement point

Signing up for email after learning that the offer is not available measures interest in updates. The key metric is the last committed click before this notice, the purchase or sign-up decision. Fix it as the primary metric before the start.

7. Stopped too early

After two days, one variant is ahead and the test is ended. Early results fluctuate strongly, and a short period without a weekend represents the target group poorly. Decide beforehand from which sample size and certainty you will decide, and stick to it.

8. Mobile view not checked

A large share of visitors comes via smartphone. If a price wraps there or the button sits below the visible area, the test measures the layout, not the offer. Check every variant on common screen sizes.

9. No fair resolution

Anyone who decides learns right afterwards that no purchase has taken place and what it was about. Only then is an email sign-up offered. That is our standard: nobody should feel deceived, and no data is collected under false pretences.

Checklist before the start

• The decision and its consequences are written down beforehand.
• Exactly one variable differs, everything else is identical.
• The primary metric is fixed.
• Sample size, runtime and stopping rule are set.
• Ads and pages match, also on mobile.
• The resolution comes before any data collection.