When is a painted door test worth it, and what does it measure that an A/B test does not?

Every year, large budgets flow into products, plans and campaigns whose demand nobody measured beforehand. A painted door test brings this signal forward. But not every question suits it. Here is when it is the right method and when you are better off choosing another.

What a painted door test is

People from your target audience see an offer as it would later look on the market: in an ad and on an offer page. Comparable groups each see one variant, for example a different price or a different concept. What is measured is who decides for the offer. Nothing is sold and no payment is taken; right after the decision, everyone learns transparently that the offer is not available yet. The method is also known as a smoke test or fake door test, as part of pretotyping.

What it is suited for

• Prices: which price holds, and how much demand does an increase cost?
• Concepts: which of several ideas do customers choose?
• Offer models and plans: tiers, terms, subscription or purchase?
• Value proposition: which promise leads to a purchase, not just a click?
• Features: which features do customers pay for?
• Markets and target groups: where and with whom is demand highest?

This applies to new products as well as existing ones: a large share of tests concern prices, plans and messages in the running business.

When it does not fit

1. B2B buying processes

Where several departments decide over months, a test with individual people does not reflect the buying process. We therefore do not use painted door tests for B2B offers.

2. Categories with strict advertising restrictions

Tobacco, alcohol or gambling are subject to strict rules on advertising platforms. If an offer cannot be shown realistically, behaviour cannot be measured cleanly either.

3. Purchases that require a lot of advice

Some decisions are only made after consultation, a test drive or long research. Here we measure an earlier, clearly defined step, such as choosing a plan or requesting advice, and explain openly what the result says and what it does not.

4. When you want to understand the why

A painted door test shows what people do, not why. Motives, barriers and the language of your target group are best explored with interviews, focus groups or surveys.

What it measures that an A/B test does not

An A/B test optimises what is already on the market: it compares variants of an existing page with the visitors who come anyway. A painted door test examines an offer before it exists or before it changes, with specifically recruited, comparable groups from your target audience.

• Timing: A/B test after launch, painted door test before.
• Visitors: for an A/B test, existing traffic; for a painted door test, a specifically reached target audience, unbranded if you wish.
• Subject: for an A/B test, mostly page details; for a painted door test, the decision itself, such as price, concept or plan.
• Result: for an A/B test, more conversions on the page; for a painted door test, the basis for an investment or pricing decision.

What it does not replace

• Survey: Strength: Fast, broad, inexpensive · Best for: Orientation and hypotheses
• Conjoint: Strength: Many attributes at once · Best for: Weighting attributes
• Focus group and interview: Strength: Deep insight into motives · Best for: Sharpening ideas, the why
• Painted door test: Strength: Real behaviour for a few variants · Best for: The final decision

Checklist before you start

• The decision is clearly named, and there is a budget that depends on it.
• The variants differ in exactly what is to be decided.
• The target group can be reached online.
• The success criterion is fixed before the start, not afterwards.