A/B tests are the standard way to continuously improve digital products. They require that every variant tested can actually be delivered and fulfilled.
Why this matters for your decision
A/B tests protect against gut decisions. Even well-founded ideas often fail to improve their target metric: in a well-known analysis from practice, only about a third of experiments were successful. Testing instead of guessing saves you many missteps.
An A/B test mainly answers the question 'how': which button, which headline, which flow. The question 'whether' comes before that: whether a new offer, a different price or a new tariff is chosen at all. This question is hard to answer in the live system, because it would require launching an offer that does not exist yet, or having several prices apply to real customers at the same time.
A/B test and Painted Door Test
A Painted Door Test is methodologically related: it also randomly assigns visitors to variants that differ in exactly one attribute. But it runs outside your own systems, with its own offer pages and traffic from Google and Meta ads. Nothing is sold, and the reveal follows immediately after the click. That is why it can test offers and prices that do not exist yet without influencing existing customers.
Both have their place: pre-launch validation with the Painted Door Test, post-launch optimisation with the A/B test.
Example
An online retailer wants to introduce a pet food subscription. Before launch, a Painted Door Test at Horizon shows two variants: subscription with a 10% discount and subscription with free shipping. Measured purchase intent is 3.6% and 2.5%, and the discount variant is built.
After launch, the team uses A/B tests to optimise the page in its own shop: position of the subscription notice, wording of the button, number of steps in the checkout.
How it differs
The A/B test requires an available product and measures real usage or real purchases. The Painted Door Test measures purchase intent before building. The Fake Door Test sits in between: a button for a non-existent feature in the existing product. Multivariate tests vary several elements at once and need correspondingly more traffic.
Limitations
A/B tests need sufficient traffic in your own system and the ability to deliver every variant. Far-reaching changes to price or offer in live operation can unsettle existing customers. The Painted Door Test has different limitations: it measures intent rather than purchase and the first decision rather than repeat purchase, and it only reaches target groups that can be addressed through ads.
There are statistical parallels too: both procedures need a sample size set in advance and clear stopping rules, otherwise false winners emerge from looking too early.
Evidence
Kohavi et al. 2009: Of well-designed experiments intended to improve a key metric, only about a third succeeded in doing so. Online Experimentation at Microsoft, Third Workshop on Data Mining Case Studies and Practice Prize. Source
Frequently asked questions
Can I run a Painted Door Test on my own website?
In principle yes, if your legal department agrees. It is often easier to test outside your own systems so as not to influence existing customers.
Which is better, an A/B test or a Painted Door Test?
Neither is better. The Painted Door Test answers before the investment whether something is chosen; the A/B test then improves how it is implemented.
Does Horizon optimise landing pages?
No. Horizon uses offer pages as a measurement instrument for product and pricing decisions, not to optimise existing pages.
Which question do you need to answer before you have something to A/B test?
Bring your decision question, and we will outline a possible test design.
You will speak with Daniel Putsche
Founder & CEO, 30 minutes
Read more