Which metrics does a painted door test deliver, and which one counts for your decision?
From the click on the ad to the confirmed email: which signals a test measures, how committed they are and which metric should carry your decision.
In short
- A test delivers metrics along the whole path: ad, offer page, decision, email.
- What matters is a primary metric fixed in advance: the last committed click before the resolution.
- All rates refer to unique visitors of the offer page, not to ad impressions.
After a test, there are many numbers: click rates, costs, visitors, decisions, sign-ups. Not all of them say something about willingness to buy. Which metric counts is best decided before the test starts.
Ad: reach, click rate, cost per click
Ad data shows how well a variant attracts attention. But it depends heavily on the platform: auction, delivery and time of day. Cost per click is therefore a planning and quality figure, not a demand signal. How many visitors a variant receives is also controlled and says nothing about its appeal.
Offer page: the basis for all rates
All rates refer to unique visitors of the offer page. Only then are variants comparable. A rate per ad impression would mix the performance of the ad with that of the offer.
The decision: the primary metric
The key metric is the last committed click before the resolution. On a single-stage page, that is the buy button on the offer page. If there is an additional product page, the click on “Add to cart” counts there; the click on the first page is then only an intermediate step. The closer the click is to a real purchase, the less curiosity it contains.
Email and confirmation: additional validation
After the resolution, visitors can sign up voluntarily and confirm their email. These signals are valuable because they show more commitment. But they happen after visitors know that the offer is not available and therefore measure interest in updates, not willingness to buy. They complement the primary metric but do not replace it.
Levels of commitment
- Ad clicked: attention.
- Offer page read: interest.
- Purchase or sign-up click: decision, the primary metric.
- Email entered and confirmed: additional commitment.
Cost per purchase decision
Ad budget and decisions give the cost per purchase decision. It helps to assess later marketing. We only report it as a number once there are enough decisions per variant; below that, we show it as a direction.
When a difference is real
Whether a variant is really ahead is shown by the statistical analysis of the primary metric: a significance test and the probability that a variant is the best. Both rules are fixed before the start. The result is then labelled confirmed, directional or inconclusive. What that looks like in practice is shown in the worked example in the article on analysing two price variants.
Frequently asked questions
Why does the email sign-up not count as the main result?
Because it happens after the resolution. Anyone who signs up already knows that the offer is not available. The sign-up shows interest; the purchase click before it shows the decision.
Does a high click rate on the ad say something about demand?
Only to a limited extent. It shows that the ad attracts attention. Whether people would buy is only shown by the decision on the offer page.
About the author
Daniel Putsche
Founder and CEO of Horizon. Works with product, pricing and insights teams to base decisions on measured purchase behaviour.
Related articles
Which decision do you want to validate?
Bring your decision question, we sketch a possible test design with you. No preparation needed on your side.
You will talk to Daniel Putsche
Founder & CEO, 30 minutes