Smoke Test

Why this matters for your decision

A smoke test answers a simple question with little effort: do people respond to this offer? It is often used very early, for example with a first draft of the value proposition, long before specifications exist. That makes it attractive for teams with many ideas and little time.

For investment decisions, however, what matters is how the test is built. A single value without a comparison, a waiting-list sign-up without a price, or traffic that the ad platform optimises for cheap clicks quickly deliver a number, but not a reliable basis.

Many teams already run smoke tests themselves. That is a good sign, because it shows that the gap between what is said and what is done is known. The question then is whether the tests are comparable enough to build benchmarks across several decisions.

What makes a reliable smoke test

Several variants instead of one page, so that a comparison emerges. One isolated variable, so that it is clear what causes the difference. A price on the page, so that purchase intent and not just curiosity is measured. A measurement point close to the decision, i.e. the last binding click. Random allocation of visitors and a sample size defined in advance.

Horizon runs smoke tests in this structured form as Painted Door Tests: realistic offer pages, Google and Meta ads, up to six variants, around four weeks to data analysis. Nothing is sold, and the reveal follows directly after the click.

Example

A telecommunications provider wants to know whether a mobile tariff with unlimited domestic data for €29.99 attracts interest. A simple smoke test with one page and a 'Join the waiting list' button results in 5% of visitors signing up.

A structured test with three price variants (€24.99, €29.99 and €34.99) and the click on 'Choose tariff' as the measurement point results in 3.0%, 2.6% and 1.5%. Only the second test shows how strongly the price moves sign-up intent.

How it differs

Smoke Test, Painted Door Test, Fake Door Test, Pretotyping and proto-selling essentially mean the same approach. There are nuances in how it is used, for example whether the test runs in an existing app or on a dedicated landing page. What matters is not the name, but whether the test is built with a price, several variants and statistical analysis.

The technical smoke test in software development has nothing to do with this.

Not the same as on-page testing

Tools for A/B testing, feature flags or click analytics also show placeholders or a fake button, but inside a product or website that already exists, for people who already use it. They optimise an existing journey. A Painted Door Test as Horizon runs it measures demand before the product exists: with dedicated offer pages, people who arrive through ads, several variants and a price. The question is not how to improve a page, but whether a new offer, price or feature is chosen at all.

Limitations

A smoke test measures a first online response. It does not explain why people respond, and it reflects neither repeat purchase nor usage. Waiting-list sign-ups without a price easily overstate demand. It is not suitable for decisions made in retail, in B2B procurement or on comparison portals.