Behavioural vs. Survey Data

What lies behind it

Survey data is created when people provide information: about their attitudes, needs and perceptions, and about what they would do. Economists call this stated preference. Behavioural data is created when people act in a situation: they click on an offer, choose a tariff, put a product in the basket or leave it there. The preference is inferred from this, the revealed preference.

The difference does not lie in the quality of the people answering or acting, but in the situation. A survey asks a question, attention is high, price and alternatives are as present as the questionnaire makes them, and an answer has no consequences. At the moment of an offer, attention is scarce, and there are alternatives, a price and effort. The same person can be honest in both situations and still show different things.

This leads to a simple division of labour: surveys are strong when it comes to motives, drivers, understanding and perception. Behavioural data is strong when the question is whether people choose an offer and which variant comes out ahead.

The distinction applies to every kind of decision, not only to innovation: a new price, an additional tariff tier, a changed claim, a brand positioning or entry into a new market. Wherever a choice is made in the end, behaviour provides the answer to the choice and the survey provides the explanation.

What surveys and behavioural data each deliver

Surveys reach many people quickly, they can explain complex matters and ask about attributes one by one. They deliver the why: which benefit convinces, which term is misunderstood, which concern holds back a sign-up. Qualitative methods such as interviews and focus groups go deeper still and generate hypotheses a questionnaire does not know.

Quantitative methods such as conjoint analysis or MaxDiff show how people weigh attributes against each other when many combinations are available. Pricing surveys such as Van Westendorp or Gabor-Granger deliver ranges and thresholds. For early phases, for narrowing down many options and for understanding a target group, these methods are often the right tool.

Their limit lies where demand is inferred from a statement about future behaviour. This is where the Say-Do Gap, hypothetical bias, social desirability and demand effects come into play, even in well-designed studies.

Behavioural data shows what people do when a choice is due. It does not depend on how well someone judges themselves, and it contains the effect of price, wording and context as it works in the market. For comparing variants, it gives a direct answer: which variant is chosen more often under the same conditions?

Classic behavioural data comes from the market: sales, usage, cancellations, A/B tests on existing offers. It has one drawback when a decision is still pending: it only comes into being once the offer exists and the investment has been made. For new offers, new prices or new tariffs, this data does not exist yet.

Behavioural tests close this gap before the market. A Painted Door Test shows a realistic offer and measures how many people choose it before the product is built or the price is introduced. This is how Horizon works: real people see the offer pages via Google and Meta ads in their usual online environment, without a panel and without incentives. Nothing is sold. Anyone who chooses an offer is told transparently afterwards that it is a test.

How big the difference is

Research shows the gap from several directions. In a meta-analysis of 77 studies, Schmidt and Bijmolt (2020) find that hypothetically stated willingness to pay is on average 21 % above real willingness to pay. Contrary to what is often assumed, the overestimation for indirect methods was not smaller than for direct questions, but larger.

Morwitz, Steckel and Gupta (2007) show that surveyed purchase intent is more weakly linked to later sales for new products than for existing ones. Chandon, Morwitz and Reinartz (2005) show that the link between intent and purchase is 58 % stronger among surveyed customers than among non-surveyed ones: the survey influences the behaviour it sets out to measure.

And Webb and Sheeran (2006) show across 47 experiments that a medium-to-large change in intention produces only a small-to-medium change in behaviour. A variant that seems clearly more convincing in a survey therefore does not necessarily lead by the same margin in behaviour.

Methods at a glance

Survey with scales: measures attitude, liking and stated willingness to buy. Strong for perception and drivers, limited for demand.

Concept test: rates fully written concepts on scales. Strong for pre-selecting many ideas, but respondents know they are rating and read more attentively than in everyday life.

Conjoint analysis and MaxDiff: show how attributes are weighted, even with many combinations. Strong for the architecture of an offer, but the choice remains hypothetical.

Pricing surveys (Van Westendorp, Gabor-Granger): deliver accepted ranges and stated willingness per price point, not demand per price.

A/B test: compares variants in the live system on existing offers. Strong for optimisation, but requires that every variant can actually be delivered.

Painted Door Test and Fake Door Test: measure behaviour on an offer that does not exist yet. The Fake Door Test usually checks a feature in an existing app, the Painted Door Test a complete offer with a price before it is built.

The methods are not mutually exclusive. The question is which method carries which part of a decision.

How to combine both

When preparing a decision, you can sort the open questions by the evidence they need. Questions about the why belong in the survey: what do people understand by a term? Which benefit matters to them? Which concern keeps them from signing up? What language do they use?

Questions about the whether belong in the behavioural test: do people choose this offer at this price? Which of three tariff variants is chosen more often? How much sign-up intent does a higher premium cost? Which claim leads more people to the offer?

Questions about the how much lie in between. A survey can provide a price range within which the tests should move. The behavioural test then shows how measured purchase intent shifts within that range.

This sorting sounds trivial, but it prevents a common mistake: refining a survey until it is expected to answer a question it was not built for.

A proven sequence starts with exploration: interviews, surveys or concept tests narrow down options and formulate hypotheses. Then the few variants that are actually to be invested in go into a behavioural test. Finally, qualitative or quantitative surveys explain why one variant is ahead and provide material for communication and product.

What matters is that each type of data answers the question it was built for. The survey delivers the why and the range, behaviour delivers the choice. Combining both avoids two typical mistakes: inferring demand from agreement, and implementing a behavioural result without an explanation.

In working with consumer insights teams and research institutes, this means: the behavioural test does not replace a study, it adds the data point that surveys lack.

In practice, one simple guiding question before every study helps: which decision should this result support, and what kind of evidence does this decision need? If it is about understanding and direction, a survey is often enough. If it is about budget, price or the choice between a few final variants, a behavioural test belongs in the process before investing. This creates no dispute over methods, but a process in which each type of data has its place.

Common misconceptions

"Behavioural data needs a finished product." Market data does, behavioural tests do not. A Painted Door Test measures the choice on a realistic offer before anything is built or priced.

"A large sample evens out the bias." A larger survey makes the result more precise, but not less hypothetical. The bias is systematic and does not shrink when more people answer.

"Behavioural tests make surveys redundant." They answer the question of choice. The question of why and the development of variants remain the task of surveys and qualitative research.

"Whatever leads in the survey also leads in behaviour." Often, but not always. Especially with variants that differ only in price, wording or one attribute, the ranking can shift.

Example

A home appliance manufacturer wants to launch a cordless kitchen machine with one of two promises: "quieter than any kitchen machine in its class" or "no cable, anywhere in the kitchen". In a survey, participants rate both promises as relevant, and the cable promise receives slightly higher scores for willingness to buy.

In the behavioural test, each person sees only one offer page, same price, same images, only the promise differs. What is measured is the last binding click before the reveal. The page with the quiet promise achieves a clearly higher measured purchase intent.

A short follow-up survey explains the difference: many consider cordless a given for a battery-powered appliance, whereas noise is a real everyday annoyance. The survey delivers the why, behaviour delivers the choice.

Related terms

Revealed preference and stated preference: the two basic concepts from economics, inferred and stated preference.

Demand effect, social desirability, acquiescence bias and framing effect: biases that mainly occur in surveys.

Painted Door Test, Fake Door Test and A/B test: methods that measure behaviour, before or after launch.

Purchase intent and sign-up intent: asked on a scale or measured in behaviour.

Claim test and environmental claims: which statement triggers purchase intent. Which environmental claim is permissible is a matter for the legal department; which permissible wording triggers purchase intent can be measured in behaviour beforehand.

Limitations

Behavioural data is not automatically better. It does not explain why people act, and it depends on how realistic the situation is in which it arises. A behavioural test with an unclear offer page mainly measures the lack of clarity.

A Painted Door Test measures purchase or sign-up intent at the moment of the offer. It does not measure satisfaction after purchase, repeat purchase or cancellation, and it describes the people reached through the chosen channels. For decisions made mainly in personal sales or offline, the transfer requires care.

Surveys, on the other hand, have strengths that behavioural tests do not: they can examine many attributes at once, they also reach target groups that are hard to address online, and they provide the language in which customers think about a topic. The best evidence emerges when both types of data keep their role.

All terms in this topic

• Revealed Preference
• Stated Preference
• Demand Characteristics
• Social Desirability Bias
• Acquiescence Bias
• Framing Effect
• Painted Door Test
• Fake Door Test
• A/B Test
• Purchase Intent
• Sign-up Intent
• Claims Testing
• Environmental Claims