8 biases that shape survey results, and how to spot them.

Surveys are indispensable for understanding the motives, barriers and language of a target group. As a basis for a concrete decision, they have one weakness: they measure what people say about their future behaviour. Eight biases are particularly relevant for pricing, product and plan decisions.

1. Hypothetical willingness to pay

Accepting a price in a survey costs nothing. A meta-analysis of more than 100 studies shows that respondents state a willingness to pay around 20% higher than what they actually pay (Schmidt and Bijmolt, 2020). For a plan or a premium, this means: the stated number is a starting point, not a basis for a decision.

2. Social desirability

People answer the way that goes down well: more sustainable, healthier, more prudent (Krumpal, 2013). For supplementary insurance, a green energy plan or a premium product, surveys therefore often overestimate demand.

3. Acquiescence

Many respondents tend to agree with statements rather than disagree, regardless of the content (Krosnick, 1999). Questions like “Would you use this offer?” therefore produce more yeses than a neutral wording or a real choice between alternatives.

4. Framing

The same information has a different effect depending on how it is phrased (Tversky and Kahneman, 1981). “€10 discount” and “€10 surcharge without the promotion” are mathematically identical but are judged differently. In a survey, the wording rather than the offer often determines the result.

5. Anchoring

A number mentioned beforehand influences the following estimate (Tversky and Kahneman, 1974). Anyone who sees a high price first then states a higher willingness to pay. In price surveys with several price points, this is hard to avoid.

6. Order effects

The position of an answer option influences how often it is chosen (Krosnick and Alwin, 1987). In concept comparisons, the variant shown first or last often benefits. Rotation helps but does not eliminate the effect.

7. Full attention instead of a real purchase situation

In a survey, an offer gets full attention. On the market, it competes with alternatives, distraction and limited time. What convinces in the survey is often lost in the real environment.

8. The question changes behaviour

Simply asking about an intention can change later behaviour (Sherman, 1980; Morwitz, Johnson and Schmittlein, 1993). Someone asked whether they will buy a product engages with it differently from someone who comes across it in everyday life.

What behavioural measurement changes

A test with real purchase behaviour avoids several of these biases because nobody is asked: people see a realistic offer and decide. Each group sees exactly one variant, which removes anchoring and order effects between variants. Behavioural measurement does not replace research into motives, though. Why people decide the way they do is still best explored with interviews and surveys.