Ground Truth

Why this matters for your decision

New methods, such as synthetic respondents or digital twins, are often described with agreement scores. These scores are only as meaningful as the reference measure against which they were measured. If the reference measure is a survey, a high level of agreement shows that the method replicates survey answers well. Whether people choose an offer has not yet been checked.

For decisions about new offers, prices or tariffs, this means: the question about the reference measure belongs at the start of every method assessment, not at the end.

Survey or behaviour as the benchmark

A language model reproduces what people have said and written. Even studies that attest a high level of agreement for synthetic respondents measure them against surveys, not against behaviour. Two widely noted papers show this: in one, AI agents reach 86% of the consistency with which people repeat their own answers in a large social survey. In the other, synthetic answers to purchase intent questions reach around 90% of the test-retest reliability of human respondents. Both results are methodologically remarkable, and both use survey answers as the reference measure.

Survey answers themselves, however, deviate from behaviour: intentions are acted upon in about half of cases, and hypothetically stated willingness to pay is on average higher than the real one. A method that replicates surveys well inherits this deviation.

Example

A team is assessing a method that simulates willingness to buy for three variants of a supplementary dental tariff. The provider's documentation shows a high level of agreement with an earlier survey. The team asks two questions: what was the agreement measured against, and were the tested offers already on the market?

The answer: measured against scale answers, for existing tariffs. For the new tariff, there is therefore no evidence based on behaviour. The team uses the method for pre-selection and checks the two remaining variants in a behavioural test.

How it differs

Validity describes whether a method measures what it is supposed to measure; the reference measure is the benchmark against which this is checked. Reliability describes whether repeated measurements agree, and says nothing about whether the right thing is being measured. A method can be highly reliable and still be calibrated against the wrong reference measure.

Limitations

Observed behaviour in the Painted Door Test is not a complete reference measure for the later market either: it shows the decision when faced with a realistic offer, not repeat purchases, returns or competitor reactions. The most honest benchmark for a method is therefore always the one that comes closest to your own decision.

Synthetic respondents make exploration faster: hypotheses, segments, a first pre-selection. Whether people choose a new offer is shown by their behaviour. That is why Horizon measures the purchase intent of real people for the final variants before the investment.