Primary research, small samples, published spreads

Four rules we hold ourselves to

Vendor research has a well-earned reputation, and the way to be worth reading is to adopt constraints that make flattering conclusions harder to produce. These are ours.

State the sample size at the top

Not in a footnote. If a finding rests on twenty-three customers, that appears in the first paragraph, because a reader deciding how much weight to give a number needs it before the number rather than after.

Publish the spread, not just the median

A median describes the customer in the middle. The interquartile range describes what happens if you are not that customer, and it is the figure a business case should be built on. Publishing it costs us deals against competitors quoting single numbers.

Include the findings that make us look bad

Our vendor deduplication rate is 69%. Our multi-line allocation rate is 66%. The bottom quartile of our customers sits materially below every median we publish. All of that is on the benchmarks page rather than omitted from it.

Name the bias in the sample

Our stalled-projects sample is self-selecting toward severe failures. Our pricing sample is biased upward because those quotes were shown to a competitor by buyers who thought the numbers were high. Both are stated on their pages, because a bias you name is one a reader can adjust for.

What we do not do

We do not license third-party data and present it as our own, run surveys where respondents self-report figures they have not measured, or publish anything where a customer could be identified from the aggregates.

We also do not extrapolate. Our sample skews toward services businesses, software companies, and professional services, and under-represents distribution and manufacturing. Where a finding probably does not generalise, the page says so rather than presenting a national figure derived from sixty-one companies.

Every paper is dated and reviewed at a stated cadence. Where we correct something, the correction is noted rather than silently applied, because a research page that changes quietly is one you cannot cite.