I was recently reminded about the risks of taking guidance based on surveys. Claims such as growth, adoption and intent can all be highly varied based on survey structure and administration. My husband, Tim Bates is an expert Product Manager, works with the national organization PDMA that trains and publishes on best practices. We discussed at length the value and effectiveness of surveys and how they can be effectively used (or abused). Below is some guidance for end users and vendors when using a survey to form a decision.
- Surveys are only effective as a quantitative research tool. A survey can only confirm hypothesis. Much like scientific analysis, it is difficult to use a survey to deny, but unfortunately the absence of information often leads people to wrong conclusions.
- Surveys are effective in identifying corner cases as they would receive overwhelming rejection or approval. They can also be effective (providing the survey is well formed and interviewee is motivated), for identifying requirements or features when they are well defined, known and closely aligned.
- Surveyors will target a particular audience using lists formed via trades, their own databases or even the ‘phonebook.’ Who responds and how the responder was contacted has influence on the result. If the intendeds are not similar in nature to your company or your position, or your issue, the outcome is not relevant.
- When reviewing a survey or poll, weighting is always the first thing to ascertain. This is particularly true where the surveyor may have a prejudiced outcome which happens often when trying to prove a point such as propensity to buy. To adjust for a lack of independent responses, there are times a pollster will adjust the weighting of each type of respondent. For example if 1000 people are polled, with a response of 40% New England, 20% West coast and 40% Southern states, and the pollster believes the greater population is more 50/40/10 they will minimize the intendeds who responded to make up the lack of West coast respondents and create their expected balance through weighting.
In general, the most reliable surveys are 6 to 10 questions sent to a true random group of the target audience with no controls that will result in the most honest representation of their opinions. This is hard to do and often the reason people are surprised the resulting votes people make with their ballots or money.
