A lot of the blogs I write cite the results of surveys on topics I think are relevant to the broadband industry. Every time I see survey results, I ask myself how much trust I can put into the survey results.

There are three elements needed to produce a survey with reliable results. The first element involves who takes the survey. The second important issue is the number of people who take the survey, and the final issue is the quality of the questions asked.

Most surveys cited in articles don’t describe how they found the people who were surveyed. For a survey to be statistically valid, the people chosen to take the survey should be chosen at random. That’s a lot harder than it sounds. Consider a survey that wants to know how many people don’t have a cellphone. The normal process of randomly calling people using landline and cellular numbers will not find people who don’t have any phone service. A lot of surveys cited in articles are clearly not random. For example, a survey given to people who follow a specific website is going to be biased by the type of people who like that website.

Surveys don’t always have to be random to be useful. There is a survey done every year for executives and engineers of cable companies that asks about future technology plans. This survey is clearly not random, so the results don’t have any statistical validity. But since a similar survey is given every year, the survey is great at seeing trends. You probably can’t believe any specific percentages of results that come from this kind of survey, but you can put faith in year-over-year trends.

The number of surveys taken matters. Many business surveys are conducted to be 95% accurate, plus or minus 5%. That accuracy says that if you were to ask the same questions to 100% of the target audience,  the response you receive would be between 90% and 100% the same as the survey results. Consider the following table of the numbers of surveys required to reach a 95% overall accuracy within a given precision for a universe consisting of 5,000 homes, 50,000 homes, or 500,000 homes.

People are almost always amazed at the small number of surveys needed to get a statistically valid and believable answer. What really surprises them is the number of surveys to be able to rely on a survey covering 500,000 homes versus one covering 5,000 homes. The second and third columns show how many additional surveys are needed to have more precision. Many political surveys shoot for an accuracy of plus or minus 3%, which looks like the second column. It’s rare to see somebody shoot for 1% accuracy since it requires such a large number of completed surveys.

The last element of a believable survey is good questions that are not biased. It’s easy to spot obviously biased questions. A survey that asks, “Don’t you just hate customer service at Comcast?” is biased, while the question, “How would you rate Comcast customer service on a scale of 1 to 5?” is not. A lot of bias in questions is more subtle, and the folks writing good surveys work hard to make sure that questions don’t lead respondents to specific responses.

A final element of believing surveys is the topic being covered. People are far more likely to give a false answer when asked about politics or religion than they are about being asked about their preference for a product or company. Professional survey companies say that nearly half of respondents lie about their incomes.

The surveys I mention in my blogs are all over the map in terms of accuracy and reliability. Very few of them describe the process of selecting respondents, and it’s likely that many of the surveys were not administered to a random sample of people. Many published surveys mention the number of surveys given, but don’t mention the universe of possible respondents and don’t report on the accuracy the survey supposedly measures. Many surveys don’t show the exact questions that were asked but instead summarize the results without being specific. The bottom line is that readers should always take survey results with a grain of salt.