Tag Archives: surveys

Believing Surveys

A lot of the blogs I write cite the results of surveys on topics I think are relevant to the broadband industry. Every time I see survey results, I ask myself how much trust I can put into the survey results.

There are three elements needed to produce a survey with reliable results. The first element involves who takes the survey. The second important issue is the number of people who take the survey, and the final issue is the quality of the questions asked.

Most surveys cited in articles don’t describe how they found the people who were surveyed. For a survey to be statistically valid, the people chosen to take the survey should be chosen at random. That’s a lot harder than it sounds. Consider a survey that wants to know how many people don’t have a cellphone. The normal process of randomly calling people using landline and cellular numbers will not find people who don’t have any phone service. A lot of surveys cited in articles are clearly not random. For example, a survey given to people who follow a specific website is going to be biased by the type of people who like that website.

Surveys don’t always have to be random to be useful. There is a survey done every year for executives and engineers of cable companies that asks about future technology plans. This survey is clearly not random, so the results don’t have any statistical validity. But since a similar survey is given every year, the survey is great at seeing trends. You probably can’t believe any specific percentages of results that come from this kind of survey, but you can put faith in year-over-year trends.

The number of surveys taken matters. Many business surveys are conducted to be 95% accurate, plus or minus 5%. That accuracy says that if you were to ask the same questions to 100% of the target audience,  the response you receive would be between 90% and 100% the same as the survey results. Consider the following table of the numbers of surveys required to reach a 95% overall accuracy within a given precision for a universe consisting of 5,000 homes, 50,000 homes, or 500,000 homes.

Households ± 5% ± 3% ± 1%
5,000 357 1,351 3,845
50,000 382 1,784 12,486
500,000 384 1,843 16,106

People are almost always amazed at the small number of surveys needed to get a statistically valid and believable answer. What really surprises them is the number of surveys to be able to rely on a survey covering 500,000 homes versus one covering 5,000 homes. The second and third columns show how many additional surveys are needed to have more precision. Many political surveys shoot for an accuracy of plus or minus 3%, which looks like the second column. It’s rare to see somebody shoot for 1% accuracy since it requires such a large number of completed surveys.

The last element of a believable survey is good questions that are not biased. It’s easy to spot obviously biased questions. A survey that asks, “Don’t you just hate customer service at Comcast?” is biased, while the question, “How would you rate Comcast customer service on a scale of 1 to 5?” is not. A lot of bias in questions is more subtle, and the folks writing good surveys work hard to make sure that questions don’t lead respondents to specific responses.

A final element of believing surveys is the topic being covered. People are far more likely to give a false answer when asked about politics or religion than they are about being asked about their preference for a product or company. Professional survey companies say that nearly half of respondents lie about their incomes.

The surveys I mention in my blogs are all over the map in terms of accuracy and reliability. Very few of them describe the process of selecting respondents, and it’s likely that many of the surveys were not administered to a random sample of people. Many published surveys mention the number of surveys given, but don’t mention the universe of possible respondents and don’t report on the accuracy the survey supposedly measures. Many surveys don’t show the exact questions that were asked but instead summarize the results without being specific. The bottom line is that readers should always take survey results with a grain of salt.

Why Homes Don’t Have Broadband

I write all of the time about the rural digital divide – about homes that have no broadband options or that have terrible options such as extremely slow DSL or wireless service. The COVID-19 crisis has reminded us that there are also a lot of homes in cities and towns that don’t have broadband.

John B. Horrigan published a paper earlier this year titled Measuring the Gap that makes the point that the reasons that homes don’t have broadband are complicated. There have been studies over the years that have tried to pin down the primary reason that homes don’t have broadband, but by doing so the studies have glossed over the fact that most homes have multiple reasons for not having broadband.

A good example of this is a Pew Research Center survey in 2019 that explored the issue. In that survey:

  • 50% of respondents said that high prices is a reason for not having broadband, but only 21% said price is the primary reason.
  • 45% of respondents said they relied on smartphones that could do everything they need, but only 23% said that was the primary reason for not buying broadband.
  • 43% said they were able to get access to the Internet from a source outside the home, but only 11% gave that as the primary reason.
  • 45% said that the cost of a computer is too expensive, but only 10% gave that as the primary reason.

As Horrigan points out, sometimes there is bias in the questions being asked in a survey. If the surveyor has pre-conceived ideas about why folks don’t have broadband they will miss some of the reasons. Consider a 2017 survey from the California Emerging Technology Fund. This survey showed different reasons than Pew for why homes don’t have broadband because the survey asked different questions. The survey showed:

  • 69% said the cost of monthly access and of affording a computer or smartphone was too high. 34% listed this as the primary reason for not having broadband.
  • 44% said it was too difficult to set up a computer and to learn how to use broadband, which 12% gave this as the primary reason.
  • 42% said they were concerned about privacy and computer viruses, while 21% gave this as the primary reason for not having broadband.
  • 41% said they had a lack of interest in being online, with 22% giving this as the primary reason for not having broadband.

The results of those two surveys are drastically different because the surveys asked different questions. If a survey doesn’t provide the option to say that privacy is a reason for not having broadband, then that gets missed. People can only respond to the questions asked in a survey as presented to them. For example, there were 12% of respondents in the second survey above that worried about privacy as their primary reason for not having broadband. There had to be people that felt the same way in the Pew survey, but since that question was never asked, respondents were forced to pick from among the choices they were given.

This highlights one of the issues of using surveys to find out why people do certain things. Surveys are best used when measuring what people do. For example, a well-designed survey can make a great and reliable estimate of the number of homes in a community that don’t have a home computer. But it’s a lot tougher to use a survey to find out why homes don’t have computers since there might be dozens of reasons for not having one.

Another issue to consider is that people might not tell a surveyor the truthful answer to a question if they think the response is personal. For example, people don’t like to admit that using a computer is too hard for them or that they are intimidated by technology. Many people are not going to tell a stranger that they can’t figure out how to use a computer. However, those same people might willingly share that they would be more likely to use a computer if they had better training. The manner of asking this sort of question can change the response.

This blog is not meant to bash surveys, because a survey is one of the best tools available for understanding broadband in a market. A survey can quantify how many people use different ISPs and can measure their happiness with the various ISPs already in the market. A survey can provide a decently reliable estimate of the percentage of the community that will consider switching to a new ISP. But surveys are a lot less reliable when they ask people to reveal personal reasons why they do or don’t do something – for the simple reason that people are often unwilling to share their shortcomings and fears with a stranger.

This is something to keep in mind if you want to use a survey to understand broadband in your community. Asking questions about sensitive subjects produce unreliable results. As an example, surveys do a lousy job of predicting what people are willing to pay for broadband. A survey can quantify what somebody would like to pay for broadband, but that is not the same question of what they will pay. I’ve seen surveys convince ISPs to set low broadband rates due to faulty survey questions. It’s somewhat meaningless when somebody who is already paying $75 per month for broadband tells you they would only change to a new ISP that charges $45. Such a respondent is likely somewhat embarrassed to admit they are paying too much for broadband today, and that bias makes their answer unreliable.

Writing good survey questions is an art. I’ve been doing that for twenty years and I still find situations where it’s nearly impossible to get the answers that clients are hoping for when the survey probes into questions that customers don’t necessarily want to answer.

2015 Broadband Growth

S vurveOne of the things I’ve figured out about the telecom industry is that statistics are often used to tell very different stories. Consider this example regarding wireline broadband adoption:

In December Pew Research released the results of a survey that suggested that overall wireline broadband adoption had dropped to 67% in 2015, down from a high of 70% in 2013. This was the first time I had ever heard any suggestion that the total number of landline broadband connections have flattened out, let alone dropped.

Pew went on to say that main culprit for the drop in broadband adoption is broadband prices and that a lot of homes feel they cannot afford a broadband connection, and instead rely solely upon broadband from their smartphone. That sounds plausible, and Pew was comparing to a very similar survey they had given in 2013.

But the Leichtman Research Group just released a report saying that the big cable companies added 3.3 million broadband customers in 2015. They said that during the year that the large telcos lost 187,000 landline broadband connections, meaning an overall net increase of over 3.1 million new broadband connections for the year.

The Census estimates there were 124.6 million housing units in the country in 2015, so the big companies in total brought broadband to an additional 2.5% of the total market. That sure does not sound like a year in which broadband has declined as suggested by Pew. And Leichtman has shown total market growth for the last several years as well.

In this case you have to believe the Leichtman numbers. They gather total subscriber numbers from all of the large carriers – cable companies and telcos. Since almost all of these companies are publicly traded, and since Wall Street keeps a close eye on subscribers, one has to think that the Leichtman numbers are pretty accurate.

On the other hand the Pew numbers come from nationwide surveys. Pew did three surveys in 2015 with a total of 6,687 adult respondents. The 2013 numbers they are comparing to was based on surveys of 6,010 adults.

I have always been suspicious of nationwide surveys. Our firm gives surveys and I have found that local surveys can be very accurate and the results can often be correlated with externally collected facts. For instance, I’ve had clients do surveys to find out how many customers their competition has in a market, and these surveys often prove themselves to be valid by also accurately showing the market penetration of my clients. That makes it easy to believe that the numbers for the other competitors in the market are also accurate.

I know that Pew is very careful about how they randomly choose survey subjects. For instance they will call people with cellphones as well as those with landline telephones. If you crunch through the statistical formulas that describe the predicted accuracy of a nationwide survey, then the Pew surveys should be very accurate.

The Liechtman numbers are not a 100% count of broadband customers and only count the customers of the biggest broadband providers – but those providers are something like 95% of the whole market. I know enough about a lot of companies in the rest of the market, the smaller carriers, to know that many of them are still seeing healthy broadband customer growth.

I have no way to explain this difference and I suspect that Pew can’t either. Their survey should be pretty accurate. Yet sometimes nationwide surveys just don’t give accurate results. This can often be seen with elections where different surveys given at almost the same time show fairly disparate predictions. The trouble is that surveys from groups like Pew influence decision makers and there are now going to be those who think that broadband growth has topped out. I was just on a call last week where somebody mentioned the Pew numbers. And while the Pew numbers of total broadband users might not be totally accurate, one can still believe that  their observation that some people are finding broadband increasingly expensive probably is valid. The problem is, you just can’t really know how many people that might be.