Broadband and the Census

The US government is gearing up to begin the 2020 census which will be administered starting next April 20. For the first time the Census is going to rely heavily on people answering the census questions online. Live census takers will then follow-up with those that don’t submit the online response.

This seems like an odd decision since there are still a lot of people who don’t have home broadband. This seems like a poorly conceived idea by those of us who understand the FCC’s dirty little secret – the FCC has no idea how many homes don’t have broadband.

As a country, we care a lot about an accurate Census. The census data is used for multiple government purposes. The 10-year Census is used to redraw both federal and state political boundaries every ten years. The Census is used to determine the number of US House Representatives allowed for each state. The government uses the Census to allocate the funding for numerous federal programs that allocate funding by population. If an area of the country is undercounted they lose both political representation and federal funding for a wide variety of purposes.

This all means that there is a significant downside risk for any part of the country that is undercounted in the Census. The Census is hiring 175,000 fewer door-to-door people nationwide to follow-up on those that don’t answer the first wave of the Census, and one has to wonder if they are going to be equipped when huge portions of rural America doesn’t respond to the online census request.

As I said earlier, we have no idea as a country how many people don’t have home broadband. According to the FCC maps, there are still 21 million people in rural America with no access to broadband. However, everybody understands that this number is understated due to the idiotic rules used to count broadband customers by the FCC. We use a self-reporting system where ISPs tell the FCC about their broadband coverage. We know that many ISPs have overstated the speeds they can deliver along with the areas of their coverage. That’s bad enough, but the FCC then compounds this error by assuming that if a census block has at least one broadband customer that the whole block has broadband. A census block is normally 600-800 homes and anybody living in rural America understands how large such an area can cover.

We have other people counting broadband that paint a very different picture than the FCC. The one with the widest reach and most credibility is Microsoft. They are able to measure the speed of downloaded software upgrades – a method that tells the real broadband situation at a home. Microsoft estimates that 162 million people in the US don’t have access to broadband that meets the FCC’s definition of 25/3 Mbps. But Microsoft has no way of counting homes with no broadband.

This is not just a rural problem. It’s always been suspected that there are millions of homes in older urban areas that don’t have access to broadband. There are apartments and little pockets of neighborhoods everywhere that were bypassed by the cable companies when they built their networks in the 1970s and 80s. Folks who study this issue estimate that there could be as many as 10 million people in urban areas without broadband access.

Even more importantly, there are millions of people that elect not to buy broadband or who access the Internet only using a cellphone. There are still homes everywhere that either can’t afford the Internet or who refuse to go online. Even among houses with broadband there are going to be many who don’t have good enough computer skills or the language skills to find and complete the Census questions online.

My guess is that the Census Bureau is going to be totally overwhelmed by the levels of non-response of households that don’t take the Census online. There will be huge geographic rural areas where few people respond online. There will be people everywhere who don’t have access to broadband or are unable to navigate the online questionnaire.

In the past the US Census Bureau believes they got a pretty high response. They got a decently high response from households that completed the paper census forms and had an army of census takers that tracked down houses that didn’t respond. If the completion ratio for the Census slips even a few percent, then areas without good broadband are likely to be disadvantaged in the many ways that Census data affects states.

The Census was moved online to save money. I think that the decision to go online is probably ten years premature and that the Census Bureau is probably totally unprepared for what’s going to happen next April. I hope I’m wrong.

FCC Modifies Broadband Mapping Parameters

Last week the FCC decided to change the method of collecting data to support its broadband maps. It’s widely understood that the current mapping system badly misstates broadband coverage. That’s a big problem since the FCC uses the faulty broadband mapping data to make decisions like determining eligibility for broadband grants.

The most important new change is that ISPs have to produce mapping ‘polygons’ to show where they have existing customers. The ISP polygons can cover areas without current customers only where an ISP “has a current broadband connection or it could provide such a connection within ten business days of a customer request and without an extraordinary commitment of resources or construction costs exceeding an ordinary service activation fee.”

The new polygons fix one of the big flaws in the current broadband map. The polygons are going to make a noticeable difference when showing coverage for a cable company or a fiber-to-the-home network. Those networks have hard boundaries – there is always a last home served at the edge of the service area after which nobody else is covered. Today’s mapping by census block doesn’t recognize the hard boundaries of these networks and often counts customers outside these networks as having access to fast data speeds. This is particularly a problem in rural areas where a large area outside a small town might be counted as having 100 Mbps or faster broadband when there is no broadband.

Unfortunately, I don’t see the new maps making a big difference for the rest of rural America unless the ISPs providing DSL and fixed wireless service get scrupulously honest with reporting.  I contend that it is difficult, and perhaps impossible to accurately map these technologies – particularly for disclosing the broadband speed available at a given customer location.

Consider DSL. There are several factors that affect the speed of a DSL product. The one everybody knows is that the amount of delivered bandwidth decreases with distance from the DSLAM (the DSL core modem). However, the quality of DSL performance also depends upon the gauge of the copper serving a customer (there are different sizes of copper in a network), the quality of that copper (copper deteriorates over time), issues with the drop wire (drop wires can suffer from a variety of issues separate from issues in the network), the age and type of DSL electronics (there is still plenty of DSL from the 1990s), and the telco technology used on a given copper route to boost or extend signals. There are also customers who can’t get DSL due to the simple issue that a telco has no spare pairs of copper with which to serve them.

It is not unusual for two customers who are side by side to have a drastically different DSL experience – one might have a decent speed and one might not be able to get any DSL service. There is no way for a telco to reflect these highly local conditions on a broadband map. I’m doubtful that the big telcos even track the speeds available to existing customers. The telcos can’t know anything about homes that don’t have their service today.

The same goes for fixed wireless. Broadband speeds also decrease with distance from the tower. Wireless broadband speeds can vary with temperature and humidity. There is a definite fall-off in speed during precipitation. Wireless broadband using unlicensed spectrum is subject to interference, which can mysteriously come and go. The biggest obstacle for many wireless customers is foliage and other obstacles between a customer and the wireless tower. Just like with DSL, wireless companies don’t have any idea what speed they can deliver to a customer who is not on their network. They usually only know what’s available after climbing on a roof to investigate a connection.

Another big issue the FCC didn’t address is reporting of actual speeds. Our examination of the FCC mapping data for both DSL and fixed wireless shows that many ISPs don’t try to report actual broadband speeds. Instead, we see marketing speeds or something other speed standard being reported. Even if these providers map the polygons correctly, we won’t have a good idea of rural broadband coverage unless the ISPs try hard to report actual speeds. We hear from customers all the time that are being sold a rural broadband product that is marketed to deliver speeds of 10 Mbps, 15 Mbps, or 25 Mbps but which delivers only a few Mbps. If the maps don’t reflect the actual speeds they will still be largely worthless.

One last issue is a head-scratcher. Many rural networks are oversubscribed, meaning there are more customers than can comfortably be accommodated at the busiest usage times on the networks. How do you report the broadband speed for a customer who can get 20 Mbps downloads at 4:00 AM but 3 Mbps in the evening?

I applaud the FCC for finally getting rid of the census blocks. But we can’t pretend that this fix is going to make much of a difference for most of rural America. The rural broadband gap is mostly due to the neglected copper networks of the largest telcos. I can’t imagine any way to ever accurately map DSL and fixed wireless technologies., which means the maps are still going to be terrible in the places we most care about. The FCC is still going to harming rural America if they use the new maps to make decisions for important things like awarding grant money. The only real fix is to throw the maps away for those purposes and do something more sensible. For example, grant money ought to always be available to somebody that wants to build fiber to replace big telco copper – we don’t need a map to know that is good policy.

We Need a Challenge Process for Broadband Maps

We all know that the broadband maps maintained by the FCC are terrible. Some of the inaccuracy is due to the fact that the data in the maps come from ISPs. For example, there are still obvious examples where carriers are reporting their marketing speeds rather than actual speeds, which they might not know. Some of the inaccuracy is due to the mapping rules, such as showing broadband by census block – when a few customers in a block have decent broadband it’s assumed that the whole census block has it. Some of the inaccuracy is due to the vagaries of technology – DSL can vary significantly from one house to the next due to the condition of local copper; wireless broadband can vary according to interference and impediments in the line-of-sight. The maps can be wrong due to bad behavior of an ISP who has a reason to either overstate or understate their actual speeds (I’ve seen both cases).

None of this would matter if the maps were just our best guess at seeing the state of broadband in the country. Unfortunately, the maps are used for real-life purposes. First, the maps are used at the FCC and state legislators to develop and support various policies related to broadband. It’s been my contention for a long time that the FCC has been hiding behind the bad maps because those maps grossly overstate the availability of rural broadband. The FCC has a good reason to do so because they are tasked by Congress to fix inadequate broadband.

Recently the maps have been used in a more concrete way and are used to define where grants can or cannot be awarded. Used in this manner the maps are being used to identify groups of homes that don’t already have adequate broadband. The maps were the basis of determining eligible areas for the CAF II reverse auction and now for the e-Connectivity grants.

This is where bad mapping really hurts. Every rural county in the country knows where broadband is terrible or non-existent. When I show the FCC maps to local politicians they are aghast at how inaccurate the maps are for their areas. The maps often show large swaths of phantom broadband that doesn’t exist. The maps will show towns that supposedly have universal 25/3 Mbps broadband or better when the real speeds in the town are 10 Mbps or less. The bad maps hurt every one of these places because if these maps were accurate these places would be eligible for grants to help fix the poor broadband. A lot of rural America is being royally screwed by the bad maps.

Of even more dismay, the maps seem to be getting worse instead of better. For example, in the CAF II program, the big telcos were supposed to bring broadband of at least 10/1 Mbps to huge swaths or rural America. A lot of the areas covered by the CAF II program are not going to see any improvement of broadband speeds. In some cases, the technology used, such as AT&T’s use of fixed cellular can’t deliver the desired speeds to customers who live too far from a tower. I also believe we’re going to find that in many cases the big carriers are electing to only upgrade the low-hanging fruit and are ignoring homes where the CAF upgrade costs too much. These carriers are likely to claim they’ve made the upgrades on the maps rather than admit to the FCC that they pocketed the subsidy money instead of spending it to improve broadband.

There have been a few suggested fixes for the problem. A few states have tried to tackle their own broadband maps that are more accurate, but they can’t get access to any better data from the ISPs. There are a few states now that are asking citizens to run speed tests to try to map the real broadband situation, but unless the speeds tests are run under specific and rigorous conditions they won’t, by themselves, serve as proof of poor broadband.

The easiest fix for the problem is staring us right in the face. Last year the FCC got a lot of complaints about the soon-to-be-awarded Mobility Fund Phase II grants. This money was to go to cellular carriers to bring cell coverage to areas that don’t have it. The FCC maps used for those efforts were even worse than the broadband maps and the biggest cellular companies were accused of fudging their coverage data to try to stop smaller rival cell providers from getting the federal money. The outcry was so loud that the FCC created a challenge process where state and local governments could challenge the cellular coverage maps. I know a lot of governments that took part in these challenges. The remapping isn’t yet complete, but it’s clear that local input improved the maps.

We need the same thing for the FCC broadband maps. There needs to be a permanent challenge process where a state or local government can challenge the maps and can supply what they believe to be a more accurate map of coverage. Once counties understand that they are getting bypassed for federal grant money due to crappy maps they will jump all over a challenge process. I know places that will go door-to-door if the effort can help bring funds to get better broadband.

Unfortunately, only the FCC can order a challenge process, and I don’t think they will even consider it unless they got the same kind of outcry that came with the Mobility II Funding. It’s sad to say, but the FCC has a vested interest in burying their head in the sand and pretending that rural broadband is okay – otherwise they have to try to fix it.

I think states ought to consider this. If a state undertakes a program to allow challenges to the map, then governors and federal legislators can use the evidence gathered to pressure the USDA to accept alternate maps for areas with poor broadband. These challenges have to come from the local level where people know the broadband story. This can’t come from a state broadband mapping process that starts with carrier data. If local people are allowed to challenge the maps then the maps will get better and will better define areas that deserve federal grants. I believe a lot of county governments and small towns would leap at the opportunity to tell their broadband story.

Upgrading FCC Broadband Statistics

The NCTA – The Internet & Television Association that represents the large cable companies and telcos has filed a complaint with the FCC asserting that the agency is not updating broadband maps in a timely manner, and this is understating the amount of broadband deployed in the country.

They have a good point, in that the FCC recently released broadband data from 2016 while they already have received June 2017 data. The recently released data is now more than two years behind the actual broadband deployments in the country.

There may have been years in the past where this kind of time delay didn’t make that much difference, but we are now at a time when there are massive amounts of broadband upgrades happening across the country. The big telcos are well into the CAF II upgrades that are upgrading huge swaths of rural America to speeds of at least 10/1 Mbps. There is a lot of upgrades at smaller telcos that are implementing upgrades from the A-CAM program that requires upgrades to at least 25/3 Mbps – although many of them are upgrading to fiber with gigabit speeds. We now see cable companies starting to implement DOCSIS 3.01 upgrades that can increase their download speeds to a gigabit. And there are numerous overbuilders upgrading broadband all over the place by building fiber or fixed wireless technology. We will soon see the CAF II reverse auctions building yet more rural broadband, with a significant percentage of those upgrades being at 100 Mbps or faster.

This means that the FCC’s broadband maps and the underlying databases are far out of synch and provide the wrong narrative about broadband coverage. The members of NCTA want to get credit for the upgrades they are making, which means that numerous households are no longer considered as unserved, with many of them getting a broadband option for the first time.

There are practical and policy ramifications due to the delay in upgrading the maps. For example, some of the federal loan and grant programs score applicant projects according to whether they are upgrading rural areas that are unserved or underserved – and the FCC data overstates the households that are classified as unserved.

There are also real-life implications for communities. Consider Otter Tail County, Minnesota. Looking at the current FCC maps shows the County with a paltry 2% of households able to get download speeds of 100 Mbps. That is a truthful depiction just looking back a year or two. The cable companies serving the towns in the County have had maximum speeds of no more than 60 Mbps and the rural areas all have broadband using DSL, fixed wireless or satellite.

However, that map doesn’t reflect what’s happening in the County today and what will be happening there in the next few years. Charter has promised to upgrade to faster speeds nationwide and their customers in the County ought to be at speeds far above the 100 Mbps threshold. A lot of the rural areas are served by small telcos that are using A-CAM funding to build fiber. In this past summer alone there were dozens of construction crews building fiber around the County. There are also a few pockets of the County that have gotten upgrades to fiber that were assisted with broadband grants from the State of Minnesota. My quick assessment show that the County will soon have 100 Mbps broadband for 70% to 80% of households when the known upgrades are finished over the next few years. And even most of the areas not getting 100 Mbps broadband will still be seeing speed improvements. That facts on the ground in Otter Tail County paint a drastically different picture than what is shown by the current FCC maps. I have no doubt that this same thing is true in numerous other rural counties.

I understand that the FCC wants to use actual data to create their maps. But I’m mystified why they don’t want to brag about the programs they have sponsored that will improve broadband. It should be easy for them to overlay a map of the expected upgrades that will come from the CAF II and A-Cam programs. These future-looking maps are a better picture of the rural broadband situation.

There are obviously numerous upgrades happening that the FCC can’t know about – they have no way of knowing about upgrades being done with non-FCC funding. But there isn’t much excuse for the FCC to be issuing data and maps that are more than two years out of synch at the date of publication. It’s not a difficult  technical challenge to quickly map ISP broadband data as it’s submitted – numerous states already readily create their own versions of these maps. And it shouldn’t be hard for the FCC to create overlays showing the upcoming successes due to the upgrades they have fostered.

Regulating From Broadband Maps

One of the more bizarre things we do in the US is regulate broadband based upon broadband maps. There are numerous federal grant and subsidy programs that rely upon these maps (and the underlying databases that support them) as well as various state programs. The FCC also uses this same data when reporting broadband penetration in the country to Congress each year, as just occurred on February 9.

The maps are intended to show how many households can purchase broadband of various speeds. Currently the arbitrary speed thresholds tested are download speeds of 10 Mbps, 25 Mbps and 100 Mbps. These speeds are measured due to past decisions by the FCC. For example, the FCC chose a 10/1 Mbps speed goal for any company that accepted CAF II money to upgrade rural broadband. The FCC’s current definition of broadband is still set at 25/3 Mbps.

Anybody that understands broadband networks knows that much of the data included in the databases and the mapping is incorrect, and sometimes pure fantasy. That makes sense when you understand that the speeds in this mapping process are all self-reported by ISPs.

There are numerous reasons why the speeds in these databases are not an accurate reflection of the real world:

  • There are still ISPs that report advertised speeds rather than actual speeds received by customers.
  • Any speeds represented for a whole DSL network are inaccurate by definition. DSL speeds vary according to the size of the copper wires, the condition of the copper cable and the distance from the source of the DSL broadband signal. That means that in a DSL network the speeds available to customers vary street by street, and even house by house. We’ve always known that DSL reported in the mapping databases is overstated and that most telcos that report DSL speeds report theoretical speeds. I’m not sure I blame them, but the idea of any one speed being used to represent the performance of a DSL network is ludicrous.
  • The speeds in the database don’t recognize network congestion. There are still many broadband networks around that bog down under heavy usage, which means evenings in a residential neighborhood. Nobody wants to be told that their network is performing at 10 Mbps if the best speed they can ever get when they want to use it is a fraction of that.
  • The speeds don’t reflect that ISPs give some customers faster speeds. In networks where bandwidth is shared among all users on a neighborhood node, if a few customers are sold a faster-than-normal speed, then everybody else will suffer corresponding slower speeds. Network owners are able to force extra speed to customers that pay a premium for the service, but to the detriment of everybody else.
  • The maps don’t reflect the way networks were built. In most towns you will find homes and businesses that were somehow left out of the initial network construction. For example, when cable companies were first built they largely ignored business districts that didn’t want to buy cable TV. There are lots of cases of apartment and subdivision owners that didn’t allow in the incumbent telco or cable company. And there are a lot of homes that just got missed by the network. I was just talking to somebody in downtown Asheville where I live who is not connected to the cable network for some reason.
  • Not all ISPs care about updating the databases. There are many wireless and other small ISPs that don’t update the databases every time they make some network change that affects speeds. In fact, there are still some small ISPs that just ignore the FCC mapping requirement. At the other extreme there are small ISPs that overstate the speeds in the databases, hoping that it might drive customer requests to buy service.
  • One of the most insidious speed issues in networks are the data bursts that many ISPs frontload into their broadband products. They will send a fast burst of speed for the first minute or two for any demand for bandwidth. This improves the customer experience since a large percentage of requests to use bandwidth are for web searches or other short-term uses of bandwidth. Any customer using this feature will obtain much faster results from a speed test than their actual long-use data speeds since they are actually testing only the burst speed. A rural customer using burst might see 4 Mbps on a speed test and still find themselves unable to maintain a connection to Netflix.
  • Sometimes there are equipment issues. The best-known case of this is a widespread area of upstate New York where Charter has kept old DOCSIS 1.0 cable modems in homes that are not capable of receiving the faster data speeds the company is selling. It’s likely that the faster network speed is what is included in the database, not the speed that is choked by the old modems.
  • And finally, speed isn’t everything. Poor latency can ruin the utility of any broadband connection, to the point where the speed is not that important.

Unfortunately, most of the errors in the broadband databases and maps overstate broadband speeds rather than under-report them. I’ve worked with numerous communities and talk to numerous people who are not able to get the broadband speeds suggested by the FCC databases for their neighborhoods. Many times the specific issue can be pinned down to one of the above causes. But that’s no consolation for somebody who is told by the FCC that they have broadband when they don’t.