Will Covid-19 Traffic Kill the Internet?

This is the question being asked all across the industry as the volume of data traffic has leaped upward due to students and employees working from their homes. We got our first glimpse of the impact of the crisis when Verizon announced a week into the crisis that they were seeing a 22% increase in data traffic in their network. More recently AT&T announced a 27% increase in network traffic. In perhaps a peek at what might be coming, Italy, which has been in a lockdown for longer than the US has seen a 90% increase in Internet traffic.

The answer to the question differs depending on somebody’s perspective of the network. For example, Evan Swartztrauber, described as an advisor to the FCC, says that the US Internet network is handling the surge in traffic just fine. He says the increased volume is significant, but it’s not at the same level as what is seen during the Superbowl or the finale of Game of Thrones. That’s reassuring news to hear, but he’s talking from the perspective of the big Internet POPs and the long-haul networks that carry Internet traffic from city to city. Even his answer is a bit glib because we’ve just seen more than a year’s growth in traffic in a matter of weeks and there must be places in the Internet backbone that need to be beefed up to meet the increased demand.

The question that matters is if Internet performance is getting worse for the average user, which is a question about the last mile network. I’ve been checking in on clients to understand the impact. So far, everybody with a fiber-to-the-home network says they are weathering the increased volumes, although several clients are looking into increasing bandwidth in a few parts of the network, such as between the core and field huts. Several clients who operate HFC or DSL networks have told me that their biggest problem is with upload speeds. People working from home as well as students are using a lot more upload bandwidth as they communicate with office and school servers. Gamers also need significant upload bandwidth. These technologies were not designed to handle significant amounts of uploaded bandwidth and customer performance is seriously degrading.

Many clients also say that they are increasing the bandwidth needed to connect to the Internet. Luckily most of them can do this easily, but some rural clients are constrained on the ability to easily add more bandwidth.

What nobody is talking about is the last-mile networks that were already broken. For example, I helped a rural county to get citizens to take speed tests right before the pandemic and we found almost no rural households in that county with broadband speed greater than 5 Mbps – and most are far under that modest number. These customers are served with DSL or fixed wireless broadband, and the local telco and WISPs are obviously bandwidth restricted either due to older technology or due to lack of backbone bandwidth.

Rural networks that are already underperforming might easily collapse under increased bandwidth usage. A 30% increase in usage won’t cut speeds by just 30%, the extra usage is likely to crash the networks. A large portion of rural America already has dreadful broadband. There are terrible ramifications if a network that is only delivering 3 Mbps broadband today gets further stressed. Degraded usage means that a home where a student might have been able to connect to a school server before Covid-19 might now be unable to maintain a connection. Good luck to somebody trying to connect to an office server as they work from home for the first time. And considering that some of these stressed rural networks have upload speeds measured in kilobits per second, good luck to anybody wanting to make a video connection for school or working from home.

Perhaps it’s true that the overall US Internet is not in danger yet of collapsing. Networks are going to see additional stress if the shelter-at-home restrictions carry through April and into May or June. What all of the national headlines are missing is that many rural Internet networks were barely functional before the pandemic. I’ve talked to numerous rural businesses in the last year that don’t even have adequate broadband to sustain a credit card transaction. I hear from homes across the country where the Internet is too slow, or latency too high to sustain connections to a school network to do homework. The current burst in new traffic is going to mean that the Internet performance for many rural users is going to go from barely functional to non-functional.

We might see a little relief if some of the biggest bandwidth users of the web cut back on broadband demand. Google announced that they are going to reduce the quality of video signals from YouTube as a way to cut back on the volume of data hitting networks. There is pressure on Netflix to do the same. AT&T announced that Netflix’s traffic volumes have hit an all-time high. Netflix announced that it is going to reduce traffic volumes by 25% in Europe but hasn’t made the same claim yet for the US. Unfortunately, these fixes are unlikely to make a big difference. the problems in last mile networks is due to having many more Internet users than before the pandemic, and the sheer number of users along with their attempts at using bandwidth-hungry applications is going to kick rural networks in the teeth.

This pandemic has highlighted the horrendous inadequacies of rural broadband. The shortfalls of rural broadband already existed, but with the added traffic volumes, rural broadband is going to significantly worsen. Unfortunately, we didn’t see much funding to help rural broadband as part of the recent stimulus plan. I’m pretty sure politicians with rural constituents are going to hear a lot about this – at least constituents with enough bandwidth to tell their story.

Expect a New Busy Hour

One of the many consequences of the coronavirus is that networks are going to see a shift in busy hour traffic. Busy hour traffic is just what is sounds like – it’s the time of the day when a network is busiest, and network engineers design networks to accommodate the expected peak amount of bandwidth usage.

Verizon reported on March 18 that in the week since people started moving to work from home that they’ve seen a 20% overall increase in broadband traffic. Verizon says that gaming traffic is up 75% as those stuck at home are turning to gaming for entertainment. They also report that VPN (virtual private network) traffic is up 34%. A lot of connections between homes and corporate and school WANs are using a VPN.

These are the kind of increases that can scare network engineers, because Verizon just saw a typical year’s growth in traffic happen in a week. Unfortunately, the announced Verizon traffic increases aren’t even the whole story since we’re just at the beginning of the response to the coronavirus. There are still companies figuring out how to give secure access to company servers and the work-from-home traffic is bound to grow in the next few weeks. I think we’ll see a big jump in video conference traffic on platforms like Zoom as more meeting move online as an alternative to live meetings.

For most of my clients, the busy hour has been in the evening when many homes watch video or play online games. The new paradigm has to be scaring network engineers. There is now likely going to be a lot of online video watching and gaming during the daytime in addition to the evening. The added traffic for those working from home is probably the most worrisome traffic since a VPN connection to a corporate WAN will tie up a dedicated path through the Internet backbone – bandwidth that isn’t shared with others. We’ve never worried about VPN traffic when it was a small percentage of total traffic – but it could become one of the biggest continual daytime uses of bandwidth. All of the work that used to occur between employees and the corporate server inside of the business is now going to traverse the Internet.

I’m sure network engineers everywhere are keeping an eye on the changing traffic, particularly to the amount of broadband used during the busy hour. There are a few ways that the busy hour impacts an ISP. First, they must buy enough bandwidth to the Internet to accommodate everybody. It’s typical to buy at least 15% to 20% more bandwidth than is expected for the busy hour. If the size of the busy hour shoots higher, network engineers are going to have to quickly buy a larger pipe to the Internet, or else customer performance will suffer.

Network engineers also keep a close eye on their network utilization. For example, most networks operate with some rule of thumb, such as it’s time to upgrade electronics when any part of the network hits some pre-determined threshold like 85% utilization. These rules of thumb have been developed over the years as warning signs to provide time to make upgrades.

The explosion of traffic due to the coronavirus, might shoot many networks past these warning signs and networks start experiencing chokepoints that weren’t anticipated just a few weeks earlier. Most networks have numerous possible chokepoints – and each is monitored. For example, there is usually a chokepoint going into neighborhoods. There are often chokepoints on fiber rings. There might be chokepoints on switch and router capacity at the network hub. There can be the chokepoint on the data pipe going to the world. If any one part of the network gets overly busy, then network performance can degrade quickly.

What is scariest for network engineers is that traffic from the reaction to the coronavirus is being layered on top of networks that already have been experiencing steady growth. Most of my clients have been seeing year-over-year traffic volumes increases of 20% to 30%. If Verizon’s experience in indicative of what we’ll all see, then networks will see a year’s typical growth happen in just weeks. We’ve never experienced anything like this, and I’m guessing there aren’t a lot of network engineers who are sleeping well this week.

FCC Reports on Poor Rural 4G Coverage

The FCC released a report in January that shows that the cellular networks of the major carriers underperform in rural America. This is no news to anybody who lives and works in a rural county. The tests allowed the FCC to conclude that the national coverage maps for 4G LTE are largely fiction in rural America.

The FCC conducted 25,000 tests in twelve states to verify the coverage maps of Verizon, T-Mobile, and US Cellular. The majority of tests were done in Arizona, New Mexico, Oklahoma, Vermont, Alabama and Montana. Speeds were tested from both stationary locations and in a moving vehicle. AT&T and Sprint weren’t tested because the maps they provided to the FCC showed only the combined upload and download speeds – something that is meaningless to test. The other three carriers reported what they claimed were actual upload and download speeds, shown separately.

The FCC undertook the testing in response to numerous complaints filed in the FCC’s docket for the Mobility Fund Phase II grants. The intention of this fund was to improve 4G coverage in rural areas with little or no cellular coverage. Smaller cellular carriers and the public complained to the FCC that the cellular data coverage claimed by the large cellular carriers was overstated. Small cellular carriers worried that the overstatements would stop them from asking for funding for areas that need upgrading. Local governments were worried that the overstated coverage meant that their areas wouldn’t see upgrades and they’d be doomed for another decade with poor cellular coverage.

The tests were conducted in areas where the carrier maps showed cellular data coverage. The results of the testing were rather bleak. 16% of all calls tried on Verizon were unable to make a data connection. The failures to connect were 23% on T-Mobile and 38% on US Cellular.

Overall, the three carriers met the FCC’s minimum requirement of 5 Mbps download for 4G only 62% of the time. That was 64% on Verizon, 63% on T-Mobile and only 45% for US Cellular. However, even within those reported results, the testers said that they experienced intermittent dropped calls on all three networks.

The FCC responded to these tests by revamping the reporting of cellular data speeds in the future, asking for far more granular speed data by location. The FCC also convened a group of experts to recommend to the FCC how to better test cellular speeds. Finally, the FCC issued an Enforcement Advisory on the accuracy of the cellular data on form 477. That’s a step short of issuing fines and likely will have little impact on the carriers. It doesn’t appear that any of them have pared back their national coverage maps that still claim coverage across most of rural America.

There are significant real-life implications of overstated cellular coverage maps. Just like with the RDOF grant program that will rely on faulty maps of landline broadband, poor maps of cellular coverage mean that many areas with overstated cellular coverage won’t be eligible for federal grants to help fix the problem.

The big downside is that many rural households have no 4G LTE coverage, or at best have slow and intermittent 4G data available. These are often the same areas where landline broadband is slow or non-existent. As hard as it is to live without good cellular coverage or good landline broadband, homes without both are cut off from the rest of the world. To make matters worse, there is still 3G coverage in a lot of rural America and all of the carrirs have plans to cut that dead over the next few years.

The FCC has revamped the Mobility Fund II grant program by doubling the amount of funding to $9 billion and renaming it as the 5G Fund. That’s a silly name because the goal of the program is to bring at least minimal 4G coverage to rural areas, not 5G. Remember that the grant program was originally aimed only at areas that showed no coverage by the carriers. Ideally the FCC would also  direct funding to the many areas where the carriers were lying about their coverage – but It’s doubtful that they have any meaningful maps of real 4G coverage.

Can 5G Replace WiFi?

Verizon recently posted a webcast with investors where Ronan Dunne, EVP and CEO of the Verizon Consumer Group said that he believed that 5G hotspots using millimeter wave spectrum will eventually displace WiFi in homes.

He cites major benefits of 5G over WiFi. He believes that a 5G network will be more reliable and more secure. He thinks that people will value the safety that comes from having traffic inside their home being encrypted as it rides Verizon’s 5G network compared to the more public nature of WiFi where every neighbor can see a home’s WiFi network.

He also cites the convenience of being able to transfer 5G traffic between networks. He paints a picture where a customer making a call or watching a video using a home 5G hotspot will be able to walk out the door and seamlessly continue the session outside on their cellphone. That’s pretty slick stuff should that ever come to pass.

The picture he’s painting for Verizon investors is a future where homes buy a Verizon 5G subscription to use in place of WiFi. This is part of Verizon’s ongoing effort to find a business case for 5G. His vision of the future is possible, but there are a lot of hurdles for Verizon to overcome to achieve that vision.

It’s going to get harder to compete with WiFi since the technology is getting a lot better with two major upgrades. First, the industry has introduced WiFi 6, which brings higher quality performance, lower latency, and faster data rates. WiFi 6 will use techniques like improved beamforming to greatly reduce interference between WiFi uses within the home.

Even more importantly, WiFi will be incorporating the new 6 GHz spectrum band that will increase bandwidth capabilities by adding seven 160 MHz bands and fourteen 80 MHz bands. It will be much easier to put home devices on separate channels when these new channels are added to the existing channels available on 2.4 and 5 GHz. This means that 5G will be competing against a much improved WiFi compared to the technology we all use today.

Another big hurdle for Verizon to overcome is that WiFi is ubiquitous today. WiFi is built into a huge number of devices, and a homeowner might already own a dozen or more devices capable of using WiFi. Verizon will have to somehow convince homeowners that 5G is so superior that it’s worth replacing the panoply of WiFi devices.

Another hurdle is that there is going to be WiFi vendors painting almost the same picture as Verizon. The makers of WiFi routers are already envisioning future devices that will introduce millimeter-wave spectrum including 5G into the home. There are vendors already working on devices that will provide both WiFi 6 and 5G using millimeter-wave connections simultaneously, using the publicly available 60 GHz V band. These solutions envision offering everything that Verizon can do, except the ability to roam seamlessly in and out of a home – and it will be done by selling a box instead of a new monthly subscription.

Another interesting hurdle to switching home networks to 5G is that there might be separate 5G solutions for each cellular carrier that uses different bands of spectrum. It’s relatively easy for device makers today to build a cellphone or other device that can use different cellular carriers because the carriers all use similar spectrum. But as each cellular company picks a different mix of frequencies moving forward, there is likely going to be cellphones and other devices that are specific to one carrier. It’s impossible to build a cellphone with today’s battery technology that can receive a huge range of spectrums – the multiple antenna systems would drain a cellphone dry in no time.

The largest hurdle of all is that WiFi is free to use after buying a WiFi router or meshed WiFi devices for the home. There is no monthly subscription fee to use the wireless WiFi connections within the home. Verizon clearly foresees a world where every home has a new monthly subscription to use its in-home 5G network.

Mr. Dunne makes one good point. It’s becoming increasingly clear that public WiFi networks are susceptible to hacking. A 5G network controlled by a carrier should be a lot safer than a WiFi hotspot managed by a coffee shop. The big question is if this enough incentive for people to buy 5G-capable devices or for coffee shops to switch to 5G networks. Even should coffee shops go with a 5G solution, will homes follow suit?

Mr. Dunne vision has an underlying assumption that people will value data security enough to be willing to pay more for it. He envisions people choosing a managed network when they have a choice. He could be right, and perhaps there will be enough data breaches in coming years with WiFi that the paradigm will change from WiFi to 5G. But it’s going to be incredibly hard to dislodge WiFi, particularly when it’s evolving and improving along with 5G.

Even if Mr. Dunne is right, this shift is not coming soon, probably not within this decade. For now, WiFi has won the device war and any shift to 5G would drag out over many years. It’s going to be incredibly difficult for the cellular carriers to convince everybody to switch to 5G.

I sympathize with Mr. Dunne’s dilemma. Investors want to understand where the revenues will come from to fund the expensive upgrades to 5G. Verizon and the other cellular carriers have tossed out a lot of ideas, but so far none of them have stuck to the wall.  Investors are getting rightfully nervous since there doesn’t appear to be any significant 5G revenues coming in the next few years. The carriers keep painting pictures of an amazing 5G future as a way to not have to talk about lack of 5G revenues today.

Is it Time for Honest Pricing?

Verizon is getting a lot of positive press from changing its product pricing to be more transparent. I look at the new pricing structure. and see both plusses and minuses.

The new pricing is a straightforward menu of prices as follows:

  • There are three broadband products: 100 Mbps for $40, 300 Mbps for $60 and 1 gigabit for $80. The first two products charge $15 monthly for a router – a router is included in the gigabit product price.
  • There are new, and lower-priced cable TV options. Your FiOS TV is $50 (plus $12 for a settop box). A customer chooses 5 channels and Verizon provides 120 other channels based upon that choice. More FiOS TV provides 300 channels for $70 and includes one free settop box. The Most FiOS TV is $90 and comes with 425 channels, one free settop box and some use of a DVR. Verizon is also now reselling YouTube TV for $50.
  • A basic voice line is $20 and comes with caller ID. I assume voice mail is extra. They didn’t include it in the announcement, but there must be a higher-priced product that includes unlimited long distance.
  • Customers must use autopay and paperless billing to get any of these products.

There are some definite positives from the new pricing:

  • Verizon says they have eliminated hidden fees.
  • Verizon has eliminated term contracts for these products. It’s not clear in the announcement, but customers under current contracts are likely going to have to finish those contracts before moving to the new prices.
  • This eliminates the games that Verizon and other big ISPs have played with bundling discounts. With bundle discounts, customers got some nice price breaks for buying multiple products. However, the discounts were never associated with any product, and customers found that when they tried to drop any one product that they lost the bundling discount. This new pricing is a menu and customers can pick what they want to buy, and due to monthly billing can add or subtract products later with no penalty.
  • Along those same lines, Verizon will have finally taken out all of the hassles of trying to negotiate for standalone products. Customers can pick any products from this new menu of services, including standalone broadband.

Of course, there are also negatives:

  • The settop box and router fees at $12 and $15 are outrageous considering that in both cases the box that Verizon is providing likely costs around $100. These add-ons costs are still going to be mentioned only in the small print in advertising, which still smacks of hidden fees.
  • Verizon is also including a $15 per month product they call TechSure Plus that provides for 24/7 technical support. Reading between the lines, this product means customers will have to pay Verizon $180 per year to avoid the long phone waits for customer service. The unspoken threat is that customers without this service will go to the end of the customer service queue. It’s a ballsy product statement by Verizon – pay extra if you ever want to talk to us.
  • All of this is only available where Verizon has fiber – and there is still a lot of their market that is not wired with fiber.

The new pricing is a definite challenge to the big cable companies that Verizon competes with. The $55 price for 100 Mbps broadband ($40 for the broadband and $15 for the router) sets a market bottom price and is a definite challenge to the cable companies. It’s likely that a big majority of Verizon customers will choose this product because of the low price and because most homes today are going to be happy with 100 Mbps service on fiber. This product will make the big cable companies sharpen their pencils for their base broadband product and also might make them hesitate from the annual broadband rate increases they’ve now built into their planning.

It’s going to be interesting to see how Comcast and the other big cable companies react to these new prices. They can advertise promotional pricing that can beat the Verizon rates, but those specials will likely still include hidden fees, and the rates will spring back to full price at the end of the promotional period. The big cable companies also need to be careful about offering lower prices only where there is FiOS – this will annoy the hell out of customers in other markets who will understand they are subsidizing lower rates in Verizon markets.

It’s not going to be surprising to see Verizon take away customers from the cable companies with these prices. The prices are not particularly low, but for the most part, they are honest and transparent – a refreshing change from a big ISP.

The Frontier Bankruptcy

Bloomberg reported that Frontier Communications is hoping to file a structured bankruptcy in March. A structured bankruptcy is one where existing creditors agree to cut debt owed to them to help a company survive. There is no guarantee that the existing creditors will go along with Frontier’s plan, and if not, the bankruptcy would be handed to a bankruptcy court to resolve.

It’s been obvious for a long time that Frontier is in trouble. Three years ago, the stock sat at over $51 per share. By January 2018 it had fallen to $8.26 per share, and to $2 per share a year ago. As I write this blog the stock sits at 59 cents per share.

Frontier has been losing customers rapidly. In the year ending September 30, 2019 the company lost 6% of its broadband customers (247,000), with 71,000 of the losses occurring during the third quarter of last year.

For those not familiar with the history of Frontier, the company started as Citizens Telephone Company, a typical small independent telco. The company grew by buying telephone customers from GTE, Contel, and Alltel. The company became Frontier when they bought the remains of the Rochester Telephone Company from Global Crossings. Since then Frontier went on a buying spree and purchased large numbers of customers from Verizon.

Frontiers woes intensified in 2016 when they bungled the takeover of Verizon FiOS customers while taking on huge debt. There were major outages in some major markets that drove customers to change to the cable company competitor. However, Frontier’s biggest problem is due to operating a lot of rural copper networks. The copper networks they purchased had been maintained poorly before acquired by Frontier. For example, Frontier bought all of the Verizon customers in West Virginia, and Verizon had been ignoring the market and had been trying to sell it for over fifteen years.

Frontier got a small boost when the FCC gave them $1.7 billion to upgrade rural DSL to speeds of at least 10/1 Mbps. This month Frontier reports that it has not fully met that requirement in parts of thirteen states. Customers in many places where Frontier has supposedly made the upgrades are saying that speeds are not yet at the required 10/1 Mbps.

Frontier’s real problem is that their rural properties are being overbuilt by other ISPs. For example, Frontier properties are the targets of funding for many state broadband grants. Most of the rural Frontier network is going to be targeted in the upcoming $16 billion RDOF grants this year. It would not be surprising to see the company quietly disappear from rural America as others build better broadband.

Meanwhile, other than in properties that formerly were Verizon FiOS on fiber, the company’s networks in towns are also providing DSL. We’ve seen every telco that offers DSL in urban areas like AT&T and CenturyLink lose a lot of customers year-after-year to the cable companies. It’s increasingly difficult for DSL to keep customers with speeds between 10 Mbps and 50 Mbps when competing against cable products of 100 Mbps and higher.

Last May, Frontier announced the sale of its properties in Washington, Oregon, Idaho and Montana to WaveDivision Capital. That sale was for $1.35 billion, which doesn’t make a big dent in the company’s $16.3 billion in long-term debt. Frontier has also shed 10% of its workforce in an attempt to control costs.

Frontier may get the structured bankruptcy they are seeking or may have to give up more to survive this current bankruptcy. However, restructuring their debt is not going to make up for the huge amounts of its network that sits on dying copper. They are not the only company facing this issue and CenturyLink has even more rural copper. However, CenturyLink has a thriving business in big cities and would be stronger if regulators ever allow it to walk away from rural copper.

The harder question to answer is if there is a viable company remaining after Frontier finally sheds or loses its rural customer base. I don’t know enough to make any prediction on that, but I can predict that the company’s problems will not be over even after making it through this bankruptcy.

Taking Advantage of the $9B 5G Fund

The FCC will be moving forward with the $9 billion 5G Fund – a new use of the Universal Service Fund – that will be providing money to expand cellular coverage to the many remote places in the US where 4G cell coverage is still spotty or nonexistent. There is a bit of urgency to this effort since the big cellular companies all want to shut down 3G within a year or two. This money will be made available to cellular carriers, but the funding still opens up possible benefits for other carriers and ISPs.

Some of this funding is likely to go towards extending fiber into rural places to reach cell towers, and that opens up the idea of fiber sharing. There are still a lot of places in the country that don’t have adequate fiber backhaul – the data pipes that bring traffic to and from the big hubs for the Internet. In the last six months alone I’ve worked with three different rural projects where lack of backhaul was a major issue. Nobody can consider building broadband networks in rural communities if the new networks can’t be connected to the web.

By definition, the 5G Fund is going to extend into rural places. If the FCC was maximizing the use of federal grant funds, they would demand that any fiber built with this new fund would be available to others at reasonable rates. This was one of the major provisions of the middle mile networks built a decade ago with stimulus funding. I know of many examples where those middle mile routes are providing backhaul today for rural fixed wireless and fiber networks. Unfortunately, I don’t see any such provisions being discussed in the 5G Fund – which is not surprising. I’m sure the big cellular companies have told the FCC that making them share fiber with others would be an inconvenience, so this idea doesn’t seem to be included in the 5G Fund plan.

I think there is a window of opportunity to partner with wireless carriers to build new fiber jointly. The cellular carriers can get their portion of new fiber funded from the 5G Fund and a partner can pick up new fiber at a fraction of the cost of building the route alone. This could be the simplest form of partnership where each party owns some pairs in a joint fiber.

This is worth considering for anybody already thinking about building rural fiber. The new routes don’t have to be backhaul fiber and could instead be a rural route that is part of a county-wide build-out or fiber being built by an electric cooperative. If somebody is considering building fiber into an area that has poor cellular coverage, the chances are that there will be 5G Fund money coming to that same area.

It has always been challenging to create these kinds of partnerships with AT&T and Verizon, although I am aware of some such partnerships. Both Sprint and T-Mobile have less rural coverage than the other carriers and might be more amenable to considering partnerships – but they might be consumed by the possibility of their merger.

There are a lot of other cellular carriers. The CTIA, the trade association for the larger cellular carriers, has thirty members that are facility-based cellular providers. The Competitive Carriers Association (CCA) has over one hundred members.

Ideally, a deal can be made to share fiber before the reverse auction for the 5G Fund. Any carrier that has a partner for a given route will have a bidding advantage since cost-sharing with a partner will lower the cost of building new fiber. It might be possible to find partnerships after the auction, but there could be restrictions on the newly built assets as part of the grants – we don’t know yet.

My recommendation is that if you are already planning to build rural fiber that you look around to see if one of the cellular carriers might be interested in serving the same area. Both parties can benefit through a cost-sharing partnership – but the real winners are rural customers that gain access to better cellular service and better broadband.

Using Wireless Backhaul

Mike Dano of Light Reading reports that Verizon is considering using wireless backhaul to reach as many as 20% of small cell sites. Verizon says they will use wireless backhaul for locations where they want to provide 5G antennas but can’t get fiber easily or affordably. The article sites an example of using wireless backhaul to provide connectivity where it’s hard to get the rights-of-way to cross railroad tracks.

This prompts me today to write about the issues involved with wireless backhaul. Done well it can greatly expand the reach of a network. Done poorly it can degrade performance or cause other problems. This is not an anti-Verizon blog because they are one of the more disciplined carriers in the industry and are likely to deploy wireless backhaul the right way.

Dano says that Verizon has already addressed one issue that is of concern today to municipalities that are seeing small cell deployments. Cities are worried about small cell devices that are large and unsightly. There are already pictures on the web of small cells gone awry where a mass of different electronics are pole-mounted to create an unsightly mess. Verizon describes their solution as integrated, meaning that no additional external antennas are needed – implying that the backhaul is likely using the same frequencies being used to reach customers. The small cell industry would do well to take heed of Verizon’s approach. It looks like courts are siding with municipalities in terms of being able to dictate aesthetic considerations for small cells.

Another issue to consider is the size of the wireless backhaul link. For instance, if Verizon uses millimeter wave backhaul there is a limitation today of being able to deliver about 1-gigabit links for 2 miles or 2-gigabit links for about a mile. The amount of bandwidth and the distance between transmitters differ according to the frequency used – but none of the wireless backhaul delivery technologies deliver as much bandwidth as fiber. Verizon has been talking about supplying 10-gigabit links to cell sites using next-generation PON technology. Wireless backhaul is going to be far less robust than fiber. This is likely not an issue today where many cell sites are using less than 2 gigabits of bandwidth. However, as the amount of broadband used by cellular networks keeps doubling every few years it might not take long for many cell sites to outgrow a wireless backhaul link.

The primary issue with wireless backhaul is the bandwidth dilution from feeding multiple wireless sites from one fiber connection. Consider an example where one cell site is fiber-fed with a 10-gigabit fiber backhaul. If that site them makes 2-gigabit wireless connections to four other cell sites, each of the 5 sites is now upward limited to 2 gigabits of usage. The bandwidth of the four secondary sites is limited by the 2-gigabit link feeding each one. The core site loses whatever bandwidth is being used by the other sites.

That’s probably a poor example because today most cell sites use less than 2 gigabits of bandwidth. Verizon’s use of 10-gigabit fiber backhaul moves them ahead of the rest of the industry that has cell sites with 1- to 5-gigabit backhaul connections today. The weaknesses of wireless backhaul are a lot more apparent when the wireless network beings at a site that only has a 1- or 2-gigabit fiber connection.

I’m sure that over time that Verizon plans to build additional fiber to relieve network congestion. Their use of wireless backhaul is going to push off the need for fiber by a decade or more and is a sensible way to preserve capital today.

The issues with wireless backhaul are far more critical for carriers that don’t have Verizon’s deep pockets, fiber networks, or discipline. It’s not hard today to find wireless networks that have overdone wireless backhaul. I’ve talked to numerous rural customers who are buying fixed wireless links from WISPs who are delivering only a few Mbps of bandwidth. Some of these customers are getting low speeds because they live too far away from the transmitting tower. Sometimes speeds are low because a WISP oversold the local antenna and is carrying more customers than the technology comfortably can serve.

But many rural wireless systems have slow speeds because of overextended wireless backhaul. In many cases in rural America, there are no fiber connections available for fixed wireless transmitters, which are often installed on grain elevators, water towers, church steeples or tall poles. I’ve seen networks that are making multiple wireless hops from a single gigabit fiber connection.

I’ve also seen preliminary designs for wireless ‘mesh’ networks where pole-mounted transmitters will beam wireless broadband into homes. Every wireless hop in these networks cuts the bandwidth in half at both radio sites (as bandwidth is split and shared). If you feed a mesh wireless network with a gigabit of bandwidth, then by the fifth hop a transmitter only sees 62 Mbps of raw bandwidth (which is overstated because by not accounting for overheads). It’s not hard to do the math to see why some rural wireless customers only see a few Mbps of bandwidth.

I’m sure that Verizon understands that many of the cell sites they serve today wirelessly will eventually need fiber, and I’m sure they’ll eventually build the needed fiber. But I also expect that there will be networks built with inadequate wireless backhaul that will barely function at inception and that will degrade over time as customer demand grows.

Shame on the Regulators

It’s clear that even before the turn of this century that the big telcos largely walked away from maintaining and improving residential service. The evidence for this is the huge numbers of neighborhoods that are stuck with older copper technologies that haven’t been upgraded.  The telcos made huge profits over the decades in these neighborhoods and ideally should not have been allowed to walk away from their customers.

In the Cities. Many neighborhoods in urban areas still have first or second-generation DSL over copper with fastest speeds of 3 Mbps or 6 Mbps. That technology had a shelf-life of perhaps seven years and is now at least fifteen years old.

The companies that deployed the most DSL are AT&T and CenturyLink (formerly Quest). The DSL technology should have been upgraded over time by plowing profits back into the networks. This happened in some neighborhoods, but as has been shown in several detailed studies in cities like Cleveland and Dallas, the faster DSL was brought to more affluent neighborhoods, leaving poorer neighborhoods, even today, with the oldest DSL technology.

The neighborhoods that saw upgrades saw DSL speeds between 15 Mbps and 25 Mbps. Many of these neighborhoods eventually saw speeds as fast as 50 Mbps using a technology that bonded two 25 Mbps DSLs circuits. There are numerous examples of neighborhoods with 50 Mbps DSL sitting next to ones with 3 Mbps DSL.

Verizon used a different tactic and upgraded neighborhoods to FiOS fiber. But this was also done selectively although Verizon doesn’t seem to have redlined as much as AT&T, but instead built FiOS only where the construction cost was the lowest.

In Europe, the telcos decided to complete with the cable companies and have upgraded DSL over time, with the fastest DSL today offering speeds as fast as 300 Mbps. There is talk coming out of DSL vendors talking about ways to goose DSL up to gigabit speeds (but only for short distances). The telcos here basically stopped looking at better DSL technology after the introduction of VDSL2 at least fifteen years ago.

By now the telcos should have been using profits to build fiber. AT&T has done this using the strategy of building little pockets of fiber in every community near to existing fiber splice points. However, the vast majority of rural households served by AT&T are not being offered fiber, and AT&T said recently that they have no plans to build more fiber. CenturyLink built fiber to past nearly 1 million homes a few years ago, but that also seems like a dead venture going forward. But now, in 2019, each of these telcos should have been deep into urban neighborhoods in their whole service area with fiber. Had they done so they would not be getting clobbered so badly by the cable companies that are taking away millions of DSL customers every year.

Rural America. The big telcos started abandoning rural America as much as thirty years ago. They’ve stopped maintaining copper and have not voluntarily made any investments in rural America for a long time. There was a burst of rural construction recently when the FCC gave them $11 billion to improve rural broadband to 10/1 Mbps – but that doesn’t seem to be drawing many rural subscribers.

It’s always been a massive challenge to bring the same speeds to rural America that can be provided in urban America. This is particularly so with DSL since the speeds drop drastically with distance. DSL upgrades that could benefit urban neighborhoods don’t work well in farmland. But the telcos should have been expanding fiber deeper into the network over time to shorten loop lengths. Many independent telephone companies did this the right way and they were able over time to goose rural DSL speeds up to 25 Mbps.

The big telcos should have been engaging in a long-term plan to continually shorten rural copper loop lengths. That meant building fiber, and while shortening loop lengths they should have served households close to fiber routes with fiber. By now all of the small towns in rural America should have gotten fiber.

This is what regulated telcos are supposed to do. The big telcos made vast fortunes in serving residential customers for many decades. Regulated entities are supposed to roll profits back into improving the networks as technology improves – that’s the whole point of regulating the carrier of last resort.

Unfortunately, the industry got sidetracked by competition from CLECS. This competition first manifested in competition for large business customers. The big telcos used that competition to convince regulators they should be deregulated. Over time the cable companies provided real residential competition in cities, which led to the de facto total deregulation of telcos.

In Europe, the telcos never stopped competing in cities because regulators didn’t let them quit. The telcos have upgraded to copper speeds that customers still find attractive, but the telcos all admit that the next upgrade needs to be fiber. In the US, the big telcos exerted political pressure to gain deregulation at the first hint of competition. US telcos folded and walked away from their customers rather than fighting to maintain revenues.

Rural America should never have been deregulated. Shame on every regulator in every state that voted to deregulate the big telcos in rural America. Shame on every regulator that allowed companies like Verizon palm off their rural copper to companies like Frontier – a company that cannot succeed, almost by definition.

In rural America the telcos have a physical network monopoly and the regulators should have found ways to support rural copper rather than letting the telcos walk away from it. We know this can be done by looking at the different approaches taken by the smaller independent telephone companies. These small companies took care of their copper and most have now taken the next step to upgrade to fiber to be ready for the next century.

The Market Uses for CBRS Spectrum

Spencer Kurn, an analyst for New Street Research recently reported on how various market players plan to use the 3.5 GHz CBRS spectrum recently approved by the FCC. I described the FCC’s order in this recent blog. As a quick refresher, this is a large swath of spectrum and the FCC has approved 80 MHz of spectrum for public use and will be auctioning 70 MHz of the spectrum in 2020.

Cellular Bandwidth. Kurn notes that Verizon plans to use the new spectrum to beef up 4G bandwidth now and eventually 5G. Verizon plans to use the spectrum in dense markets and mostly outdoors. Cable companies like Comcast and Charter that have entered the wireless business are also likely to use the spectrum in this manner.

I’ve been writing for a while about the crisis faced by cellular network. In urban areas they are seeing broadband usage double almost every two years and keeping up with that growth is a huge challenge. It’s going to require the combination of new spectrum, more cell sites (mostly small cells), and the improvements that come with 5G, mostly the frequency slicing.

It’s interesting that Verizon only sees this as an outdoor solution, but that makes sense because this spectrum is close in characteristics as the existing WiFi bands and will lose most of its strength in passing through a wall. It also makes sense that Verizon will only do this in metro areas where there is enough outdoor traffic for the spectrum to make a difference. I’ve seen several studies that say that the vast majority of cellular usage is done indoors in homes, businesses, and schools. But this spectrum still becomes one more piece of the solution to help relieve the pressure on urban cell sites.

For this to be of use the spectrum has to be built into cellular handsets. Apple recently announced that they are building the ability to receive Band 48 of CBRS into their new models. They join the Samsung Galaxy S10 and the Google Pixel 3 with the ability to use the spectrum. Over time it’s likely to be built into many phones, although handset manufacturers are always cautious because adding new spectrum bands to a handset increases the draw on the batteries.

Point-to-Multipoint Broadband. Numerous WISPs and other rural ISPs have been lobbying for the use of the spectrum since it can beef up point-to-multipoint broadband networks. These are networks that put a transmitter on a tower and then beam broadband to a dish on a subscriber premise. This technology is already widely in use mostly using the 2.4 GHz and 5.0 GHz WiFi spectrum. Layering on CBRS will beef up the broadband that can be delivered over a customer link.

It will be interesting to see how that works in a crowded competitive environment. I am aware of counties today where there are half a dozen WISPs all using WiFi spectrum and the interference degrades network performance for everybody. There are five SAS Administrators named by the FCC that will monitor bandwidth usage and who also will monitor interference. The FCC rules don’t allow for indiscriminate deployment of public CBRS spectrum and we’ll have to see how interference problems are dealt with.

One interesting player in the space will be AT&T who intends to layer the frequency onto their fixed wireless product. AT&T widely used the technology to meet their CAF II buildout requirements and mostly has used PCS spectrum to meet the FCC requirement to deliver at least 10/1 Mbps speeds to customers. Adding the new spectrum should significantly increase rural customer speeds – at least for those with a few miles of AT&T towers.

Cable Company Edge-out. The most interesting new players considering the market are the cable companies. Kurn believes that the big cable companies will use the spectrum to edge out to serve rural customers with fixed wireless around their existing cable networks. He says the cable networks could theoretically pass 6 – 7 million new homes if this is deployed everywhere. This is an ideal application for a cable company because they typically have fiber fairly close the edge of their service areas. The point-to-point wireless product operates best when the radios are fiber-fed and cable companies could deliver a product in the 50-100 Mbps range where they have line-of-sight to customers.

We’ve already seen one cable company tackle this business plan. Midco was awarded $38.9 million in the CAF II reverse auctions to deploy 100 Mbps broadband in Minnesota and the Dakotas. Midco is going to need this spectrum, and probably even more to deliver 100 Mbps to every customer. Their deployment is not really an edge-out, and the company plans to build networks that will cover entire rural counties with fixed wireless broadband.