Tag Archives: network resiliency

Broadband Resiliency

It’s almost impossible to talk about broadband at the community level without talking about resiliency and redundancy. It’s hard to find rural communities that haven’t experienced a broadband outage due to a fiber being cut somewhere. The issue hits the news when there are reports of regional or national broadband outages.

It’s something that regulators talk about a lot. The FCC talks about having goals for resiliency in many different dockets. The FCC, NTIA, and USDA require that grant proposals promise to promote resiliency. While it’s something that gets talked about, there are no national or state plans for broadband resiliency. There are no specific standards related to resiliency for large networks. There are no national, state, or regional standards for broadband resiliency.

We don’t have to look far to find a similar industry that dealt with the issue. In August 2003, a high-voltage power line in rural Ohio brushed against an untrimmed tree and set off a huge blackout that lasted for two days, and that spread to cover 50 million people in the Midwest, Northeast, and into Canada.

This was a wake-up call for the power industry. At that time, local electric grids were set up to interface automatically with neighboring grids to make it easy to pass excess power from region to region. While that was beneficial to electric companies, it also meant that a problem on one grid could quickly spread to neighboring grids. Electric companies nationwide responded to this and similar large outages by dividing the national electric grid into regional grids that could block rolling blackouts or brownouts. There are still occasional regional electric outages, but even within regions, electric grids have been modified to hinder widespread outages.

When the only communication grids in the country were provided by telephone companies, the communications networks had some similar protections built in. The Bell companies and a few other large telephone companies operated a nationwide network comprised of regional hubs centered around large tandem switching centers. While a local community could lose voice traffic if wires were cut, it was not possible for any local event to knock out multiple regions or have a nationwide impact.

Today’s communications networks are configured differently, both in terms of fiber resiliency and electronics resiliency.

The long-haul fiber grid is comprised of fiber routes built by numerous fiber builders like Lumen (Level 3), Zayo, AT&T, Verizon, large cable companies, and many other regional fiber providers. There is no nationwide planning or coordination for the placement of long-haul fiber. Companies build fiber routes in places and along routes where they think they can make money. The first major fiber long-haul routes were built to connect major and regional Internet POPs. Today, the routes have been extended to reach numerous data centers, which, more often than not, are off the beaten path. From a fiber routing perspective, the long-haul fiber network mostly has route redundancy. If a major fiber gets cut, most of the traffic can be rerouted in different directions.

Redundancy is far more hit-and-miss regionally, or inside a state. Any state with large rural areas can still point out communities that have only one option for fiber backhaul (and even some with none). Redundancy at the regional and local level is often rare, and a fiber cut can knock out a lot of customers or even a whole town or a region. There is also no consistency in pricing, with local middle-mile fiber often priced at exorbitant rates. We know how to create more route resiliency – by building the needed middle-mile fiber network. This is obviously not a national priority since the IIJA legislation provided only $1 billion in grants to address this issue nationwide.

Electronic resiliency has gotten worse over time. The big carriers that operate the Internet have consolidated network operations so that there are only a handful of hubs and a few technicians that monitor the big nationwide networks. This consolidation has greatly increased the risk of large scale outages. A hardware or software failure at one of these hubs can spread and affect networks all over the country. We’ve unfortunately been seeing more of these big outages, which were not possible a few decades ago.

There are no easy solutions for creating the resiliency needed to prevent widespread broadband outages. I fear that the advent of AI could make things worse before it might make things better since it is going to encourage even more consolidation of network operation and monitoring.

Electric companies fixed their grids by getting all of the major electric companies in each region in a room to hammer out a plan to improve resiliency. I’m not sure what it might take to lead the big fiber carriers to have that same conversation – maybe it will take a multiple day catastrophic broadband outage like happened to the electric grids in 2003.

Improving Network Resiliency

The FCC, in Docket FCC 22-50, is requiring changes that it hopes will improve the reliability and resiliency of cellular networks to be better prepared for and respond better to emergencies. The order cites recent emergencies like Hurricane Ida, the earthquakes in Puerto Rico, severe winter storms in Texas, and worsening hurricane and wildfire seasons. This makes me wonder if we might someday see similar requirements for ISPs and broadband networks.

The FCC wants to leverage the industry-developed Wireless Network Resiliency Cooperative Framework as a starting point for introducing new rules it is calling the Mandatory Disaster Response Initiative (MDRI).

The new rules first codify and make mandatory the existing voluntary industry framework to apply to all facility-based mobile wireless providers. That framework includes five principles: (1) providing for reasonable roaming under disaster arrangements when technically feasible; (2) fostering mutual aid among wireless providers during emergencies; (3) enhancing municipal preparedness and restoration by convening with local government public safety representatives to develop best practices and establishing a provider/PSAP contact database; (4) increasing consumer readiness and preparation through development and dissemination with consumer groups of a Consumer Readiness Checklist; and (5) improving public awareness and stakeholder communications on service and restoration status through FCC posting of data on cell site outages on a county-by-county basis.

The new rules require cellular network owners to regularly test its emergency capabilities. This is in response to network failures in some of the disasters mentioned| in the order where network owners were not prepared to deal with an emergency. The new order further requires cellular network owners to file a report with the FCC after every declared emergency to describe in detail how the carrier ended up responding to the emergency.

It’s a change that is overdue because, as the FCC notes, lives are dependent during an emergency on a functioning cellular network. It’s a shame that the FCC has to make such an order. There was a time when big carriers and telcos took social obligations like emergency preparedness seriously and took pride in the ability to respond to emergencies. I can recall decades ago how big telcos would publicize how quickly they were able to restore service, even after disasters completely destroyed central offices and networks.

But as the number of cellular carriers has grown and as the industry is getting more competitive and seeing lower margins, functions like emergency preparedness, that don’t contribute to the bottom line slowly slide through lack of attention and funding.

I suspect at some point that we’ll see similar rules for broadband networks. I’m aware of numerous examples in recent years where failures in the backhaul fiber network have isolated towns from the Internet. I’ve mentioned Project Thor in Colorado a few times, which is a municipally-driven initiative to connect cities in northwest Colorado by fiber, which was prompted by repetitive outages on CenturyLink backhaul networks that were killing Internet access for hospitals, 911 centers, and other public safety critical infrastructure.

One issue the order doesn’t address is that there are still large parts of rural America that have poor or nonexistent cellular coverage. The coverage maps of the big cellular carriers are a joke in much of rural America. My consulting firm does surveys, and it’s not unusual in rural counties to see 30% or more of residents claiming to have no cellular coverage at their home. For these folks, a broadband connection is their lifeline to the world in the way that a cellular connection is vital to others during and after an emergency.