Limiting Large Network Outages

Ookla recently published an interesting article that emphasizes what I have been telling folks for a long time. Not that many years ago, telephone and broadband networks were structured in such a way that most outages were local events. A fiber cut might kill service to a neighborhood; an electronics failure might kill service to a larger area, but for the most part, outages were contained within a discrete and local area.

There were exceptions. Rural areas have been susceptible to fiber cuts in the fiber that provides Internet backbone. Years ago, I worked with Cook County, Minnesota, which would lose voice and broadband every time there was a cut in the single fiber between Minneapolis and northern Minnesota that supported the area. A public-private partnership was created to build the THOR network to solve backhaul failures in a large chunk of southeastern Colorado.

https://www.ookla.com/articles/building-digital-resilience-strategies-2025

As the article points out, this has all changed because network operators have consolidated and interconnected networks across large geographic areas. Ookla says that the new phenomenon of large scale outages is a direct result of digital transformation. As carriers, companies, and governments have grown increasingly reliant on cloud services, managed providers, and interconnected networks, they now have to live with outages where what used to be a local problem can cascade across a region, or even across the country.

The article looks at the recent power outage in Spain and Portugal that quickly grew from a local outage to a power outage across much of the Iberian Peninsula. Ookla points out that in today’s world, there is not that much difference between outages of a power grid, a cellular network, or a fiber network.

The article points out that outages can cascade much faster than anybody expects. The difference between a temporary disruption and a system-wide crisis depends on how quickly the network operators can recognize and analyze the causes of a problem. Ookla says there are five key steps needed to keep disruptions from escalating. Every major network outage is likely due to network operators failing at one of the early steps of this process.

  • Detection: Spot the first signs of trouble across multiple data sources, from outage reports to operator dashboards.
  • Attribution: Identify the root cause of the problem, whether it’s an internal software bug, a fiber cut, or a regional power failure.
  • Communication: Share timely, accurate information with stakeholders and the public to reduce confusion.
  • Remediation: Act quickly to contain damage, restore critical services, and prevent cascading failures.
  • Learning: Capture lessons from each event and feed them back into playbooks, exercises, and long-term resilience planning.

Ookla believes that the local reaction within the first hour can make a huge difference in the extent and length of an outage. There was one power company in Iberia that was able to isolate itself from the cascading shutdown because it was prepared to react quickly. I wonder how many local ISPs are ready to quickly react to problems caused outside their local network. The Ookla article suggests that local operators can do a lot more to protect themselves and their customers against major outages.

Portugal and Net Neutrality

Last week I talked about FCC Chairman Ajit Pai’s list of myths concerning net neutrality. One of the ‘myths’ he listed is: Internet service will be provided in bundles like cable television as has happened in Portugal.

This observation has been widely repeated on social media and has been used as a warning of what would happen to us Internet access without net neutrality. The social media postings have included a screen shot of the many options of ‘bundles’ available from the mobile carrier Meo in Portugal. Taken out of context this looks exactly like mobile data bundles.

Meo offers various packages of well-known web applications that customers can buy to opt the applications from monthly data caps. For example, there is a video bundle that includes Netflix, YouTube, Hulu, ESPN, Joost and TV.Com. There are a number of similar bundles like the social bundle that includes Facebook and Twitter, or the shopping bundle that contains Amazon and eBay.

But the reality is that these bundles are similar to the zero-rating done by cellular carriers in the US. The base product from Meo doesn’t block any use of cellular data. These ‘bundles’ are voluntary add-ons and allow a customer to exclude the various packaged content from monthly data caps. If a customer uses a lot of social media, for example, they can exclude this usage from monthly data caps by paying a monthly fee of approximately $5.

The last FCC headed by Tom Wheeler took a look at zero-rating practices here in the US. They ruled that the zero-ratings by AT&T and Verizon violated net neutrality because each carrier has bundled in their own content. But the FCC found that T-Mobile did not violate net neutrality when they included content from others in their zero-rating package. The current FCC has not followed through on those rulings and has taken no action against AT&T or Verizon.

The Meo bundles are similar to the T-Mobile zero-rating packages, with the difference being that the Meo bundles are voluntary while T-Mobile’s are built into the base product. The FCC is correct in pointing out that Portugal did not create mobile ‘bundles’ that are similar to packages of cable TV channels. If anything, I see these bundles as insurance – in effect, customers spend a small amount up front to avoid larger data overages later.

It is also worth noting that Portugal is a member of the European Union which has a strong set of net neutrality rules. But the EU is obviously struggling with zero-rating in the same way we are in the US. The real question this raises is if zero-rating is really a violation of net neutrality. It’s certainly something that customers like. As long as we have stingy monthly data caps then customers are going to like the idea of excusing their most popular apps from measurement against those caps. If cellular carriers offered an actual unlimited data then there would be no need for zero-rating.

I disagreed with the Wheeler FCC’s ruling on T-Mobile’s zero-rating. That ruling basically said that zero-rating is okay as long as the content is not owned by the cellular carrier. This ignores that fact that zero-rating of any kind has a long-term negative impact on competition. T-Mobile is like Meo in that they exclude the most popular web applications from data ca measurement. One of the major principles of net neutrality is to not favor any Internet traffic, and by definition, zero-rating favors the most popular apps over newer or less popular apps.

If enough customers participate in zero-rating the popular apps will maintain prominence over start-ups apps due to the fact that customers can view them for free. This is not the same thing as paid prioritization. That would occur if Netflix was to pay T-Mobile to exclude their app from data caps. That would clearly give Netflix an advantage over other video content. But voluntary zero-ratings by the cellular carriers has the exact same market impact as paid prioritization

None of this is going to matter, though, if the FCC kills Title II regulations. At that point not only will zero-rating be allowed in all forms, but ISPs will be able ask content payers for payment to prioritize their content. ISPs will be able to create Internet bundles that are exactly like cable bundles and that only allow access to certain content. And cellular carriers like AT&T or Comcast are going to be free to bundle in their own video content. It’s ironic that Chairman Pai used this as an example of an Internet myth, because killing net neutrality will make this ‘myth’ come true.