Tag Archives: Verizon

Extreme Market Segmentation

Linda Hardesty of Fierce Network recently quoted Verizon’s new CEO Dan Shulman as saying that the company is looking at market segmentation. He said that instead of pricing for a few market segments that Verizon might be considering hundreds of thousands of segments. ‘

Hardesty said this stunned her, because nobody has ever talked seriously about market segmentation to that extent before. What Verizon is thinking about doing is to develop multitudes of products and prices to fit small market niches, or even individuals.

This is not entirely a new idea. A few years ago, the public advocate from the California Public Service Commission conducted a detailed study and looked at broadband rates offered by Charter and other big ISPS across Los Angeles and a few other larger markets. They found that Charter prices varied by neighborhood, and surprisingly, the highest prices were in the poorest neighborhoods. Charter mostly did this through promotional and special rates, but there was a clear delineation of rates by the incomes of neighborhoods.

When Verizon talks about segmenting into hundreds of thousands of segments, they are talking about going far beyond what Charter was doing in California. Verizon is talking about profiling customers and making pricing offers specific to each customer.

You might wonder how they can do this. It’s well known that there are companies like Acxiom, Experian, and Epsilon that have created a detailed profile on families and individuals. These profiles are highly detailed and combine information about earnings and financial status, lifestyle, and buying habits. Verizon can combine this with what they know about every customer to be able to offer them the price or package they think will sell. This doesn’t just mean defining low rates, and ISPs might us the profiling to charge more to people they think will pay it.

It’s likely that any attempt to do this is going to be aided by AI. One of the axioms of the telecom industry over the years has been to keep pricing simple to make life easier for the customer service and billing process. But with AI tied to billing systems and customer service that may no longer be needed.

Shulman is oversimplifying when he says that Verizon will be expanding from four segments today, because the company surely has more than that. They company has offered varying special prices to land new customers and negotiates rates to keep existing customers, and probably has dozens of slightly different rates, if not more. But that’s a far cry from separating the market into hundreds of thousands of market segments.

There can only be two reasons why Verizon would do this. Hardesty suggests it’s to make more money by creating multiple tiers of customers, including a tier for those willing to pay premium rates for concierge service.

But I think there is an alternate possibility, which is to reduce churn. Cellular competition, in particular, has gotten fierce and the three big cellular carriers see about 1% of postpaid customers churn each month and 3-4% of prepaid customers. This high level of churn is what drives the gigantic advertising budget of each carrier, and perhaps tailoring custom plans to match the individual customer preference might increase loyalty, reduce churn, and reduce costs.

If Verizon implements this, I’m sure the rest of the industry will be watching closely. If extreme market segmentation is successful, then AT&T and T-Mobile will have to follow to keep even in the competition game. But if it’s unsuccessful, the other carriers will gleefully watch Verizon struggle. It’s going to be an interesting experiment to watch if the company really goes for it.

How Good is Rural Cellular Coverage – Part II

Yesterday’s blog looked at AT&T cellular coverage in a typical rural county in Illinois and included the following map. The map shows where AT&T can provide 5G coverage in a moving vehicle in the dark areas, and where somebody standing stationary outdoors could get a 5G signal in the lighter colored areas.

Let’s look at the maps for the other two major carriers in the same areas. The first map below is T-Mobile, and the second is Verizon.

These maps show typical coverage. The two carriers support 5G in moving vehicles in and close to towns and cities. The light colored areas are where somebody standing outdoors can likely get a 5G signal. An indoor cellular coverage map would likely not be a lot larger than the dark areas.

Taken altogether, these maps show a typical rural story of cellular coverage. Cell carriers rarely share towers, and each carrier is on different towers and has different coverage. All three carriers have areas where they have no 5G coverage, and somebody subscribed to any one carrier in this county would find a lot of dead zones. All three carriers have little or no coverage in the northwest sector. These maps show something that every rural delivery driver knows – to work in rural America means carrying multiple cellphones subscribed to different carriers.

When Chairman Carr says that 96.8% of households have 5G coverage, we have to put that into perspective. Over 80% of Americans live in cities and suburbs and likely have good cell coverage. Another substantial percentage live in smaller towns that happen to have at least one cell tower. In this particular county, 60% of people live in incorporated towns and villages, meaning there are a lot of rural residents.

What’s the point of these two blogs? The FCC considers this County to have good 5G coverage. That assumption comes largely from looking at the combined coverage of the three carriers shown for somebody standing stationary outdoors. The light colored areas of the three maps combined cover most of the county.

If the FCC ever decides to finally launch the 5G Fund for Rural America, this county will likely not be a candidate for a grant to build new cell towers. That’s unfortunate, because I estimate that 30% of the residents of this county would say they have poor cellular coverage. They will say that they don’t have good coverage indoors, and no matter which carrier they subscribe to, they hit dead spots when they drive around the county. The FCC’s assertion that 96.8% of homes have good 5G coverage can be supported by the FCC maps – but those maps don’t show the reality of the way that people judge cellular coverage.

Millimeter Wave Broadband

For those who follow everything about broadband speeds, Ookla published a recent article talking about the deployment of millimeter wave spectrum in U.S. cellular networks. You might remember the big burst of marketing in 2000 when Verizon commercials bragged about gigabit speeds on cellphones. These fast speeds were enabled by millimeter wave spectrum that had been deployed at the time in a handful of urban business districts. At the time, Verizon told investors that millimeter wave was going to be the future of cellular, and that cellular broadband was going to be able to compete head-on with cable and fiber networks. They had plans on the drawing board to deploy the technology deep in neighborhoods.

As a reminder, millimeter wave spectrum uses much higher frequencies than are normally used for cellular service. Before the Verizon marketing blitz, the company had purchased a lot of 28 GHz spectrum from Straight Path and XO Communications. That’s a significantly higher frequency than the mid-range spectrum (1 – 4 GHz) used for cellular service. It’s called millimeter wave spectrum because the radio waves for 28 GHz are extremely short. AT&T also dipped its toe into millimeter wave spectrum with the acquisition of 29 GHz spectrum.

The Ookla article points out that many of Verizon’s millimeter wave spectrum deployments are still in use, and the use of millimeter wave spectrum is growing. Ookla cites statistics compiled by its RootMetrics effort, where the company sends people to take random cellular speed tests in markets around the country. When those in-person tests are combined with the normal Ookla speed tests conducted by the public, the Ookla article shows that in the second half of 2025, that 2.2% of Verizon cellular speed tests used millimeter wave technology, while 0.2% of AT&T used the higher spectrum. T-Mobile had virtually no millimeter wave usage.

The report demonstrates the issues with using millimeter wave spectrum. The technology can deliver gigabit speed, but the effective distance from a transmitter is very short. RootMetrics found millimeter wave speed test connections mostly within 500 feet of a transmitter, even though the spectrum can theoretically carry for a half mile. That short distance limits the use of the spectrum to high traffic areas where the extra spectrum can help relieve pressure on the other cellular spectrum bands. In case you’re wondering, most high-end cellphones manufactured since about 2001 include the ability to receive the millimeter wave spectrum. Most of the rest of the world, other than South Korea, never activated millimeter wave spectrum in networks or cellphones.

Interestingly, this report also tells a similar story about C-Band spectrum (3.7 – 4.2 GHz). Most RootMetrics speed tests for connection using C-Band were found within a half mile from a tower, although the spectrum can theoretically carry for two miles. This is good proof that, while cellular speeds are improving, the fastest speeds are found relatively close to towers. The older spectrum bands used for cellular, like 700 MHz and 900 MHz, carry for many miles, but carry far less bandwidth.

The Ookla report goes into detail about the coverage found in a few markets. For example, the  report includes a map of millimeter wave just south of downtown Denver that shows small pockets of good coverage next to areas with poor coverage, again demonstrating the distance limitations on the technology. The report is well worth reading.

Customer Reactions to Outages

On January 14th of this year, Verizon had a major cellular network outage that lasted up to ten hours and that impacted more than 1.5 million wireless customers. Not all Verizon customers lost service, but the impact was felt across the country. Recon Analytics conducted a survey of 1,702 small, medium, and large Verizon business customers to understand their reaction to the outage. I can’t recall having ever seen a public survey of this type related to a single event. The results tell a sobering story for all ISPs and carriers.

The biggest impact was felt by the large businesses, and 44% of them said they were impacted by the outage. That makes sense they tend to operate in multiple locations, across multiple states, with a lot of employees that can be impacted by a day-long outage. 33% of medium-sized businesses were impacted, and 21% of small businesses. On the flip side, only 3% of large businesses were not aware of the outage, while 7% of midsize and 12% of small businesses hadn’t heard about the outage.

For all sizes of businesses, about two-thirds said the outage didn’t change their opinion of Verizon. That’s a scary statistic for anybody who sells to businesses, because it means one-third changed their opinion. About 5-6 % of all these customers said they were much more negative about Verizon, with 32-35% said they were somewhat more negative.

When asked if they were more likely after the outage to shop around for an alternative to Verizon, small businesses were the most forgiving, with 28% saying they were more likely to shop around, while 50% of midsize companies, and 59% of large businesses said they were likely to look for alternatives. Anybody who has worked is sales and marketing knows that there is a big difference between somebody who will consider changing service and somebody who will definitely change service.

But still, the survey results have to be worrisome for Verizon. The Verizon business sales team had been pushing the story for years that the reason to Verizon is because the service is reliable. An outage that lasted throughout a workday is the opposite of reliable.

While this survey concerned a long cellular outage, it’s something ISPs should also be aware of. Not only business customers, but also residential customers now believe that cellphone and broadband coverage should always be on, and never off. My friend Travis, who owned US Broadband, tells the story of how he had a ten minute outage in the middle of the night in Minneapolis, after years with no outages, and his negative Google reviews went through the roof.

It’s virtually impossible to never have a network outage, at least in some portion of a network. The Verizon outage of ten hours was particularly negative for the public, especially after the company eventually said the cause of the outage was some unspecified software issue. We don’t expect big companies to have software problems they can’t resolve quickly.

This outage raises the issue of what ISPs and carriers should do after an outage. First is probably having an internet definition of what constitutes an important outage – is it an outage that lasts ten minutes, an hour, or half a day? Should an ISP pretend short outages didn’t happen, or should they fully explain outages to every customer who was impacted?

The FCC’s Ability to Levy Fines

Today’s blog takes a deeper dive into the upcoming case at the Supreme Court concerning appeals by AT&T and Verizon over fines levied by the FCC. The original appeals followed an FCC finding that all three major U.S. cellular carriers were liable for violating customer privacy by selling access to customer location data. This data showed every place that a customer visited during the day, something that should make every cell customer uncomfortable. The FCC fined AT&T $50 million, T-Mobile  $80 million, and Verizon $47 million, with smaller fines against a few other carriers. The case at the Supreme Court looks specifically at the FCC’s ability to levy fines against the carriers for violating consumer privacy rights.

AT&T and Verizon appealed the FCC fines. Both companies were emboldened by two recent Supreme Court rulings that weakened the FCC’s authority. The first was Loper Bright Enterprises v Raimondo, which said that courts don’t have to defer to expertise at federal agencies when deciding lawsuits. The second case was SEC v. Jaresky, which said that federal agencies should give defendants a chance to have a trial by jury in a federal court rather than levying fines.

As usually happens with cases that make it to the Supreme Court, lower courts issued conflicting opinions about the FCC fines. The 5th U.S. Circuit Court of Appeals overturned the fines levied against AT&T, while the 2nd U.S. Circuit Court of Appeals upheld the FCC’s fines against Verizon.

One of the more interesting things about both appeals was that the carriers did not deny their bad actions – they had clearly allowed access to customer location data. Both appeals relied on the Supreme Court ruling in SEC v. Jaresky and argued that the FCC should have offered the carriers the chance to take the cases to court and hold a jury trial as an alternative to the FCC fines.

It’s clear that the carriers are trying to weaken or break the FCC’s ability to levy fines and are willing to go through this process as a way to avoid future fines. I find it unbelievable that the carriers would have chosen to take this specific case to a jury if they had been given that option. It’s impossible to seat a jury of people who don’t use cellphones, and I would wager that a jury would be unhappy that a carrier would sell the data to track them 24/7. It’s easy to imagine that a jury would assess damages much larger than the FCC fines if these cases had instead gone straight to court.

It will be interesting to see what the carriers do if they win this case. Winning doesn’t take them off the hook for selling customer data, and the cases would likely be remanded back to the FCC to give the carriers the option for a jury trial. As silly as it sounds, I’m dubious that, even with a second chance, the carriers would choose the jury trial. Again, the real goal of the carriers in this case is to weaken the FCC’s options in future disputes.

It will be unfortunate if the Supreme Court sides with the carriers and hobbles the FCC’s ability to levy fines. The FCC typically fines regulated companies for two reasons, for failing to comply with FCC rules or for abusing the general public. The first type of fines is generally relatively small and the big fines are saved for companies engaged in abuse and fraud of the public. In a recent action, the FCC fined Telnyx LLC for allowing foreign scam robocalls into the U.S. telephone network.

Protecting the public is one of the major roles of a regulatory agency. A policy shift that makes it difficult or impossible to hold large corporations accountable for abuses against the public would be a terrible outcome.

Some Thoughts on Convergence

An article in Light Reading reported that the largest cable companies captured about one-third of the net cellular customer additions in the fourth quarter of 2025. This statistic combines the cellular sales of Charter, Comcast, and Optimum. The overall cable industry statistics would be even higher if it included sales from Cox and Mediacom, which are privately held.

Industry analysts are using the word convergence as shorthand for competition that bundles cell service with broadband. Convergence is the newest strategy that replaces the traditional bundling strategy of selling a package of broadband, cable TV, and voice.

Industry press over the last year is full of articles that wonder about the ultimate success of the strategy. Cable companies seem to have the upper hand in a convergence bundle since they collectively pass roughly 122 million homes. I’ve read a few analysts who argue that the big telcos like AT&T and Verizon are at a disadvantage since they pass a lot fewer homes with fiber.

But I think these analysts are missing something. There are three players in the convergence battle, and each is using a different tactic:

  • Cable companies are finding success with the convergence bundle by combining full-price broadband with inexpensive cellular service. The main goal of the cable companies is to reduce broadband churn, and a customer loses their cable company cell service if they drop broadband.
  • The fiber parts of the telcos don’t seem to be pushing the convergence package to the same extent. They are mostly still betting that people like fiber a lot more than cable broadband. However, AT&T just announced a fiber/cellular bundle with gigabit and one cellphone for $90 and two cellphones $120.
  • The third competitors are the FWA cellular companies. They are bundling full-price cell service with inexpensive broadband. At least for now, they seem to be winning the convergence battle. In the fourth quarter of last year, AT&T, T-Mobile, and Verizon added over 1 million net FWA customers while the rest of the industry barely grew.

I know it seems odd to be counting the FWA competitors as different than the fiber telcos, since they are largely the same companies. But anybody who follows these companies understands there is not a lot of bleed-over between the wireline and the cellular parts of the businesses. The FWA division of the telcos are willing to compete for a fiber customer from their own sister companies.

It’s becoming clear that affordability is a major issue for a huge number of households. As long as that stays in the forefront, it seems like many households will lean towards the convergence plan that gives them a significant discount. I doubt that customers care if the discount comes from a lower price for broadband or cellular.

I think the cable companies are on to something with their focus on reducing churn. I talked to a few people in the last year who wanted to leave Charter and move to fiber broadband but didn’t want to lose their cheap cell service – and didn’t want to go through the hassle of replacing both services at the same time. The cable companies were really good at the triple play bundle in the 2010s, and a huge number of households felt they were held captive by the bundle. Are we headed back to that same place, but this time with multiple bundle options that force customers to buy both services from the same company?

Perhaps led by the recent new plan from AT&T, perhaps the fiber telcos are ready to jump into the convergence battle. I have wondered for years why they didn’t lead the market in this effort, and I guess it was due to internal battles over which division swallowed the bundling discount.

A Rural Cellular Story

I was looking through the FCC cellular map in Buncombe County, North Carolina, where I live. For those not fully familiar with the FCC broadband maps, the agency publishes two maps: the more familiar one that shows broadband coverage and a second that shows cellular coverage. You can toggle between the two maps at the FCC’s map website.

It struck me while looking at the details in the maps that rural cellular coverage is changing, and not in a good way. I started by looking at a small section of the county that is on the outer fringe of where the Asheville outer suburbs turn rural. According to the FCC cellular map, the area I selected has the following cellular coverage:

These two tables tell me the following:

  • AT&T and Verizon have some 4G coverage. But the Verizon coverage is likely very weak since they don’t claim it will work in a moving vehicle. While AT&T claims its 4G coverage will work in a moving vehicle, it’s curious that AT&T doesn’t have 5G. This tells me that the AT&T signal is also likely weak since it is outside the 5G coverage area.
  • The only carrier claiming relatively solid 5G (35/3 Mbps) is Project Genesis, which is EchoStar. The company has exited the facility-based cellular business and is in the process of dismantling cell sites.
  • T-Mobile claims both 4G and 5G for outdoor cellular coverage, but doesn’t claim it can work in a moving vehicle, meaning the coverage is also probably weak.
  • The last carrier listed is UScellular, which claims 7/1 speeds on 5G, but doesn’t claim to be able to provide coverage in vehicles. UScellular was purchased by T-Mobile, and the rumor is that any UScellular towers that already duplicate T-Mobile coverage are likely to be decommissioned.

The bottom line is that this particular neighborhood has weak cell coverage. The only carrier that claimed to be able to deliver 5G to a moving vehicle is now out of business.

I picked this neighborhood at random, but I think I would find the same story in most of the areas on the fringe of the metropolitan area. The coverage in areas that are completely rural is worse. The story I gleaned from this neighborhood is troublesome for several reasons.

  • The folks who live here don’t have a lot of options. The only carrier that might work in the way people need cellular to work is AT&T, but this neighborhood is outside the AT&T 5G coverage, and the 4G coverage is likely weak.
  • It looks like decent coverage was finally becoming available from EchoStar, but that’s now gone.
  • The speeds shown in the table are for outdoor coverage, and speeds inside homes are typically half of outdoor speeds.
  • When you look at the details in the FCC cellular map you quickly understand how the advertised national footprints of the big carriers are exaggerated.
  • The bad news is that the FCC considers this neighborhood to be served by cellular. That means if the FCC finally launches the 5G Fund for Rural America, this neighborhood will not be considered for funding to add a new cell tower.

Broadband Shorts March 2026

The following are a few topics I found interesting but which are two short to need a full blog.

Acquisitions Changing the Broadband Landscape. We’ve recently seen the closing of a number of major mergers and sales that are changing the broadband landscape.

  • On January 20, the sale of Frontier to Verizon closed. This $20 billion blockbuster sale brought 2.2 million fiber subscribers and eight million passings. Long-time followers of the industry are somewhat amused to see Verizon buy back millions of passings it sold to Frontier in the past.
  • On February 2, AT&T closed the sale of over 1 million fiber customers from Lumen, which brought four million fiber passings. This included customers in major markets like Denver, Seattle, Salt Lake City, Las Vegas, Minneapolis-St. Paul, Orlando, and Phoenix.
  • On March 10, the sale of Starry to Verizon closed. While bringing only 100,000 customers, the acquisition also brings Starry’s proprietary technology that uses 28/39 GHz millimeter wave spectrum to deliver wireless broadband, mostly to MDUs. The speculation is that Verizon will use the technology to expand to MDUs outside of its fiber footprint.
  • The huge merger between Charter and Cox Communications is still pending. The merger recently got approved by the FCC and still needs approval from several states. Cox would bring around 6 million broadband customers and 12 million passing to Charter, making the combined company the largest ISP in the country.
  • GFiber just announced a merger with Astound Broadband that would spin GFiber from Google Alphabet.

Action in the NDIA Suit. The U.S. Department of Justice sought to dismiss the lawsuit filed by the National Digital Inclusion Alliance (NDIA) that challenged the administration’s refusal to disperse the grant funding approved by Congress from the Digital Equity Act. These grants were aimed at tackling digital inclusion efforts that included bringing broadband devices to those that need them, training people how to use computers and broadband, and training for broadband-related jobs. The NDIA suit was first filed in early October 2025. I note the DOJ motion since the agency has had a low success rate in defending executive actions that killed various other federal grants. I think there is still a chance that this funding will eventually be awarded as intended.

AI Fueling Surge in Deepfake Spam. Hiya, a service that provides apps to block spam calls, released its State of the Call 2026 report, which says that AI is fueling an increase in spam calls. A survey of over 12,000 consumers across the  U.S., UK, Canada, France, Germany, and Spain showed a rise of deepfake calls, which use AI to mimic voices that are familiar to those being called. One in four Americans said they received a deepfake voice call in the last year. Americans said by nearly 2-to-1 that spammers are winning the battle over the FCC, which is trying to squelch spam calls.

AT&T Partnership with Amazon. AT&T, Amazon Web Services (AWS), and Amazon Leo announced a broadband collaboration this week that integrates AT&T into the AI and cloud capabilities of AWS. AT&T will become the preferred vendor to provide connectivity to AWS data centers. Amazon LEO has an existing arrangement with Verizon to bring fiber to ground stations, and it will be interesting over time to see if that business shifts to AT&T.

AT&T will partner with Amazon Leo to provide satellite broadband connectivity to some AT&T broadband customers. This is an interesting solution that could help AT&T more easily walk away from rural copper networks. AT&T also wants to bring satellite backup broadband to AT&T business customers.

Unintended Consequences

The industry news is always full of big events like mergers, bankruptcies, new regulations, or regulations killed. I’ve written many blogs about these kinds of issues, but I have rarely written about the unintended consequences of big industry changes. Today’s blog looks at two examples of unintended consequences.

The first is the decision  by EchoStar to abandon the facility-based cellular business. There were several factors that led to the company’s decision to abandon the business line, but the company says the primary reason was pressure from the FCC to use the spectrum it owned or return it to the FCC for auction. The FCC was also pressuring the company to build faster and to get more customers.

One of the unintended consequences of the FCC nudging EchoStar out of the cellular business is that the company decided in 2025 that its best option for maximizing value was to sell the spectrum it planned to use for cell towers. The company sold spectrum to Starlink that will support the company’s entry into the satellite cellular business. EchoStar also sold spectrum to AT&T, which was put to immediate use to boost bandwidth at 23,000 cell sites nationwide. Both of these consequences are positive for the industry and will benefit many millions of customers.

Another consequence of EchoStar abandoning cell towers is that the company walked away from a huge number of long-term leases for space on cell towers. A group of ten tower company executives met with the FCC recently and asked for help to recover the abandoned payments from EchoStar. The company says it had no choice but to walk away from the leases since it is no longer using towers, and they say this fits the “force majeure” clause in its contracts with tower owners that excuse payments in the case of an unforeseeable event. It’s going to be interesting to see if the FCC does anything, or even if they have any authority to intervene in a business contractual dispute. This same thing happens all of the time on a smaller scale when carriers and ISPs walk away from leases they no longer need, and the only real difference in the case is the magnitude of the issue. My bet is that the FCC will do nothing since this is now a commercial contract dispute, and it will probably tell tower owners to take their claims to court.

Another big piece of news is Verizon’s purchase of Frontier Communications. The purchase process started sixteen months before the deal finally closed, and much has changed in the industry since then. When the transaction was first announced, CEO Hans Vestberg touted the sale as moving Verizon forward in pursuing convergence. That’s the new industry phrase that replaces the old triple-play strategy and now refers to bundling broadband and cellphones.

Companies generally pursue mergers in an attempt to boost stock prices, and it will be interesting to see if that happens for Verizon. There are already industry analysts panning the merger, saying that it doesn’t really move the needle for Verizon. Pre-transaction, Verizon’s fiber covers 9.2% of the country, and Frontier brings another 4.3% coverage. This pales against the cable companies that lead in the convergence battle, with the biggest cable companies collectively passing 90% of households in the country.

There are also other consequences when companies merge. Verizon will claim it’s gaining efficiencies from the merger, but the real consequence is that a lot of folks at Frontier will lose jobs that would have been safe without the merger. Many of the vendors and suppliers that supported Frontier will suddenly find they have lost a giant customer. It’s likely that eventually the prices of the products at the two companies will be brought into synch, and since Verizon’s fiber prices are higher than Frontier’s, it probably means eventual price increases for Frontier customers.

A Three Nines World?

FierceNetwork recently published a thought-provoking article by Steve Saunders that asks, “is four nines the new five nines?”  That’s a question that only network engineers will understand, but it is a shorthand way to talk about the reliability of our networks.

The phrase five nines refers to having a goal for a network to be in service 99.999% of the time. That’s an incredible level of uptime, and a five nines network is expected to not be out of service more than five minutes in a year. A four nines network would have the goal of not being out of service more than 53 minutes per year, and three nines would lower the goal to 526 minutes, or just under nine hours per year.

I have a lot of clients who have signed contracts with large data customers to meet four or five nines of reliability. The only way to make that guarantee is to have a lot of redundancy. That would mean physically redundant fiber routes to protect against fiber cuts. It would mean self-healing electronics that quickly adapt to fiber outages or the loss of a key set of electronics. It means having software that can quickly be reset as needed.

In the last few years, we’ve seen network outages of major proportions. The latest outage by Verizon knocked a lot of customers out for half a day. There have been multiple regional and national outages due to problems in the Amazon AWS data centers. The breadth and magnitude of these regional outages is making it hard for any ISP to guarantee that networks will be reliable due to problems cause upstream by larger industry players.

As Saunders points out, the culprit of most of the big outages is software. The software that controls the Internet has grown increasingly complex. Sanders says the communications networks have grown as complex as the systems that operate a nuclear submarine.

The article points to the complexity associated with the recent big Verizon outage. The problem was something that affected the standalone 5G core network. Verizon’s core network includes electronics and software from five vendors  – Case Systems, Ericsson, Nokia, Oracle, and Red Hat / OpenShift. – along with Verizon’s own software.

Saunders says the issue is structural. While Verizon network engineers are elite, they are expected to operate networks that have grown to a level of complexity that is beyond the ability of technicians to fully understand everything. I’m sure Verizon still has an internal goal of five nines, but the company can no longer realistically understand the complexity of its network and the interplay of the many diverse components.

The problems and the outages are likely to grow worse as we continue to convert to software-defined networks, and as big companies consolidate network operations and eliminate technicians as a cost savings. We are also increasingly using AI to write complex software, which is reducing our ability to fully understand and debug problems during a crisis.

Saunders points to another issue, which is the erosion of the separation between LAN and WAN. For decades, businesses have been secure behind firewalls since they ran different software inside the company than what was used to communicate outside the company. But that distinction has become blurred as a lot of software now reaches across that barrier.

The article’s conclusion is that we are probably going to have to learn to live with big outages. The day of expecting to be connected to super-safe networks is gone. The Verizon outage shows that we might already be living in a three nines world, something that makes every network engineer cringe.