Cellular Backup for Broadband

Amazon announced a new device that offers broadband backup for broadband outages. The new device is the eero Signal, which is an add-on to Amazon’s eero WiFi system that will automatically connect a customer to cellular broadband any time the primary broadband connection fails.

For now, the device works with 4G and retails for $99.99. For the purchase price, a customer gets the device plus six months of backup service. After that, there is an annual $99.99 subscription. A 5G version of the products is coming soon that will have a $199.99 annual subscription fee. In both cases, the customer also has to have an eero subscription, which can cost as much as $9.99 per month or $99.99 for a year.

The Signal device will connect with the strongest carrier signal in your area, meaning that Amazon has made arrangements with AT&T, T-Mobile, and Verizon. There are usage limitations, and the cellular connection can’t be used as a primary broadband connection. I’m curious how long the device will function for somebody who has a multiday outage. One of the other caveats of the device is a warning that customers can’t connect to 911 through the device.

Amazon is clearly counting on folks willing to pay extra for instantaneous broadband backup. The service seems pricy to me. I have several broadband outages every month on my cable broadband connection, and most last for 15 minutes or less. If I’m working on my computer,  I can activate my cellular hotspot within a minute if I really need an Internet connection, but I often just wait for the outage to clear. But I’m not the target customer for the product since my quick fix is only for the computer I am using. I don’t operate any critical devices on WiFi, like a security system, fire alarms, or a health monitor. The biggest draw of the service for folks with those devices is that the eero Signal product will support WiFi for all of the devices in the home, including when the customer is away from home.

In December, AT&T also announced a new broadband backup service. This is a free product that offers cellular backup to customers who buy both AT&T broadband and cellular service. This backup also provides cellular broadband for the whole home, but only if the activated cellphone is within range of the home Internet gateway. The broadband gateway will instantly start using cellular broadband if the normal broadband connection is lost. One of the best features of this service is that it requires no new hardware or a subscription. But this wouldn’t provide backup for critical devices if the customer isn’t home or in range with the cellphone.

Both companies obviously think there is a need and a market for a backup service. AT&T is oddly acknowledging that its broadband has enough outages for this service to be an issue. I’m going to guess that Amazon isn’t going to be the only hardware company to offer this. The idea of creating a recurring revenue stream has to be attractive to folks who normally sell hardware.

Broadband Usage in 4Q 2025

OpenVault recently published its Broadband Insights Report for the end of the fourth quarter of 2025. One of the most useful statistics from OpenVault is the average monthly broadband usage per household in gigabytes. Below is the trend in average monthly U.S. download and upload volumes since the third quarter of 2021. These averages include broadband used by residential and small business customers.The average U.S. broadband customer used 59 more downloaded gigabytes and 10 more uploaded gigabits per month than a year earlier. This growth means continued pressure on broadband networks because if we assume roughly 120 million broadband subscribers nationwide, this growth means over 8.3 billion more gigabytes of data cross the Internet than a year earlier.

As can be seen in the table above, upload usage has been growing at a faster pace than download usage. In a recent quarterly report, OpenVault credited the growth of upload usage to the increasing usage of video calls, cloud backup, IoT uplinks, and similar uses. To put the 10-gigabyte increase in average upload into context, it’s the equivalent of every household uploading an additional 7 standard definition movie files or 3 high definition movie files every month compared to a year earlier. I think the average household would be surprised by the volume of data they are uploading each month.

The report made an interesting comparison between HFC technology and fiber for an unnamed cable company. When comparing customers that subscribed to equivalent fast download speeds, the average customer using cable technology uploaded 58 gigabytes, while the average fiber customer uploaded 93 gigabytes. Clearly, the slow upload speeds on cable (an average of 17.3 Mbps) are restricting uploading compared to fiber customers with symmetrical data plans.

Another interesting statistic is the percentage of U.S. subscribers at different speed tiers. For the last several years, there has been a steady migration of subscribers from slower speed tiers to the fastest tiers, and there is continued erosion in customers subscribing to speeds under 100 Mbps. What’s most interesting about the two years is a big jump in subscribers in the 200-499 Mbps tier and a decrease in subscribers buying speeds faster than 500 Mbps. My best guess is that this is reflecting the continued migration of millions of homes to FWA wireless. Those households seem to be willing to accept slower speeds as a trade-off for the lower prices.

One thing this table demonstrates is the absurdity of the FCC’s current definition of broadband at 100/20 Mbps. At the end of 2025, 87.4% of U.S. broadband subscribers are buying a product with speeds of 200 Mbps download or faster.

OpenVault always includes other interesting statistics in its quarterly reports:

  • The annual average increase in usage is growing over time. In 2022, households used 50 gigabytes more than the previous year. That’s increased to 54 GB in 2023, 57 GB in 2024, and 69 GB in 2025.
  • Super Power Users (those that use more than 2 terabytes of data per month grew by 22.5% in 2025, to become 7.4% of all households.

Median usage grew from 461.2 GB in 2024 to 531.8 GB in 2025. The median is the number where 50% of customers use less and 50% use more. That’s an increase of 15.3%, which is a faster growth rate than for average usage (9.9%). OpenVault credits the faster increase in median speeds to faster growth than average for customers with smaller monthly usage.

Unintended Consequences

The industry news is always full of big events like mergers, bankruptcies, new regulations, or regulations killed. I’ve written many blogs about these kinds of issues, but I have rarely written about the unintended consequences of big industry changes. Today’s blog looks at two examples of unintended consequences.

The first is the decision  by EchoStar to abandon the facility-based cellular business. There were several factors that led to the company’s decision to abandon the business line, but the company says the primary reason was pressure from the FCC to use the spectrum it owned or return it to the FCC for auction. The FCC was also pressuring the company to build faster and to get more customers.

One of the unintended consequences of the FCC nudging EchoStar out of the cellular business is that the company decided in 2025 that its best option for maximizing value was to sell the spectrum it planned to use for cell towers. The company sold spectrum to Starlink that will support the company’s entry into the satellite cellular business. EchoStar also sold spectrum to AT&T, which was put to immediate use to boost bandwidth at 23,000 cell sites nationwide. Both of these consequences are positive for the industry and will benefit many millions of customers.

Another consequence of EchoStar abandoning cell towers is that the company walked away from a huge number of long-term leases for space on cell towers. A group of ten tower company executives met with the FCC recently and asked for help to recover the abandoned payments from EchoStar. The company says it had no choice but to walk away from the leases since it is no longer using towers, and they say this fits the “force majeure” clause in its contracts with tower owners that excuse payments in the case of an unforeseeable event. It’s going to be interesting to see if the FCC does anything, or even if they have any authority to intervene in a business contractual dispute. This same thing happens all of the time on a smaller scale when carriers and ISPs walk away from leases they no longer need, and the only real difference in the case is the magnitude of the issue. My bet is that the FCC will do nothing since this is now a commercial contract dispute, and it will probably tell tower owners to take their claims to court.

Another big piece of news is Verizon’s purchase of Frontier Communications. The purchase process started sixteen months before the deal finally closed, and much has changed in the industry since then. When the transaction was first announced, CEO Hans Vestberg touted the sale as moving Verizon forward in pursuing convergence. That’s the new industry phrase that replaces the old triple-play strategy and now refers to bundling broadband and cellphones.

Companies generally pursue mergers in an attempt to boost stock prices, and it will be interesting to see if that happens for Verizon. There are already industry analysts panning the merger, saying that it doesn’t really move the needle for Verizon. Pre-transaction, Verizon’s fiber covers 9.2% of the country, and Frontier brings another 4.3% coverage. This pales against the cable companies that lead in the convergence battle, with the biggest cable companies collectively passing 90% of households in the country.

There are also other consequences when companies merge. Verizon will claim it’s gaining efficiencies from the merger, but the real consequence is that a lot of folks at Frontier will lose jobs that would have been safe without the merger. Many of the vendors and suppliers that supported Frontier will suddenly find they have lost a giant customer. It’s likely that eventually the prices of the products at the two companies will be brought into synch, and since Verizon’s fiber prices are higher than Frontier’s, it probably means eventual price increases for Frontier customers.

AI Bots and Web Traffic

One of the fastest-growing uses of the Internet is coming from AI scrapers. These are companies that crawl the web and scrape content to sell to AI companies to train models or sell to others looking for specific kinds of content. The scraping is done by bots that visit a website and copy the content. This is all machine-to-machine traffic with no human involvement. This traffic doesn’t reach to homes or businesses but instead impacts the Internet by increasing the traffic between data centers on backbone fiber routes and at Internet hubs.

According to a report from Tollbit, at the beginning of 2025, there was one visit by an AI bot to a website for every 200 human visits. By the end of the year, that ratio dropped to one AI bot visit for every 31 human visits.

Scraping websites for content is a controversial activity since it often involves grabbing copyrighted materials from news sites and other sources. A lot of companies have tried to make it hard to scrape the content from their site, but even the most sophisticated tech companies are finding the practice difficult to stop or block. The scraping bots are often successfully snagging content that is behind paywalls.

The practice is playing havoc with the traditional web compensation model of paying for click-throughs to websites. The click-throughs generated to websites are dropping as people use AI sites to answer questions instead of visiting the raw source of information. As an increasing percentage of searches are done through AI, the traditional compensation model will become obsolete – which is bad news for content providers. This also spells long-term trouble for search engines like Google that make money by directing users to websites.

Content providers claim that the AI bots are bypassing the robot.txt web protocol that provides a directory to web scrawlers to define the parts of a website that are open to scraping and areas that are copyrighted and off limits.

The AI bots are becoming increasingly able to mimic human users to avoid capthca and other tools used to frustrate bots. There is a technology battle being waged as content providers try new ways to frustrate bots and bots find new ways to bypass any blocking tools.

There are some well-known lawsuits from content providers to try to stop scrapers. In October 2025, Reddit sued Perplexity and related businesses. The Reddit suit claims that bots are illegally taking copyrighted materials. The Reddit suit also attacks the tactic of bots that mimic human users to grab content.

Google filed a suit in December 2025 against SerpApi for grabbing copyrighted materials, and also complains that the bot company uses illegal tactics such as armies of bots to overwhelm servers and fake and constantly changing user names to frustrate security measures.

There are a lot of consequences from this growing traffic for the companies that control the web. AI bot traffic is growing at a rate of over 25% annually while human visits to websites have started to decline. People don’t feel a need to visit websites when they can get questions answered by an AI search engine. The companies funding the basic Internet infrastructure are getting no compensation for the giant use of web resources.

There is no question that this is going to change the way that content providers get compensated. A lot of content providers sell access to articles behind paywalls, and that will fail as a revenue model if their content is made widely available through AI by scrapers.

The FCC 2024 Broadband Report

The FCC recently released its Internet Access Services report for December 31, 2024. The report is generated to provide a mandated update to Congress annually on the state of broadband. The data for the report mostly comes from broadband data that ISPs report to the FCC twice each year using the BDC reporting system, with some overlay with Census data.

I’ve always hoped this report would provide useful information, but it’s challenging to glean any truly valuable information from the report. There are a lot of reasons that combine to make most of the report unusable.

  • There is still a lot of inconsistency in the way the FCC broadband map defines serviceable locations. There are numerous examples where the FCC maps include locations that don’t exist while excluding valid locations. There is still no consensus on how to count vacant homes, vacation cabins, apartments built in basements and garages, etc. The FCC broadband map concentrates on ‘mass-market’ residential and business broadband locations. This leads to inconsistent counting of large businesses, while most anchor institutions are not included in the map.
  • ISPs report broadband coverage to the FCC, which is supposed to mean locations an ISP is connected to or that it can connect within 10 business days of a customer request for service. We’ve seen many cases where ISPs exaggerate claimed service areas.
  • The real issue with the claimed coverage is that ISPs also claim the maximum broadband speed available at each location. FCC rules allow ISPs to claim marketing speeds, which may be very different than actual speeds. For example, it’s very common for ISPs to claim 100/20 Mbps coverage but deliver something much slower for download or upload speeds. This means the FCC report is nothing more than a summary of the marketing speeds claimed by ISPs.
  • The FCC makes no attempt to layer on known changes to the data. For example, the FCC maintains maps of federal broadband grant awards that are supposed to be built in coming years. The report would be a lot more useful for measuring broadband improvements if there were tables summarizing these known changes.
  • Finally, I’m doubtful, in today’s dynamic market, of the usefulness of any data captured at a snap shot in time. The Fiber Broadband Association claims there were 11.1 million fiber passings constructed during 2025 that are not reflected in the report. There have also been a lot of technology upgrades from cable companies, WISPs, and FWA providers. These changes all mean that the broadband landscape is significantly different just one year after the date of this report.

The report is full of charts and tables that sound like they should be useful, until you look at each of them in light of the above issues. For example, every table that is based on broadband speed is highly questionable due to ISPs that self-report marketing speeds.

But even many of the tables and graphs that don’t refer to speeds are puzzling, or report nothing useful.

  • Figure 4 purports to show broadband customers at the end of 2024 by technology. It shows 57.3% on cable, 26.7% on fiber, 6.6% on DSL, 7.3 % on fixed wireless, and 2% on satellite. However, even this simple table is troublesome. The three big FWA cellular providers claimed 11.6 million customers at the end of 2024, and it looks like these customers are not included in any of the categories.
  • Figure 9 shows the locations in each state with various speeds. Are there really 9% of locations in West Virginia and 5% of locations in Mississippi that can’t get a broadband speed of at least 0.2 Mbps? There are a lot of tables that analyze speeds over and under 200 kbps (this really is kilobits, which is four times faster than dial-up), which must be an obsolete Congressional reporting category.
  • Figure 17 shows all fixed connections with speeds over 200 kbps. The table combines cellphones and broadband in the same table, which demonstrates that only 14% of broadband connections are from cable broadband.
  • Figures 20 through 31 show the trend over time of the number of connections at various speeds. Because of the use of marketing speeds, the quantities are likely far off, but the trends are interesting. These figures clearly ignore FWA cellular broadband.
  • There are a lot of charts and tables about cellular speeds, which are completely worthless since most cell companies report 5G speeds to the FCC maps of either 7/1 Mbps or 35/3 Mbps, in a world where actual speeds can be hundreds of Mbps.
  • Figure 41 is interesting and shows the number of ISPs that report the use of various technologies over time. The trends are interesting but have no context. For example, how much of the drop of the number of cable ISPs is due to companies that folded versus those that were absorbed into a larger cable provider?

Revamped Opportunity Zone Financing

Opportunity Zones were created by the 2017 Tax Cuts and Jobs Act and provided an opportunity for equity investors to defer and reduce capital gains taxes when they reinvested profits from an existing project into qualified projects located in targeted and defined low-income Census Tracts. The Opportunity rules got a lot of play in some parts of the country for projects like urban renewal and renewable energy projects, but for various reasons, this never caught on for very many broadband projects.

Some of the shortfalls of the original rules have been modified by changes included in the One Big Beautiful Bill Act (OBBBA), These new rules are generically being referred to as OZ 2.0. The new rules will be effective on January 1, 2027, and States and Treasury are working to have the new rules and revised Opportunity Zones in place by then. These rules are a lot friendlier to investors and could be used as part of the funding stack for projects that include building fiber.

Following are the key changes in the OZ 2.0 rules:

  • Permanent Extension. The new rules are now permanent. The old rules expired at the end of 2026, meaning that in recent years, there was a diminished value in trying to take advantage of the benefits.
  • Rolling 5-Year Deferral. Investors can now defer capital gains for five full years from the date of an investment.
  • 10% Basis Step-Up. An investor who holds a Qualified Opportunity Fund (QOF) investment for 5 years receives a 10% step-up in basis, which means an increase in deferred gains.
  • 10-Year Gain Exclusion. If a QOF investment is held for at least 10 years, the investor gains a permanent exclusion from federal taxes for any new gains.
  • Extra Rural Benefit. Investments made through New Qualified Rural Opportunity Funds (QROFs) for projects in rural areas will earn a 30% basis step-up (instead of 10%) for a 5-year hold. Rural investment properties also face a lower substantial improvement threshold, reduced from 100% to 50% of the acquisition basis.
  • 30-Year Rule: Investments held for 30 years or longer are eligible for a basis step-up to fair market value without needing to dispose of the asset.

The new rules require redrawing the Opportunity Zone Map of eligible areas. Governors play a key role in the process, and communities that want to be included as Opportunity Zones should be lobbying their Governor to be included. An area can only be designated as an Opportunity Zone if median household incomes are less than 70% of the area-wide incomes (the old rules was 80%).

The new rules also come with increased reporting. Anybody using Opportunity Zone funding will have to file an annual report that describes the economic impact of the project as well as employment and investment data. There are defined penalties for non-compliance.

There is a lot of nonpartisan support for the new rules since they are aimed at attracting investments in both urban and rural low-income areas.

The original Opportunity Zone rules were commonly used as a way to get local investors to invest in local projects. Most large equity investors, like high-value individuals or family offices, who routinely experience capital gains from investments, and the Opportunity Zone rules allowed those gains to be deferred by investing in projects with local impact. It’s hopeful that the new rules will make it even easier to attract some equity investments into projects in low-income areas.

Starlink Adds Tiered Pricing

Starlink is in the industry news so often that it feels like I ought to have a recurring blog just for Starlink updates. The company recently made an interesting announcement that should make waves in the broadband industry. The company announced tiered pricing, with prices varying by speed, and starting at $50 per month in some selected markets.

This is something Starlink has always been able to do, and perhaps they are doing this because the company is profitable and has landed 2 million U.S. customers with its standard $120 price. It’s clear that the company’s network is gaining capacity at a steady pace as the company launches additional and better satellites. At the beginning of February, the company had over 9,300 working satellites in orbit out of 9,600 total satellites. The company is on track to reach its first constellation goal of having 12,000 satellites in orbit by the end of this year. The company will soon be introducing its next generation of satellites which will have significantly greater broadband capacity.

The new products announced by Starlink include:

  • Residential 100 Mbps. $50 per month. Speeds up to 100 Mbps.
  • Residential 200 Mbps. $80 per month. Speeds up to 200 Mbps.
  • Residential Max. $120 per month. New customers also get a mini kit that can be used for traveling or camping.

The 100 Mbps product comes with a warning that it’s not available everywhere, so we’ll have to wait for feedback from the public to see what that means. Having a $50 broadband product everywhere would be a big deal.

It will be interesting to see the actual speeds delivered for each product. Each of the three products is being advertised as ‘up to’ speeds. Starlink recently issued a report to customers that said that its average download speed at the end of 2025 was 170 Mbps, so it will be interesting to see the speeds that are delivered to subscribers of the 200 Mbps tier. Ookla issued a report last year that said that Starlink’s median speed was 104.7 Mbps at the end of the first quarter of 2025, meaning half of the speeds were faster, and half were slower than that number. There is no mention by Starlink in its announcement about upload speeds. The Ookla report showed that the median upload speed was 14.8 Mbps at the end of the first quarter of 2025.

I work in a lot of rural areas where many people either can’t or won’t pay $120 per month for broadband. Starlink could get huge market traction with a $50 product. A $50 price for 100 Mbps broadband would put huge pressure on WISPs, which generally have rates higher than that. A $50 product would also be an affordable alternative to the many rural fiber networks being built by grants.

A $50 rate also would play well in cities. That rate competes well against the low-cost options in cities from DSL or FWA wireless. Most big telcos have raised DSL rates to the range of $65 per month. The FWA carriers have list rates of around $60, but customers can get it for less by bundling with a cellphone plan and agreeing to autopay. I can’t see Starlink cleaning up in cities with a $50 rate, but it would get a lot more customers than it has today in those markets.

In other pricing news, Starlink increased the monthly data cap on its Roam product to 100 gigabytes per month. This product is also $50 per month and is aimed at RVs and hikers. The average U.S. home uses around 700 gigabytes, but this will be attractive to homes that aren’t big broadband users. Starlink significantly chokes speeds after the monthly cap is reached.

Financial Limitations on Growth

I’ve worked with rural counties across the country, and one of most common recurring themes is a hope from elected officials that local ISPs will step up and build fiber and solve the broadband gaps in the area. County officials often beg local ISPs to pursue grant funding. Far too many times, local ISPs are unable to expand to provide what they are being asked to do.

Only rarely have I ever heard a small ISP admit to local leaders the real reason it can’t solve the broadband gaps. The primary reason that ISPs can’t expand at will is the natural limits on financing for growth. I can’t recall ever hearing an ISP tell a County government that they are unable to borrow the needed funding. I think this is because they think it will diminish them in the eyes of the local officials and make them sound like a small business.

The market reality is that every ISP has a ceiling on the amount of money it can borrow. This limit is not set by the ISP – it’s set by lenders. I’ve often been asked by ISPs to quantify their borrowing limit, but it’s never an easy question to answer. When banks make loans, they look at obvious things like the ability of cash flow to cover debt payments, and at the ratio of debt and equity at a given business. There are also a lot more subtle issues involved in the decision for a bank to make loan, like the bank’s own overall equity and cash holdings, the national trend of interest rates, and the lender’s mix of different types of loans. Just like an ISP doesn’t want to publicly disclose its ability to borrow, banks rarely tell an ISP the real reasons they don’t qualify for a loan.

County officials often assume that it’s easier to find the needed funding if part of a project is funded by a grant. If anything, grant funding can make it harder to borrow, because the borrower has to satisfy the future requirements of the grant agency as well as the requirements of the bank.

This problem is not unique to ISPs, and every kind of business has a natural borrowing limit, just like every family has a limit on the amount of money they can borrow to buy a new home. We tend to think that broadband expansion projects are different because they are loans to build long-term infrastructure, but building a broadband network is not a lot different than building a new factory or a new retail location.

One unique category of ISP is startups. These are new ISPs, or ones that have only a few customers. With rare exceptions, new ISPs are unable to get loans until they have somehow self-funded enough to reach a size and a balance sheet that a bank will even consider.

Having a natural borrowing limit is not unique to small ISPs. Altice, which recently rebranded to Optimum, is in financial hot water because it looks like it is facing a huge challenge to refinance its outstanding debt that is coming due in a large balloon payment. One of the factors that convinced EchoStar to abandon the facility-based cellular business was the challenge of borrowing more to meet its existing debt payments.

The only realistic alternative to borrowing for most ISPs is to accept equity funding. Small ISP owners are generally reluctant to accept equity because it means giving up control of the business. One of the biggest benefits of being a small ISP is that the owners can carve their own path and do things their own way. That freedom quickly ceases when an ISP accepts any significant equity funding.

I guess the bottom line of this discussion is that communities have to have a realistic expectation for small ISPs in the area. These companies would probably love to expand and serve everybody, and if they are not doing so, there is a good chance that they can’t raise the funding needed to expand. Very few ISPs are going to say this in public, but don’t read the fact that they won’t expand to mean they wouldn’t love to.

FCC Robocall Filing Due for All Voice Providers

The FCC adopted new rules for voice providers that are part of the FCC’s effort to curtail robocalls. The new rules apply to every voice provider that sells a retail voice product to end-user customers, along with other categories of providers like voice wholesalers and international providers. The new rules were effective on February 5.  The key new rules include the following:

  • Every voice service provider (VSP) has to register in the Robocall Mitigation Database (RMD) by March 1. Any VSP that is already in that database must update its information by that date.
  • The FCC issued a stern warning to any VSP that doesn’t register as required. There are fines of as much as $10,000 for not registering or for submitting false or inaccurate information to the RMD. The FCC also established a $1,000 fine for any late registrations or updates.
  • VSPs will also now need to renew the registration annually.
  • Finally, VSPs must update the FCC within 10 business days any time they make a change that affects their information in the FCC’s CORES database.

The mechanics of the new rules are that every VSP must have a unique token that gets associated with the call records that identifies the VSP. In past filings, a lot of VSPs simply reported that they rely on the tokens from another upstream provider. The new rules make it clear that every company that sells voice needs to have its own token.

These new rules are going to catch a lot of ISPs and CLECs off guard. It’s likely that most companies that still operate a voice switch knew about these rules in the past – and if they didn’t, the FCC plans to hunt down non-compliant carriers through the databases.

But the big change is that this applies to ISPs that buy wholesale VoIP and repackage it to their customers. These VSPs must comply with the new rules.

The FCC plans to implement an annual $100 filing fee to accompany the annual certification. However, that hasn’t been implemented yet, and VSPs can file for free right now.

The CCG team can help with any aspect of these filings, so contact me at blackbean2@ccgcomm.com if you need help.

A Three Nines World?

FierceNetwork recently published a thought-provoking article by Steve Saunders that asks, “is four nines the new five nines?”  That’s a question that only network engineers will understand, but it is a shorthand way to talk about the reliability of our networks.

The phrase five nines refers to having a goal for a network to be in service 99.999% of the time. That’s an incredible level of uptime, and a five nines network is expected to not be out of service more than five minutes in a year. A four nines network would have the goal of not being out of service more than 53 minutes per year, and three nines would lower the goal to 526 minutes, or just under nine hours per year.

I have a lot of clients who have signed contracts with large data customers to meet four or five nines of reliability. The only way to make that guarantee is to have a lot of redundancy. That would mean physically redundant fiber routes to protect against fiber cuts. It would mean self-healing electronics that quickly adapt to fiber outages or the loss of a key set of electronics. It means having software that can quickly be reset as needed.

In the last few years, we’ve seen network outages of major proportions. The latest outage by Verizon knocked a lot of customers out for half a day. There have been multiple regional and national outages due to problems in the Amazon AWS data centers. The breadth and magnitude of these regional outages is making it hard for any ISP to guarantee that networks will be reliable due to problems cause upstream by larger industry players.

As Saunders points out, the culprit of most of the big outages is software. The software that controls the Internet has grown increasingly complex. Sanders says the communications networks have grown as complex as the systems that operate a nuclear submarine.

The article points to the complexity associated with the recent big Verizon outage. The problem was something that affected the standalone 5G core network. Verizon’s core network includes electronics and software from five vendors  – Case Systems, Ericsson, Nokia, Oracle, and Red Hat / OpenShift. – along with Verizon’s own software.

Saunders says the issue is structural. While Verizon network engineers are elite, they are expected to operate networks that have grown to a level of complexity that is beyond the ability of technicians to fully understand everything. I’m sure Verizon still has an internal goal of five nines, but the company can no longer realistically understand the complexity of its network and the interplay of the many diverse components.

The problems and the outages are likely to grow worse as we continue to convert to software-defined networks, and as big companies consolidate network operations and eliminate technicians as a cost savings. We are also increasingly using AI to write complex software, which is reducing our ability to fully understand and debug problems during a crisis.

Saunders points to another issue, which is the erosion of the separation between LAN and WAN. For decades, businesses have been secure behind firewalls since they ran different software inside the company than what was used to communicate outside the company. But that distinction has become blurred as a lot of software now reaches across that barrier.

The article’s conclusion is that we are probably going to have to learn to live with big outages. The day of expecting to be connected to super-safe networks is gone. The Verizon outage shows that we might already be living in a three nines world, something that makes every network engineer cringe.