Tag Archives: CenturyLink

Another Lumen Reinvention?

Lumen has been the hardest large big telco to figure out. Verizon, AT&T, Frontier, Windstream, and others have clearly decided that building fiber is the future path to survival. Consequently, the other telcos are far ahead of Lucent in terms of fiber passings. In the recent investor webcast, CEO Kate Johnson talked about Lumen’s upcoming fiber plans. In doing so, she mentioned that Lumen only covers 12% of its passings with fiber – far behind the other telcos.

CenturyLink was one of the first big telcos after Verizon to embrace fiber. In 2017, under CEO Glen Post, the company had plans to pass 900,000 homes and businesses with fiber, with similar plans in upcoming years. Post, with a long telco background, had a clear vision of CenturyLink becoming a fiber-based ISP, at least in the many large cities it served.

However, at the end of 2017, the company took a sharp turn when it acquired Level 3. To nobody’s surprise, Jeff Storey from Level 3 took over as CEO, and the company changed its focus from residential fiber expansion to a focus on large business customers and small cell sites – the bread and butter of Level 3. By 2019, new fiber construction had dropped to 300,000 passings, with many of those coming from connecting large buildings to the network.

In 2020, the company’s stated focus was on adding large buildings to the network, and the company added 18,000 buildings to its fiber network, while only passing 400,000 total new fiber passings. That’s a low number of new fiber passings for a company that had 4.5 million broadband customers that year. 2020 was also the year when the company rebranded to become Lumen, a move to  distance itself from identification as a copper telco.

Instead of expanding fiber, Lumen decided to ditch copper assets and announced the sale of its last-mile networks in twenty states to Apollo Global Management in 2021. This brought a cash infusion needed for expansion and got rid of deteriorating copper networks.

Last year the company announced that its major expansion thrust was to beef up its large intercity fiber network across the country, with the goal of adding over 6 million miles of fiber strand by 2026. The original CenturyLink fiber network was starting to show some age, with many routes built forty years earlier. The company planned to upgrade to the newest fiber from Corning that can support 400-gigabit electronics. The new long-haul networks have fiber bundles between 432 and 864 fiber strands – much larger than the historical networks that had 96 to 144 fibers.

Lumen has been penalized by the many changes in its future direction by seeing the stock price go into the tank. CenturyLink stock peaked at almost $49 in 2007. By 2017, the stock had slipped to the mid-$20 range. Since then, the stock has dropped steadily and recently hit $1.80 per share after sitting at $10 per share a year earlier.

CEO Kate Johnson admitted that the company needs to do something different. The company eliminated its dividends to shareholders in the fourth quarter of 2022. The company is instead going to reinvest that money into building new fiber passings and has plans this year to connect 500,000 homes and businesses in 2023. It plans to build deeply into six major metro areas this year.

The company needs to reinvent itself. Lumen lost 253,000 broadband customers in 2022 – 7.7% of its broadband base. The company lost another 56,000 broadband customers in the first quarter this year, dropping the company to fewer than 3 million broadband customers while falling to be the eighth largest ISP after being surpassed by T-Mobile.

It’s not hard to understand, in retrospect, why the company has lost value. The company has seemingly reinvented itself every year since 2017 by changing its primary focus each year. Some of the changes, like adding more business buildings and beefing up the long-haul fiber network will likely generate a lot of cash and value in the long run. But Wall Street has clearly told the company to pick a future and stick to it.

The Public Loves Fiber

The latest Customer Satisfaction Index is out from ACSI, which measures the public satisfaction of a wide range of U.S. industries and institutions. The survey this year continued to show that the public has a poor opinion of ISPs. As a group, ISPs had an average ACSI annual rating of 68. The only industry with a lower rating is gas stations at 65. Subscription TV had an average rating of 69, and the U.S. Post Office had a rating of 70.

But there is some interesting good news for some ISPs. Companies serving customers with fiber rated higher with the public than other ISPs, including cable companies using coaxial networks. Consider the following table that shows the 2023 ranking for fiber and non-fiber ISPs.

Fiber Non-Fiber
Altice 58
AT&T 80 72
Cable One 71
CenturyLink 78 62
Charter 64
Comcast 73 68
Cox 64
Frontier 74 61
Google Fiber 76
Mediacom 65
T-Mobile 73
Verizon 75
Windstream 70

For companies that offer both fiber and another technology, customers served by fiber liked an ISP more than non-fiber customers. CenturyLink has the biggest difference in satisfaction (78 for fiber and 62 for non-fiber). Frontier also has a dramatic difference (74 fiber and 61 non-fiber). The only cable company ranked for both technologies also has a sizeable difference, and Comcast has a ranking of 73 for its fiber network versus 68 for the coaxial network.

Customer satisfaction involves many other factors than just technology, but the differences for the companies that offer multiple technologies have to be mostly related to fiber. However, there are other factors in play. For example, it seems likely that CenturyLink and Frontier provide better customer service and faster repairs for fiber customers than for DSL customers.

Cable companies have to be noticing this giant difference as part of any consideration of how to upgrade their networks. The big cable companies are all at the beginning of the upgrades to improve upload speeds on coaxial networks, and they must be hoping that customers like them more after the upgrades. But there is a chance that the public has come to think of fiber as a superior technology and will not rank a coaxial system as highly even after speed increases. There is still a noticeable difference in latency and jitter between cable and fiber networks, and customers who see both in action believe fiber is better.

There is still a noticeable range of ISP rankings within each list. Non-fiber customers rate T-Mobile and AT&T the highest and rank Altice and Frontier DSL as the worst ISPs. It’s interesting to see Charter near the bottom of the rankings.

Fiber customers clearly rate AT&T as the best and Comcast Fiber as the lowest. Fiber technical performance should be consistent regardless of the ISP, so the difference in rankings between fiber providers has to be related to customer service and the other non-technical aspects of being an ISP.

Should We Trust the Companies that Created the Digital Divide?

For those of you who don’t know Bruce Kushnick, he’s been tracking the promises made and broken by Verizon since the 1990s and written extensively on the issue. His latest article is “NTIA: Require Every State Broadband Agency to Investigate Those Responsible for Creating the State’s Digital Divide.”

Bruce has been arguing eloquently for years that the big telcos like Verizon, AT&T, and CenturyLink caused the rural digital divide by extracting profits from the regulated telephone and broadband businesses in rural and low-income areas while neglecting maintenance and not using any of the profits to modernize the technology. According to Bruce, the only reason we need massive federal grant programs today is to make the investments that the big telcos refused to make for the last several decades.

He argues that the NTIA should require states to investigate how the digital divide was created in rural areas and center cities. He uses the two examples of New Jersey and Los Angeles to make his point. He’s been tracking the promises made by Verizon to the State of New Jersey for the last thirty years. Verizon repeatedly sought regulatory relief through deregulation along with rate increases that were supposed to fund modernizing the network in the State – upgrades that were never done. When Verizon finally upgraded to fiber, it did so only in neighborhoods with the lowest costs, avoiding rural areas and most low-income neighborhoods.

I’ve been tracking this issue during my career as well. Consider West Virginia. I remember when Verizon was looking for a buyer of the telco network there as far back as the early 1990s. When big companies are trying to sell a property, they do what valuation folks call ‘dressing up the pig”. This means cutting expenses to make the property look more profitable. The cuts are usually deep, and drop maintenance below the level needed to keep up with routine repairs and maintenance.

Verizon didn’t end up selling the West Virginia network until the sale to Frontier in 2010. By then, the networks had been neglected for more than fifteen years. Frontier made only minimal upgrades to the properties they purchased – but it’s hard for an outsider to know if this was due to an intention to continue to milk cash flow out of the acquired network like Verizon had done or due to a lack of the capital and impact of the heavy debt used to buy the property. In any case, the West Virginia network continued to degrade under Frontier’s ownership.

For years, Bruce has made the point that there has not been any financial or regulatory cost to the big telcos for their bad behavior. They’ve repeatedly broken promises made to states. They’ve routinely milked profits out of networks while ignoring customers as the properties deteriorate.

In fact, we’ve seen the opposite of penalties. For example, the big telcos were rewarded with over $10 billion of CAF-II subsidies to support dying and neglected rural DSL networks. That money was supposed to be used to increase rural data speeds to 10/1 Mbps at a time when that speed was already obsolete. We’ve seen far too many places where even that basic upgrade was not made.

Bruce’s conclusion is that it would be ludicrous to give grant funding now to the companies that caused the digital divide in the first place. That would be using public money to upgrade the networks for these companies when profits should have been used over the decades to do so. He makes a solid argument that giving money to these same companies will not solve the digital divide since there is no reason to think the big telcos won’t turn around and do it all over again.

A Last Gasp at Regulating Copper

The Minnesota Public Utilities Commission recently ordered a series of public hearings to investigate the quality of service on the CenturyLink copper networks. The hearings were prompted by a complaint filed by the Communications Workers of America (CWA). The complaint listed the failures of CenturyLink to meet state service standards due to the deterioration of the copper network. CWA also noted that CenturyLink is planning to eliminate half of the remaining technicians who work on copper.

Similar inquiries by other state regulators have been instituted in the last few years against CenturyLink and Frontier. I feel sorry for any customers left on deteriorating copper networks, but proceedings like this one feel like the last gasp of regulators trying to score points by beating up on the telcos that still operate copper networks.

Not that CenturyLink doesn’t deserve a lot of criticism. Its copper networks are in dreadful condition and are in the process of dying. The poor condition of the networks is due in large part to the decades-long lack of maintenance and repairs. We know this is the case because copper networks of a similar age are still operating much better in Europe. The big telcos like CenturyLink, Frontier, Verizon, and AT&T stopped caring about copper networks back in the 1990s, and the networks have been in a steady decline since then.

But U.S. copper networks are truly near the end of life. It’s impossible to neglect maintenance for over twenty years and somehow suddenly make the networks perform better. It’s hard to fathom the intentions of having regional hearings on the topic for any purpose other than letting people vent their frustration with CenturyLink. It’s hard to imagine anything changing as a result of these hearings that will improve service. There might be new fines levied on CenturyLink, but that’s less costly for the company than trying to make the copper work.

Some big telcos are working to convert copper networks to fiber. Frontier and Windstream are building a lot of fiber – and I assume they are overlashing the new fiber wires on the old copper. AT&T and Verizon are selectively expanding fiber in neighborhoods where the cost of construction meets some internally set cost test – but these two companies are quietly moving most copper customers onto cellular connections.

CenturyLink has been up and down on the decision to overbuild residential fiber. It currently looks like the company is only building ‘strategic’ fiber, which I interpret to mean business districts and large apartment complexes. It seems unlikely that CenturyLink will overbuild much more of its residential copper in Minnesota or elsewhere with fiber.

I would bet that if CenturyLink could wave a magic wand and be rid of copper, it would do so. It’s harder each year to maintain copper networks, and a move to eliminate half of the remaining copper technicians shows that the company is finally throwing in the towel. But giving up on copper still means walking away from a lot of revenue.

There are still plenty of customers who want to keep using the copper networks. Say what you want about the inadequacies of DSL, but in most urban markets where my firm does surveys, we still find 10% to 20% of households are still using DSL. These are households for whom the price is more important than broadband speed.

CenturyLink and the other big telcos have recaptured the cost of the copper networks many times over and decided many years ago not to reinvest profits back into new and upgraded networks. We’re now reduced to watching the last death throes of copper networks, and it’s not pretty.

Lumen’s Fiber Path Forward

Lumen is taking a different path forward than the other big telcos. AT&T continues to build fiber in selected clusters, mostly in cities, rather than concentrate on building entire markets. Frontier, Windstream, and Consolidated are all concentrating on upgrading existing telco DSL networks to fiber.

Lumen has a different path forward. In a recent press release, the company announced a major upgrade to its long-haul fiber routes that cross the country. The company’s main fiber strategy is to beef up the intercity network with plans to add six million miles of fiber to existing fiber routes by 2026. In case you are wondering how there can possibly be six million route miles of fiber in the country – that count is miles of individual fibers. This is a marketing trick that long-haul fiber providers have been using for years to make networks seem gigantic.

The existing Lumen long-haul fiber network came to the company in two acquisitions. The original network came when CenturyLink bought US West, which had earlier merged with Qwest, a major builder of long-haul networks. The network was strengthened when CenturyLink purchased Level 3 Communications.

The original Quest fiber is getting dated in terms of capacity and performance. Much of this fiber was built thirty and forty years ago. While most of the fiber is still functional, fiber glass technology has improved drastically since then. Lumen will be using two low-loss types of fiber from Corning. This newer fiber is far clearer than older fiber and will increase the distance between repeater points while also allowing for using the fastest 400-gigabit electronics today and faster electronics later.

Earlier this year, Lumen announced it is improving its Ethernet architecture in forty cities this year. This means upgrading local networks to major customers to be able to provide speeds up to 30 gigabits. While this upgrade will mostly benefit business customers, this also will improve the local fiber backbone in these cities to 100 gigabits, which should improve performance for all broadband customers.

Lumen is also pursuing a last-mile fiber expansion. In August, the company announced fiber expansion plans in Denver, Minneapolis, and Seattle. The company had a target for this year to pass one million locations with fiber but has fallen a little behind due to supply chain and logistics.

Unlike the other telcos, Lumen hasn’t been talking much about the upcoming rural grant funding. This doesn’t mean the company might not pursue those opportunities since rural fiber expansion creates monopolies. But major residential expansion does not seem to be a key part of the Lumen plan, at least compared to plans for companies like Frontier, which says it plans to pass 12 million homes with fiber.

Another big unknown is if the company is still trying to sell any of its remaining copper networks like it did with sale of the twenty easternmost states to Apollo Global Management. It would be a more drastic affair to liquidate last-mile customers in the states where US West was formally the Bell company incumbent provider.

Any more sales of last-mile networks would be an interesting step where Lumen would be retracting to be a large business ISP. The company already had a sizable share of the business market that got bolstered by the acquisition of Level 3.

Lumen shares one characteristic with all of the big telcos in that it knows it must reinvent itself. After many years of no activity, Verizon is expanding FiOS again while also pushing a nationwide FWA network. AT&T is fully committed to building last-mile fiber networks and continues to add millions of new passings per year. The smaller telcos like Frontier and Windstream have clearly decided they must build fiber or fade away. Lumen is still the big wild card that hasn’t fully committed to any single expansion strategy and is pursuing different paths. From folks who track what the big ISPs are doing, if nothing else, this makes them the most interesting company to watch.

Can the Big Telcos Turn the Corner with Fiber?

I was asked an interesting question recently: will fiber help the big telcos turn the corner to success? It’s a good question when looking at telcos like Frontier, Windstream, Lumen, and any others who are late to the game for converting copper to fiber. There are a lot of factors that will come into play, so the answer is likely to be different by telco.

On the plus side is a general consensus by many households that fiber is the best technology. There is a sizable percentage of homes in any market that will move to fiber given a chance. I’m sure this differs by community, but my experience is that 20% to 30% of homes will almost automatically switch to fiber, and that percentage is likely growing. It seems that all of the talk about broadband over the last few years has sold the idea that fiber is a superior technology.

We know telcos are hoping this perception is true. AT&T is the most optimistic of the big telcos and says that it will get a 50% market penetration anywhere it builds residential fiber. That’s an extraordinary prediction after considering the 10%-15% of homes in most places that still don’t have broadband and another 10%-15% of homes that will choose the low-cost alternative like DSL or cellular broadband just because of price. A 50% market share would mean largely obliterating the cable company in a given market.

But any perceived superiority of fiber is going to be relatively short-lived as cable companies upgrade networks to have faster upload speeds. Fiber today is winning the battle for consumers who care about upload speeds – but what percentage of homes is that? Fiber also has noticeably better latency and jitter – a connection on fiber is perceived by the eye to be faster even if the speeds are the same. But how many households care about that?

Telcos have a long way to go to get back to a decent market share. By delaying the transition to fiber, big telco let cable companies clobber them quarter after quarter in taking away DSL customers. While they hope that having a superior technology will help them claw back those lost customers, it’s no slam dunk that they will. The cable companies have smartly bundled broadband with cheap cellular service. The telcos are also going to see fierce competition for price-conscious consumers as they see cellular broadband offered by T-Mobile, Verizon, Dish Networks, and maybe AT&T.

One of the biggest handicaps that telcos face is that they have destroyed their brand names through poor treatment of customers. Big telcos have slowly let the copper networks die by cutting back on maintenance staff. There are millions of consumers who have a poor opinion of a telco because of week-long DSL outages or repair technicians who never showed up. People are not going to easily forgive them. Perhaps the smartest ones of all will be Ziply and Brightspeed, which purchased copper from Frontier and CenturyLink and rebranded to feel like a new company.

Another challenge will be for the big telcos to earn a decent margin from the conversion to fiber. The big telcos have a cheaper path to upgrade than fiber overbuilders due to the savings from overlashing fiber onto existing copper wires. But they are still making a significant outlay to make the conversion. There is no new revenue to the telcos from existing copper customers they move to fiber – and that means that they are paying for the new fiber networks only with the revenues from customers they lure back from cable companies. We won’t know for a few years what that means for the bottom line, but my back-of-the-envelope math says they’ll have a hard time making any noticeable return on the conversion to fiber – at least for the first 5-10 years. In the long run, the fiber customers will become cash cows, just like what happened with copper customers in the past. But will the long run be good enough for Wall Street, which will want to see a fast turnaround?

I think many of the big telcos are banking on getting giant federal grants to help them get back on their feet. But there are a lot of factors that say this might not be the great strategy they are trying to sell. First, most grants will be in the range of 75% grant funding. But covering the 25% is still expensive when the cost to reach rural passings ranges from $7,000 – $15,000. There are also higher operating costs in rural America due to longer truck rolls.

The biggest hurdle for getting grants is that the awards are going to be made at the State level – there won’t be any FCC to influence through lobbying. Many states are beyond angry with the big telcos since they rightly blame them for the poor condition of rural broadband. Additionally, the most likely grant winner in any county will be the one that partners with the county. I’ve worked in nearly 100 counties in recent years, and not one of them had any desire to partner with one of the big telcos. I know the big telcos all have huge goals of winning grant funding – I’m going to be really surprised if they achieve it.

To summarize and answer the original question – there is no guaranteed path to success for a telco that finally gets around to converting to fiber. The incremental new revenues from the conversion may not be high enough to make the math work. The big telcos will be battling a negative public perception of them as quality and reliable ISPs. They might be successful just because of the advantages that fiber has over cable company copper networks – but those advantages might not be enough to make a bottom-line difference.

National Broadband Growth is Slowing

Leichtman Research recently released the broadband customer statistics for the end of the fourth quarter of 2021. The numbers show that broadband growth has slowed significantly for the sixteen largest ISPs tracked by the company. LRG compiles these statistics from customer counts provided to stockholders, except for Cox which is privately owned.

Net customer additions sank each quarter during the year.  The first quarter of 2021 saw over 1 million net new broadband customers. That dropped to just under 900,000 in the second quarter, 630,000 in the third quarter, and now 423,000 in the fourth quarter. The statistics for all of 2021 and for the fourth quarter are as follows:

Annual % 4Q %
4Q 2021 Change Change Change Change
Comcast 30,574,000 1,327,000 4.3% 213,000 0.7%
Charter 28,879,000 1,210,000 4.2% 190,000 0.6%
AT&T 15,384,000 120,000 0.8% (6,000) 0.0%
Verizon 7,129,000 236,000 3.3% 28,000 0.4%
Cox 5,380,000 150,000 2.8% 20,000 0.4%
CenturyLink 4,767,000 (248,000) -5.2% (70,000) -1.5%
Altice 4,389,600 (3,400) -0.1% (1,900) 0.0%
Frontier 2,834,000 (35,000) -1.2% 10,000 0.4%
Mediacom 1,438,000 25,000 1.7% (3,000) -0.2%
Windstream 1,109,300 55,200 5.0% 17,500 1.5%
Cable ONE 992,000 63,000 6.4% 25,000 2.4%
Atlantic Broadband 698,000 18,778 2.7% (222) 0.0%
WOW! 498,800 12,900 2.6% 2,200 0.4%
TDS 493,300 32,700 6.6% 3,200 0.6%
Cincinnati Bell 436,100 3,900 0.9% 1,000 0.2%
Consolidated 401,357 (16,793) -4.2% (6,097) -1.6%
Total 105,403,457 2,951,285 2.8% 422,681 0.4%
Cable 72,849,400 2,803,278 3.8% 445,078 0.6%
Telco 32,554,057 148,007 0.5% (22,397) -0.1%
           
Fixed Wireless 874,000 719,000 82.3%    

There are a few interesting things to keep an eye on in the future. The growth for Comcast and Charter have slowed significantly and my prediction is that there will come a quarter within a year where one or both of them will lose net customers. For several years running, Frontier has been bleeding customers but seems to be turning it around. The big loser is now CenturyLink.

For some reason, LRG is leaving out fixed cellular customers. At the end of 2021, T-Mobile reported 646,000 fixed cellular customers, with 546,000 added in 2021. Verizon is up to 228,000 fixed cellular customers, up by 173,000 during 2021. The two companies, along with AT&T, are making a major push in this market and expect to add millions of customers in 2022 – many at the expense of the other ISPs on the list. It’s an odd choice to exclude these customers since the speeds on fixed cellular are faster than the DSL delivered by the telcos on the list. Also missing are other big providers that are probably larger than Consolidated, like a few of the largest WISPs and fiber overbuilders like Google Fiber.

But even after counting the growth of fixed cellular broadband, it’s obvious that the broadband market growth has cooled. The burst of new customers in 2020 and the first half of 2021 were clearly fueled by homes buying broadband during the pandemic.

It’s also worth noting that the numbers for WOW! and Atlantic Broadband (now Breezeline) have been adjusted for the sale of customers by WOW!.

The FCC and Broadband Outages

Comcast had a widespread network outage in early November. The problems started in San Francisco and spread the next day to Chicago, Philadelphia, parts of New Jersey, and three other states. The outage knocked out broadband customers along with Comcast cellular customers. Comcast has never disclosed the reason for the outage and announced only that it was due to a ‘network issue’.

In 2020 CenturyLink suffered an even larger outage that not only knocked out CenturyLink customers but spread into other networks, including Amazon, Cloudflare, and Hulu. The problem was blamed on a software update that blocked the establishment of Border Gateway Protocol (BGP) sessions and impeded broadband traffic routing.

T-Mobile also had a major network outage in 2020 that knocked out broadband customers and also cut off some voice calls and most texting for nearly a whole day. T-Mobile blamed the issue on problems with a leased circuit that was compounded by two previously undetected flaws in third-party software. Reports at the time said that the electronics failed on a leased circuit, and then the backup circuit also failed. This then caused a cascade that brought down a large part of the T-Mobile network.

In 2019 CenturyLink had perhaps the largest outage that knocked out much of its network and customers that relied on the Level 3 network for transport. The company blamed the outage on a bad circuit card in Denver that somehow cascaded to bring down a large swath of fiber networks in the West, including numerous 911 centers.

The FCC investigates big outages from time to time and opened an inquiry in October 2020 in a few of the outages listed above. The FCC also recently adopted a Notice of Proposed Rulemaking to investigate the disaster resiliency plans of major telecom providers to take a harder look at how cellular and broadband carriers make repairs after big storms.

Interestingly, the FCC recently fined T-Mobile $19.5 million for the 2020 outage, but not the other carriers. This is not because T-Mobile’s outage was worse than the others. T-Mobile was fined because they are a cellular carrier and still fully regulated by the FCC. But Comcast and CenturyLink are ISPs and under different regulatory rules.

Oddly, the FCC has very little power to do anything about ISP network outages because the FCC has very little regulatory authority over ISPs in general. The FCC abrogated its authority to regulate ISPs when it killed Title II regulation and handed a few vestiges of regulation to the Federal Trade Commission. The FCC only regulates ISPs tangentially through the specific authority given directly by Congress. Any authority the FCC once had as a result of claiming Title II regulatory authority is gone.

The process has finally started to seat a fifth FCC Commissioner, and the industry speculates that one of the early acts with five Commissioners will be to reinstate Title II authority. This effort might be a little more streamlined in the past because federal courts have already ruled that the FCC can choose to regulate or not regulate broadband.

Unfortunately, any move to regulate ISPs and broadband will only last until we have another shift in administration that wants to kill regulation again. We have ended up in an absurd regulatory merry-go-round where regulating or not regulating ISPs depends on the party that controls the White House. It makes no sense to not regulate ISPs at a time when cable companies have nearly total monopoly power in some markets. Overall, broadband might be the most important industry in the country because it powers just about everything else. Local jurisdictions around the country regulate occupations like nail salon technicians, plumbers, and masseuses, and yet we can’t get our act together as a country to regulate an industry where a handful of giant ISPs openly manifest monopoly behavior.

There is a really simple fix for this. Congress could give authority to the FCC to regulate broadband so that future FCCs or administrations could not undo it. It would only take a simple law that says something like, “The FCC shall regulate the broadband industry for the benefit of the citizens of the United States.” Obviously, lawyers could word this to be more ironclad – but giving the FCC the authority to regulate broadband doesn’t have to be complicated.

Network Outages Go Global

On August 30, CenturyLink experienced a major network outage that lasted for over five hours and which disrupted CenturyLink customers nationwide as well as many other networks. What was unique about the outage was the scope of the disruptions as the outage affected video streaming services, game platforms, and even webcasts of European soccer.

This is an example of how telecom network outages have expanded in size and scope and can now be global in scale. This is a development that I find disturbing because it means that our telecom networks are growing more vulnerable over time.

The story of what happened that day is fascinating and I’m including two links for those who want to peek into how the outages were viewed by outsiders who are engaged in monitoring Internet traffic flow. First is this report from a Cloudflare blog that was written on the day of the outage. Cloudflare is a company that specializes in protecting large businesses and networks from attacks and outages. The blog describes how Cloudflare dealt with the outage by rerouting traffic away from the CenturyLink network. This story alone is a great example of modern network protections that have been put into place to deal with major Internet traffic disruptions.

The second report comes from ThousandEyes, which is now owned by Cisco. The company is similar to Cloudflare and helps clients deal with security issues and network disruptions. The ThousandEye report comes from the day after the outage and discusses the likely reasons for the outage. Again, this is an interesting story for those who don’t know much about the operations of the large fiber networks that constitute the Internet. ThousandEyes confirms the suspicions that were expressed the day before by Cloudflare that the issue was caused by a powerful network command issued by CenturyLink using Flowspec that resulted in a logic loop that turned off and restarted BGP (Border Gateway Protocol) over and over again.

It’s reassuring to know that there are companies like Cloudflare and ThousandEye that can stop network outages from permeating into other networks. But what is also clear from the reporting of the event is that a single incident or bad command can take out huge portions of the Internet.

That is something worth examining from a policy perspective. It’s easy to understand how this happens at companies like CenturyLink. The company has acquired numerous networks over the years from the old Qwest network up to the Level 3 networks and has integrated them all into a giant platform. The idea that the company owns a large global network is touted to business customers as a huge positive – but is it?

Network owners like CenturyLink have consolidated and concentrated the control of the network to a few key network hubs controlled by a relatively small staff of network engineers. ThousandEyes says that the CenturyLink Network Operation Center in Denver is one of the best in existence, and I’m sure they are right. But that network center controls a huge piece of the country’s Internet backbone.

I can’t find where CenturyLink ever gave the exact reason why the company issued a faulty Flowspec command. It may have been used to try to tamp down a problem at one customer or have been part of more routine network upgrades implemented early on a Sunday morning when the Internet is at its quietest. From a policy perspective, it doesn’t matter – what matters is that a single faulty command could take down such a large part of the Internet.

This should cause concerns for several reasons. First, if one unintentional faulty command can cause this much damage, then the network is susceptible to this being done deliberately. I’m sure that the network engineers running the Internet will say that’s not likely to happen, but they also would have expected this particular outage to have been stopped much sooner and easier.

I think the biggest concern is that the big network owners have adopted the idea of centralization to such an extent that outages like this one are more and more likely. Centralization of big networks means that outages can now reach globally and not just locally like happened just a decade ago. Our desire to be as efficient as possible through centralization has increased the risk to the Internet, not decreased it.

A good analogy for understanding the risk in our Internet networks comes by looking at the nationwide electric grid. It used to be routine to purposefully allow neighboring grids to automatically interact until it because obvious after some giant rolling blackouts that we needed firewalls between grids. The electric industry reworked the way that grids interact, and the big rolling regional outages disappeared. It’s time to have that same discussion about the Internet infrastructure. Right now, the security of the Internet is in the hands of few corporations that stress the bottom line first, and which have willingly accepted increased risk to our Internet backbones as a price to pay for cost efficiency.

Partners are Where You Find Them

An interesting new partnership has been formed between Windstream and Colquitt Electric Membership Corp. of Georgia to build a rural fiber network. Windstream is a large price-cap telco that recently emerged from an interesting bankruptcy. Colquitt is a rural electric cooperative.

Only high-level terms of the partnership have been released. Windstream will own the fiber network, will provide broadband and other services, and will own the customers. Colquitt will provide access and rights-of-ways on poles and Colquitt technicians will place the new fiber on poles. Colquitt will get access to some fibers on the new network to connect electric substations and other electric network components to fiber. The partnership is described as having a network that is ‘jointly built and jointly-owned’.

The area to be served is rural and is described as having around 7 people per square mile. It’s a little hard to put that statistic into perspective because the most commonly used metric in the industry for understanding density is the number of homes per mile of road – however, the area sounds sparsely populated.

The state of Georgia decided a few years ago to allow electric cooperatives to become ISPs – a restriction that was imposed years ago by legislation prompted by telco incumbents. Many states have recently lifted such restrictions in an attempt to find more solutions to solve the rural broadband gap.

Partnerships with larger price-cap telephone companies to provide fiber broadband is a new phenomenon. An argument can be made that decisions made by price-cap telcos over the years are one of the major reasons why much of rural America is still served by DSL broadband provided over old and poorly-maintained copper networks.

But we’ve seen several similar partnerships with price-cap telcos. CenturyLink has partnered with the City of Springfield, Missouri to provide fiber. Consolidated Communications has partnered with several villages in New Hampshire to build fiber. Cincinnati Bell has partnered with the Butler Rural Electric Cooperative in Ohio.

This announcement is a reminder to rural communities and electric cooperatives that broadband partners might be found in unexpected places. It’s easy for rural folks to assume that the telcos that built and have been operating the dreadful copper networks are not interested in providing better service. In this case, the network is being built in a rural community and it’s extremely unlikely that Windstream could justify investing the full cost to build fiber – the return and payback on investment would never meet corporate earnings metrics and would make no sense as an investment. However, sharing the costs with the electric cooperative must have reduced Windstream’s costs to the point where the project makes financial sense.

That is the power of partnerships. Investing all of the cost to build fiber in this case probably didn’t make financial sense to either the electric cooperative or to Windstream. Both parties have something to gain out of the transaction. Windstream gains customers who will like the broadband service on fiber. The cooperative gets a fiber network connecting substations. Both are contributing to an improved community that will benefit both companies in the long-term. We’ve seen that fiber can reinvigorate a rural community. Many people want to live in rural areas but need good broadband to work from home – having a fiber network should attract new residents and keep residents some local people from leaving the area to find better broadband.

During the last year, my advice to rural communities is to have a serious discussion with the incumbent providers. Historically I’ve always advised to not bother with the incumbents because over decades I had never seen a large incumbent telco respond to plea to improve service. This is still a rare occurrence, but this partnership, and the ones mentioned earlier illustrate that it’s worth having the discussion on the outside chance that you hit the right note and the right opportunity to get the attention of the incumbents.