Tag: CenturyLink

Categories
The Industry Uncategorized

Lumen’s Fiber Path Forward

Lumen is taking a different path forward than the other big telcos. AT&T continues to build fiber in selected clusters, mostly in cities, rather than concentrate on building entire markets. Frontier, Windstream, and Consolidated are all concentrating on upgrading existing telco DSL networks to fiber.

Lumen has a different path forward. In a recent press release, the company announced a major upgrade to its long-haul fiber routes that cross the country. The company’s main fiber strategy is to beef up the intercity network with plans to add six million miles of fiber to existing fiber routes by 2026. In case you are wondering how there can possibly be six million route miles of fiber in the country – that count is miles of individual fibers. This is a marketing trick that long-haul fiber providers have been using for years to make networks seem gigantic.

The existing Lumen long-haul fiber network came to the company in two acquisitions. The original network came when CenturyLink bought US West, which had earlier merged with Qwest, a major builder of long-haul networks. The network was strengthened when CenturyLink purchased Level 3 Communications.

The original Quest fiber is getting dated in terms of capacity and performance. Much of this fiber was built thirty and forty years ago. While most of the fiber is still functional, fiber glass technology has improved drastically since then. Lumen will be using two low-loss types of fiber from Corning. This newer fiber is far clearer than older fiber and will increase the distance between repeater points while also allowing for using the fastest 400-gigabit electronics today and faster electronics later.

Earlier this year, Lumen announced it is improving its Ethernet architecture in forty cities this year. This means upgrading local networks to major customers to be able to provide speeds up to 30 gigabits. While this upgrade will mostly benefit business customers, this also will improve the local fiber backbone in these cities to 100 gigabits, which should improve performance for all broadband customers.

Lumen is also pursuing a last-mile fiber expansion. In August, the company announced fiber expansion plans in Denver, Minneapolis, and Seattle. The company had a target for this year to pass one million locations with fiber but has fallen a little behind due to supply chain and logistics.

Unlike the other telcos, Lumen hasn’t been talking much about the upcoming rural grant funding. This doesn’t mean the company might not pursue those opportunities since rural fiber expansion creates monopolies. But major residential expansion does not seem to be a key part of the Lumen plan, at least compared to plans for companies like Frontier, which says it plans to pass 12 million homes with fiber.

Another big unknown is if the company is still trying to sell any of its remaining copper networks like it did with sale of the twenty easternmost states to Apollo Global Management. It would be a more drastic affair to liquidate last-mile customers in the states where US West was formally the Bell company incumbent provider.

Any more sales of last-mile networks would be an interesting step where Lumen would be retracting to be a large business ISP. The company already had a sizable share of the business market that got bolstered by the acquisition of Level 3.

Lumen shares one characteristic with all of the big telcos in that it knows it must reinvent itself. After many years of no activity, Verizon is expanding FiOS again while also pushing a nationwide FWA network. AT&T is fully committed to building last-mile fiber networks and continues to add millions of new passings per year. The smaller telcos like Frontier and Windstream have clearly decided they must build fiber or fade away. Lumen is still the big wild card that hasn’t fully committed to any single expansion strategy and is pursuing different paths. From folks who track what the big ISPs are doing, if nothing else, this makes them the most interesting company to watch.

Categories
The Industry

Can the Big Telcos Turn the Corner with Fiber?

I was asked an interesting question recently: will fiber help the big telcos turn the corner to success? It’s a good question when looking at telcos like Frontier, Windstream, Lumen, and any others who are late to the game for converting copper to fiber. There are a lot of factors that will come into play, so the answer is likely to be different by telco.

On the plus side is a general consensus by many households that fiber is the best technology. There is a sizable percentage of homes in any market that will move to fiber given a chance. I’m sure this differs by community, but my experience is that 20% to 30% of homes will almost automatically switch to fiber, and that percentage is likely growing. It seems that all of the talk about broadband over the last few years has sold the idea that fiber is a superior technology.

We know telcos are hoping this perception is true. AT&T is the most optimistic of the big telcos and says that it will get a 50% market penetration anywhere it builds residential fiber. That’s an extraordinary prediction after considering the 10%-15% of homes in most places that still don’t have broadband and another 10%-15% of homes that will choose the low-cost alternative like DSL or cellular broadband just because of price. A 50% market share would mean largely obliterating the cable company in a given market.

But any perceived superiority of fiber is going to be relatively short-lived as cable companies upgrade networks to have faster upload speeds. Fiber today is winning the battle for consumers who care about upload speeds – but what percentage of homes is that? Fiber also has noticeably better latency and jitter – a connection on fiber is perceived by the eye to be faster even if the speeds are the same. But how many households care about that?

Telcos have a long way to go to get back to a decent market share. By delaying the transition to fiber, big telco let cable companies clobber them quarter after quarter in taking away DSL customers. While they hope that having a superior technology will help them claw back those lost customers, it’s no slam dunk that they will. The cable companies have smartly bundled broadband with cheap cellular service. The telcos are also going to see fierce competition for price-conscious consumers as they see cellular broadband offered by T-Mobile, Verizon, Dish Networks, and maybe AT&T.

One of the biggest handicaps that telcos face is that they have destroyed their brand names through poor treatment of customers. Big telcos have slowly let the copper networks die by cutting back on maintenance staff. There are millions of consumers who have a poor opinion of a telco because of week-long DSL outages or repair technicians who never showed up. People are not going to easily forgive them. Perhaps the smartest ones of all will be Ziply and Brightspeed, which purchased copper from Frontier and CenturyLink and rebranded to feel like a new company.

Another challenge will be for the big telcos to earn a decent margin from the conversion to fiber. The big telcos have a cheaper path to upgrade than fiber overbuilders due to the savings from overlashing fiber onto existing copper wires. But they are still making a significant outlay to make the conversion. There is no new revenue to the telcos from existing copper customers they move to fiber – and that means that they are paying for the new fiber networks only with the revenues from customers they lure back from cable companies. We won’t know for a few years what that means for the bottom line, but my back-of-the-envelope math says they’ll have a hard time making any noticeable return on the conversion to fiber – at least for the first 5-10 years. In the long run, the fiber customers will become cash cows, just like what happened with copper customers in the past. But will the long run be good enough for Wall Street, which will want to see a fast turnaround?

I think many of the big telcos are banking on getting giant federal grants to help them get back on their feet. But there are a lot of factors that say this might not be the great strategy they are trying to sell. First, most grants will be in the range of 75% grant funding. But covering the 25% is still expensive when the cost to reach rural passings ranges from $7,000 – $15,000. There are also higher operating costs in rural America due to longer truck rolls.

The biggest hurdle for getting grants is that the awards are going to be made at the State level – there won’t be any FCC to influence through lobbying. Many states are beyond angry with the big telcos since they rightly blame them for the poor condition of rural broadband. Additionally, the most likely grant winner in any county will be the one that partners with the county. I’ve worked in nearly 100 counties in recent years, and not one of them had any desire to partner with one of the big telcos. I know the big telcos all have huge goals of winning grant funding – I’m going to be really surprised if they achieve it.

To summarize and answer the original question – there is no guaranteed path to success for a telco that finally gets around to converting to fiber. The incremental new revenues from the conversion may not be high enough to make the math work. The big telcos will be battling a negative public perception of them as quality and reliable ISPs. They might be successful just because of the advantages that fiber has over cable company copper networks – but those advantages might not be enough to make a bottom-line difference.

Categories
Uncategorized

National Broadband Growth is Slowing

Leichtman Research recently released the broadband customer statistics for the end of the fourth quarter of 2021. The numbers show that broadband growth has slowed significantly for the sixteen largest ISPs tracked by the company. LRG compiles these statistics from customer counts provided to stockholders, except for Cox which is privately owned.

Net customer additions sank each quarter during the year.  The first quarter of 2021 saw over 1 million net new broadband customers. That dropped to just under 900,000 in the second quarter, 630,000 in the third quarter, and now 423,000 in the fourth quarter. The statistics for all of 2021 and for the fourth quarter are as follows:

Annual % 4Q %
4Q 2021 Change Change Change Change
Comcast 30,574,000 1,327,000 4.3% 213,000 0.7%
Charter 28,879,000 1,210,000 4.2% 190,000 0.6%
AT&T 15,384,000 120,000 0.8% (6,000) 0.0%
Verizon 7,129,000 236,000 3.3% 28,000 0.4%
Cox 5,380,000 150,000 2.8% 20,000 0.4%
CenturyLink 4,767,000 (248,000) -5.2% (70,000) -1.5%
Altice 4,389,600 (3,400) -0.1% (1,900) 0.0%
Frontier 2,834,000 (35,000) -1.2% 10,000 0.4%
Mediacom 1,438,000 25,000 1.7% (3,000) -0.2%
Windstream 1,109,300 55,200 5.0% 17,500 1.5%
Cable ONE 992,000 63,000 6.4% 25,000 2.4%
Atlantic Broadband 698,000 18,778 2.7% (222) 0.0%
WOW! 498,800 12,900 2.6% 2,200 0.4%
TDS 493,300 32,700 6.6% 3,200 0.6%
Cincinnati Bell 436,100 3,900 0.9% 1,000 0.2%
Consolidated 401,357 (16,793) -4.2% (6,097) -1.6%
Total 105,403,457 2,951,285 2.8% 422,681 0.4%
Cable 72,849,400 2,803,278 3.8% 445,078 0.6%
Telco 32,554,057 148,007 0.5% (22,397) -0.1%
           
Fixed Wireless 874,000 719,000 82.3%    

There are a few interesting things to keep an eye on in the future. The growth for Comcast and Charter have slowed significantly and my prediction is that there will come a quarter within a year where one or both of them will lose net customers. For several years running, Frontier has been bleeding customers but seems to be turning it around. The big loser is now CenturyLink.

For some reason, LRG is leaving out fixed cellular customers. At the end of 2021, T-Mobile reported 646,000 fixed cellular customers, with 546,000 added in 2021. Verizon is up to 228,000 fixed cellular customers, up by 173,000 during 2021. The two companies, along with AT&T, are making a major push in this market and expect to add millions of customers in 2022 – many at the expense of the other ISPs on the list. It’s an odd choice to exclude these customers since the speeds on fixed cellular are faster than the DSL delivered by the telcos on the list. Also missing are other big providers that are probably larger than Consolidated, like a few of the largest WISPs and fiber overbuilders like Google Fiber.

But even after counting the growth of fixed cellular broadband, it’s obvious that the broadband market growth has cooled. The burst of new customers in 2020 and the first half of 2021 were clearly fueled by homes buying broadband during the pandemic.

It’s also worth noting that the numbers for WOW! and Atlantic Broadband (now Breezeline) have been adjusted for the sale of customers by WOW!.

Categories
Regulation - What is it Good For?

The FCC and Broadband Outages

Comcast had a widespread network outage in early November. The problems started in San Francisco and spread the next day to Chicago, Philadelphia, parts of New Jersey, and three other states. The outage knocked out broadband customers along with Comcast cellular customers. Comcast has never disclosed the reason for the outage and announced only that it was due to a ‘network issue’.

In 2020 CenturyLink suffered an even larger outage that not only knocked out CenturyLink customers but spread into other networks, including Amazon, Cloudflare, and Hulu. The problem was blamed on a software update that blocked the establishment of Border Gateway Protocol (BGP) sessions and impeded broadband traffic routing.

T-Mobile also had a major network outage in 2020 that knocked out broadband customers and also cut off some voice calls and most texting for nearly a whole day. T-Mobile blamed the issue on problems with a leased circuit that was compounded by two previously undetected flaws in third-party software. Reports at the time said that the electronics failed on a leased circuit, and then the backup circuit also failed. This then caused a cascade that brought down a large part of the T-Mobile network.

In 2019 CenturyLink had perhaps the largest outage that knocked out much of its network and customers that relied on the Level 3 network for transport. The company blamed the outage on a bad circuit card in Denver that somehow cascaded to bring down a large swath of fiber networks in the West, including numerous 911 centers.

The FCC investigates big outages from time to time and opened an inquiry in October 2020 in a few of the outages listed above. The FCC also recently adopted a Notice of Proposed Rulemaking to investigate the disaster resiliency plans of major telecom providers to take a harder look at how cellular and broadband carriers make repairs after big storms.

Interestingly, the FCC recently fined T-Mobile $19.5 million for the 2020 outage, but not the other carriers. This is not because T-Mobile’s outage was worse than the others. T-Mobile was fined because they are a cellular carrier and still fully regulated by the FCC. But Comcast and CenturyLink are ISPs and under different regulatory rules.

Oddly, the FCC has very little power to do anything about ISP network outages because the FCC has very little regulatory authority over ISPs in general. The FCC abrogated its authority to regulate ISPs when it killed Title II regulation and handed a few vestiges of regulation to the Federal Trade Commission. The FCC only regulates ISPs tangentially through the specific authority given directly by Congress. Any authority the FCC once had as a result of claiming Title II regulatory authority is gone.

The process has finally started to seat a fifth FCC Commissioner, and the industry speculates that one of the early acts with five Commissioners will be to reinstate Title II authority. This effort might be a little more streamlined in the past because federal courts have already ruled that the FCC can choose to regulate or not regulate broadband.

Unfortunately, any move to regulate ISPs and broadband will only last until we have another shift in administration that wants to kill regulation again. We have ended up in an absurd regulatory merry-go-round where regulating or not regulating ISPs depends on the party that controls the White House. It makes no sense to not regulate ISPs at a time when cable companies have nearly total monopoly power in some markets. Overall, broadband might be the most important industry in the country because it powers just about everything else. Local jurisdictions around the country regulate occupations like nail salon technicians, plumbers, and masseuses, and yet we can’t get our act together as a country to regulate an industry where a handful of giant ISPs openly manifest monopoly behavior.

There is a really simple fix for this. Congress could give authority to the FCC to regulate broadband so that future FCCs or administrations could not undo it. It would only take a simple law that says something like, “The FCC shall regulate the broadband industry for the benefit of the citizens of the United States.” Obviously, lawyers could word this to be more ironclad – but giving the FCC the authority to regulate broadband doesn’t have to be complicated.

Categories
Technology The Industry

Network Outages Go Global

On August 30, CenturyLink experienced a major network outage that lasted for over five hours and which disrupted CenturyLink customers nationwide as well as many other networks. What was unique about the outage was the scope of the disruptions as the outage affected video streaming services, game platforms, and even webcasts of European soccer.

This is an example of how telecom network outages have expanded in size and scope and can now be global in scale. This is a development that I find disturbing because it means that our telecom networks are growing more vulnerable over time.

The story of what happened that day is fascinating and I’m including two links for those who want to peek into how the outages were viewed by outsiders who are engaged in monitoring Internet traffic flow. First is this report from a Cloudflare blog that was written on the day of the outage. Cloudflare is a company that specializes in protecting large businesses and networks from attacks and outages. The blog describes how Cloudflare dealt with the outage by rerouting traffic away from the CenturyLink network. This story alone is a great example of modern network protections that have been put into place to deal with major Internet traffic disruptions.

The second report comes from ThousandEyes, which is now owned by Cisco. The company is similar to Cloudflare and helps clients deal with security issues and network disruptions. The ThousandEye report comes from the day after the outage and discusses the likely reasons for the outage. Again, this is an interesting story for those who don’t know much about the operations of the large fiber networks that constitute the Internet. ThousandEyes confirms the suspicions that were expressed the day before by Cloudflare that the issue was caused by a powerful network command issued by CenturyLink using Flowspec that resulted in a logic loop that turned off and restarted BGP (Border Gateway Protocol) over and over again.

It’s reassuring to know that there are companies like Cloudflare and ThousandEye that can stop network outages from permeating into other networks. But what is also clear from the reporting of the event is that a single incident or bad command can take out huge portions of the Internet.

That is something worth examining from a policy perspective. It’s easy to understand how this happens at companies like CenturyLink. The company has acquired numerous networks over the years from the old Qwest network up to the Level 3 networks and has integrated them all into a giant platform. The idea that the company owns a large global network is touted to business customers as a huge positive – but is it?

Network owners like CenturyLink have consolidated and concentrated the control of the network to a few key network hubs controlled by a relatively small staff of network engineers. ThousandEyes says that the CenturyLink Network Operation Center in Denver is one of the best in existence, and I’m sure they are right. But that network center controls a huge piece of the country’s Internet backbone.

I can’t find where CenturyLink ever gave the exact reason why the company issued a faulty Flowspec command. It may have been used to try to tamp down a problem at one customer or have been part of more routine network upgrades implemented early on a Sunday morning when the Internet is at its quietest. From a policy perspective, it doesn’t matter – what matters is that a single faulty command could take down such a large part of the Internet.

This should cause concerns for several reasons. First, if one unintentional faulty command can cause this much damage, then the network is susceptible to this being done deliberately. I’m sure that the network engineers running the Internet will say that’s not likely to happen, but they also would have expected this particular outage to have been stopped much sooner and easier.

I think the biggest concern is that the big network owners have adopted the idea of centralization to such an extent that outages like this one are more and more likely. Centralization of big networks means that outages can now reach globally and not just locally like happened just a decade ago. Our desire to be as efficient as possible through centralization has increased the risk to the Internet, not decreased it.

A good analogy for understanding the risk in our Internet networks comes by looking at the nationwide electric grid. It used to be routine to purposefully allow neighboring grids to automatically interact until it because obvious after some giant rolling blackouts that we needed firewalls between grids. The electric industry reworked the way that grids interact, and the big rolling regional outages disappeared. It’s time to have that same discussion about the Internet infrastructure. Right now, the security of the Internet is in the hands of few corporations that stress the bottom line first, and which have willingly accepted increased risk to our Internet backbones as a price to pay for cost efficiency.

Categories
The Industry

Partners are Where You Find Them

An interesting new partnership has been formed between Windstream and Colquitt Electric Membership Corp. of Georgia to build a rural fiber network. Windstream is a large price-cap telco that recently emerged from an interesting bankruptcy. Colquitt is a rural electric cooperative.

Only high-level terms of the partnership have been released. Windstream will own the fiber network, will provide broadband and other services, and will own the customers. Colquitt will provide access and rights-of-ways on poles and Colquitt technicians will place the new fiber on poles. Colquitt will get access to some fibers on the new network to connect electric substations and other electric network components to fiber. The partnership is described as having a network that is ‘jointly built and jointly-owned’.

The area to be served is rural and is described as having around 7 people per square mile. It’s a little hard to put that statistic into perspective because the most commonly used metric in the industry for understanding density is the number of homes per mile of road – however, the area sounds sparsely populated.

The state of Georgia decided a few years ago to allow electric cooperatives to become ISPs – a restriction that was imposed years ago by legislation prompted by telco incumbents. Many states have recently lifted such restrictions in an attempt to find more solutions to solve the rural broadband gap.

Partnerships with larger price-cap telephone companies to provide fiber broadband is a new phenomenon. An argument can be made that decisions made by price-cap telcos over the years are one of the major reasons why much of rural America is still served by DSL broadband provided over old and poorly-maintained copper networks.

But we’ve seen several similar partnerships with price-cap telcos. CenturyLink has partnered with the City of Springfield, Missouri to provide fiber. Consolidated Communications has partnered with several villages in New Hampshire to build fiber. Cincinnati Bell has partnered with the Butler Rural Electric Cooperative in Ohio.

This announcement is a reminder to rural communities and electric cooperatives that broadband partners might be found in unexpected places. It’s easy for rural folks to assume that the telcos that built and have been operating the dreadful copper networks are not interested in providing better service. In this case, the network is being built in a rural community and it’s extremely unlikely that Windstream could justify investing the full cost to build fiber – the return and payback on investment would never meet corporate earnings metrics and would make no sense as an investment. However, sharing the costs with the electric cooperative must have reduced Windstream’s costs to the point where the project makes financial sense.

That is the power of partnerships. Investing all of the cost to build fiber in this case probably didn’t make financial sense to either the electric cooperative or to Windstream. Both parties have something to gain out of the transaction. Windstream gains customers who will like the broadband service on fiber. The cooperative gets a fiber network connecting substations. Both are contributing to an improved community that will benefit both companies in the long-term. We’ve seen that fiber can reinvigorate a rural community. Many people want to live in rural areas but need good broadband to work from home – having a fiber network should attract new residents and keep residents some local people from leaving the area to find better broadband.

During the last year, my advice to rural communities is to have a serious discussion with the incumbent providers. Historically I’ve always advised to not bother with the incumbents because over decades I had never seen a large incumbent telco respond to plea to improve service. This is still a rare occurrence, but this partnership, and the ones mentioned earlier illustrate that it’s worth having the discussion on the outside chance that you hit the right note and the right opportunity to get the attention of the incumbents.

Categories
The Industry

Who’s Chasing RDOF Grants?

There is a veritable Who’s Who of big companies that have registered for the upcoming RDOF auction. All of the hundreds of small potential bidders to the auction have to be a bit nervous seeing the list of companies they could end up bidding against.

As a reminder, RDOF stands for Rural Digital Opportunity Fund and is an auction that starts in October that will award up to $16.4 billion in broadband funding. The money will be awarded by reverse auction in a process that favors faster technologies, but also favors those willing to take the lowest amount of grant per customer. The areas that are eligible for the funding are among the most remote places in the country, which is why the list of potential large bidders is puzzling.

There are some big cable companies on the list: Altice, Charter Communications, Cox Communications, Atlantic Broadband, Midco, and Mediacom Communications. These companies serve many of the county seats or other nearby towns to many of the RDOF areas. One has to wonder what these companies have in mind. The only one that has chased any significant federal grants in the past is Midco in Minnesota and North Dakota. Midco has been using grant money to extend fiber backhaul to connect its smallest markets, to build last-mile broadband in some tiny towns, and to build fixed wireless in rural areas surrounding its cable markets.

One has to wonder if the other cable companies have a similar plan. It’s incredibly inefficient to build traditional hybrid coaxial-fiber networks in rural areas, so it’s unlikely that the cable companies will be extending their existing networks. The RDOF auction is being done by Census blocks, which in rural areas can cover a large area. The winner of the auction for a given Census block must offer service to everybody in that block. I also have a hard time envisioning all of these big cable companies getting into the wireless business like Midco is doing, so their presence in the auction is a bit of a mystery.

Then there are the traditional large telcos including Frontier, Windstream, Consolidated Communications, and CenturyLink. These companies already serve many of the areas that are covered by the reverse auction. These are the rural areas where these companies have largely neglected the old copper wiring and either offer no broadband or dreadfully slow DSL. The minimum technology allowed to enter the auction must deliver 25/3 Mbps broadband. It’s almost painful to think that these companies would chase the funding and promise to upgrade DSL to 25/3 Mbps after these companies largely botched an upgrade to 10/1 Mbps DSL in the just-ending CAF II grants. The cynic in me says they are willing to pretend to upgrade DSL all over again if that means substantial grant money. I have to think that some of these companies are considering deploying fixed wireless. To the extent any of these companies is willing to take on new debt or use equity, they could also build fiber. None of these companies has built a substantial amount of fiber to truly rural places, but may these grants are the inducement they were waiting for.

Verizon and U.S. Cellular have registered for the auction. You have to think the cellular carriers will be deploying fixed cellular broadband like the 4G FWA product that Verizon just announced recently. These companies already have equipment on towers in many of the RDOF grant areas and would love to grab a subsidy to roll out a product they might be selling in these areas anyway.

Then there are the satellite companies SpaceX, Hughes Network Systems, and Viasat. Viasat has won federal grant money before for selling broadband from its high-altitude satellites. SpaceX is the wildcard since nobody knows anything about the pricing or real speeds they can provide. We know that Elon Musk has been lobbying the FCC to let him have a shot at the billions up for grabs in this auction.

There is another interesting wildcard with Starry. Their business plan is currently selling fixed wireless to large apartment buildings in center cities and they’ve developed a proprietary technology that’s perfect for that application. They must have something else in mind in chasing grant money in remote areas that are 180 degrees different than their normal business model. Starry founder Chet Kanojia is incredibly creative, so he probably has a new technology in mind if he wins auction funding.

There may be other big players in the auction as well since many of the registered bidders are participating under partnerships or corporations that are disguising their identity for now. I think one thing is clear and some of the rural ISPs and cooperative who think nobody else is interested in their markets will get a surprise early in the auction. These big companies didn’t register for the grant auction to sit on the sidelines.

Categories
The Industry

Walking Away from Copper

It’s been clear for many years that the big telcos are looking for ways to walk away from legacy copper networks. Big telco copper is getting old and most was built in the 1950s and 1960s. All of this copper is far past the 40-year expected lives that the telcos claimed when they built the networks. Even old copper can be made to work if it was well-maintained, but the big telcos stopped doing routine maintenance on copper decades ago. For years, the big telco maintenance policy has been to patch problems without improving or fixing network issues.

In some cases, the big telcos have gone through the formal FCC process of formally retiring copper. This requires giving customers a 90-day notice that copper will be deactivated and providing customers an alternative to copper.  For example, Verizon posts notices of copper retirement on this web site. There have been no announced retirements this year, likely due to the COVID-19 pandemic, but Verizon was active last year, like in this September notification for Massachusetts. CenturyLink has made similar notices in parts of Arizona, Colorado, Minnesota, Nebraska, Oregon, Utah, and Washington.

In all of these cases, Verizon and CenturyLink made the announcements in communities where the carriers can provide fiber-to-the-home. It’s a natural technological progression to replace old copper technology with new fiber, and customers who lose copper to move to fiber have little room to complain.

But what about all of the places where the telcos never plan to offer fiber? There are still huge areas, including big parts of major cities where the telcos have no plans of migrating to fiber. What will happen to folks in regions where the copper is rotting past the point of usefulness, like is described in this article from last year in Fauquier County, Virginia? In that county the copper barely works for voice, which is sadly becoming the norm and not the exception.

There is nothing the big telcos can do with copper that has gone past the point of no return. No telco is going to replace bad copper and none of the big telcos are going to extend fiber into rural America or into urban neighborhoods where construction is too expensive. Verizon might be replacing copper with fiber around Boston, as indicated by the above filing, but the telco has no plans for building fiber in western Massachusetts, Cape Cod, or the many other places in the state where it never built FiOS fiber.

We might have gotten a glimpse into Verizon’s strategy when the company recently unveiled a 4G fixed wireless product. This 4G Home product promises to deliver 25 Mbps broadband using the 4G cellular network and Verizon could point to this product as a justification to abandon DSL over copper.

On paper, the 4G Home product will outproduce rural DSL, which typically has speeds well under 10 Mbps. But the Fauquier County article pointed out another ugly truth – much of rural America has poor cellular coverage to go along with outdated copper. The 4G Home product is not going to work for homes that are more than a mile or two from a cell tower. 4G Home is not going to be a reasonable substitute for DSL in communities like the towns on Cape Cod where the density is too high to support a lot of subscribers using 4G data as a landline data substitute – even a small customer penetration would swamp the 4G LTE network in populated areas.

AT&T has a similar fixed wireless product it introduced during the past year as the solution for meeting the company’s rural CAF II requirements. I’ve been tracking this product on the web and still don’t see local articles or chatter from many folks who have changed to the wireless product. AT&T has implemented this product to satisfy the FCC (and to keep the CAF II grant funding), but for some reason the company doesn’t seem to be pushing the product very hard.

The bottom line is these telcos will have to walk away from copper at some point within the next decade for the simple reason that the networks will stop functioning. From what I can see, both the FCC and many state regulatory commissions refuse to acknowledge that copper is dying and keep pretending that the telcos can somehow make this work. These networks are dying. The telcos might toss a bone to regulators by halfheartedly offering a wireless substitute for DSL. But the telcos are under no obligation to offer a replacement if the copper dies. Sadly, we’re going to look up five years from now and find a lot of rural homes without a telephone line and a cellular connection. There was a time when that was unthinkable, but it’s the coming reality.

Categories
Current News

Google Fiber Comes to Iowa

The City of West Des Moines recently announced a deal with Google Fiber to bring fiber to pass all 36,000 residents and businesses in the city. This is a unique business model that can best be described as open-access conduit.

The city says that the estimated cost of the construction is between $35 million and $40 million and that the construction of the network should be complete in about two-and-a-half years. The full details of the plan have not yet been released, but the press is reporting that Google Fiber will pay $2.25 per month to the city for each customer that buys service from Google Fiber.

What is most unique about this arrangement is that conduit will be built along streets and into yards and parking lots to reach every home and business. I know of many cities that lease out some empty conduit to ISPs and carriers, but the big limitation of most empty conduit is that it doesn’t provide easy access to get from the street to reach a customer. West Des Moines will be spending the money to build the conduit to serve the last hundred feet.

This business arrangement will still require Google Fiber to pull fiber throughout the entire empty conduit network – but that is far cheaper for the company than building a network from scratch. The big cost of building any fiber network is the labor needed to bring the fiber along every street – and the city has absorbed that cost. The benefit of this arrangement for Google Fiber is obvious – the company saves the cost of building a standalone fiber network in the City. It’s the cost of financing expensive networks up-front that makes ISPs hesitant to enter new markets.

From a construction perspective, I’m sure that the City is building fiber with some form of innerduct – which is a conduit with multiple interior tubes that can accommodate multiple fibers (as is shown in the picture accompanying this blog). This would allow additional ISPs to coexist in the same conduits. If the conduits built through yards also include innerduct it would make it convenient for a customer to change fiber ISPs – disconnect fiber from ISP A and connect to the fiber from ISP B.

The City is banking on other ISPs using the empty conduit because Google Fiber fees alone won’t compensate the city for the cost of the conduit. The press reported that Google Fiber has guaranteed the City a minimum payment of at least $4.5 million over 20 years. I’m sure the City is counting on Google Fiber to perform a lot better than that minimum, but even if Google Fiber connects to half of all of the customers in the City, the $2.25 monthly fee won’t repay the City’s cost of the conduit.

This business model differs significantly from the typical open-access network model. In other open-access networks, the City pays for 100% of the cost of the network and the electronics up to the side of a home or business. The typical monthly fee for an ISP to reach a customer in these open access-networks ranges between $30 and $45 per month. Those high fees invariably push ISPs into cherry-picking and only pursuing customers willing to pay high monthly rates. The $2.25 fee in West Des Moines won’t push ISPs to automatically cherry-pick or charge a lot.

Any ISP willing to come to the city has a few issues to consider. They avoid the big cost of constructing the conduit network. But a new ISP will still need to pay to blow fiber through the conduit. Any new ISP will also be competing against Google Fiber. One of the most intriguing ISPs already in the market is CenturyLink. The company has shown in Springfield, Missouri that it is willing to step outside the traditional business model and use somebody else’s network. I would have to imagine that other ISPs in the Midwest perked up at this announcement.

In announcing the network, the City said that they hoped this network would bring fiber to everybody in the City. Google Fiber doesn’t typically compete on price. Earlier this year Google Fiber discontinued its 100 Mbps broadband connection for $50. Many homes are going to find the $70 gigabit product from Google Fiber to be unaffordable. It will be interesting over time to see how the city plans on getting broadband to everybody. Even municipalities that own their own fiber network are struggling with the concept of subsidizing fiber connections below cost to make them affordable.

One thing this partnership shows is that there are still new ideas to try in the marketplace. For an open-access conduit system to be effective means attracting multiple ISPs, so this idea isn’t going to work in markets much smaller than West Des Moines. But this is another idea for cities to consider if the goal is to provide world-class broadband for citizens and businesses.

Categories
The Industry

Big Regional Network Outages

T-Mobile had a major network outage last week that cut off some voice calls and most texting for nearly a whole day. The company’s explanation of the outage was provided by Neville Ray, the president of technology.

The trigger event is known to be a leased fiber circuit failure from a third party provider in the Southeast. This is something that happens on every mobile network, so we’ve worked with our vendors to build redundancy and resiliency to make sure that these types of circuit failures don’t affect customers. This redundancy failed us and resulted in an overload situation that was then compounded by other factors. This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls.

In plain English, the electronics failed on a leased circuit, and then the back-up circuit also failed. This then caused a cascade that brought down a large part of the T-Mobile network.

You may recall that something similar happened to CenturyLink about two years ago. At the time the company blamed the outage on a bad circuit card in Denver that somehow cascaded to bring down a large swatch of fiber networks in the West, including numerous 911 centers. Since that outage, there have been numerous regional outages, which is one of the reasons that Project THOR recently launched in Colorado – the cities in that region could no longer tolerate the recurring multi-hour or even day-long regional network outages,

Having electronics fail is a somewhat common event. This is particularly true on circuits provided by the big carriers which tend to push the electronics to the max and keep equipment running to the last possible moment of its useful life. Anybody visiting a major telecom hub would likely be aghast at the age of some of the electronics still being used to transmit voice and data traffic.

I can recall two of my clients that have had similar experiences in the last few years. They had a leased circuit fail and then also saw the redundant path fail as well. In both cases, it turns out that the culprit was the provider of the leased circuits, which did not provide true redundancy. Although my clients had paid for redundancy, the carrier had sold them primary and backup circuits that shared some of the same electronics at ley points in the network – and when those key points failed their whole network went down.

However, what is unusual about the two big carrier outages is that the outages somehow cascaded into big regional outages. That was largely unheard of a decade ago. This reminds more of what we saw in the past in the power grid, when power outages in one town could cascade over large areas. The power companies have been trying to remedy this situation by breaking the power grid into smaller regional networks and putting in protection so that failures can’t overwhelm the interfaces between regional networks. In essence, the power companies have been trying to introduce some of the good lessons learned over time by the big telecom companies.

But it seems that the big telecom carriers are going in the opposite direction. I talked to several retired telecom network engineers and they all made the same guess about why we are seeing big regional outages. The telecom network used to be comprised of hundreds of regional hubs. Each hub had its own staff and operations and it was physically impossible for a problem from one hub to somehow take down a neighboring hub. The worst that would happen is that routes between hubs could go dark, but the problem never moved past the original hub.

The big telcos have all had huge numbers of layoffs over the last decade, and those purges have emptied out the big companies of the technicians that built and understood the networks. Meanwhile, the companies are trying to find efficiencies to get by with smaller staffing. It appears that the efficiencies that have been found are to introduce network solutions that cover large areas or even the whole nation. This means that the identical software and technicians are now being used to control giant swaths of the network. This homogenization and central control of a network means that failure in any one place in the network might cascade into a larger problem if the centralized software and/or technicians react improperly to a local outage. It’s likely that the big outages we’re starting to routinely see are caused by a combination of the  failure of people and software systems.

A few decades ago we somewhat regular power outages that affected multiple states. At the prodding of the government, the power companies undertook a nationwide effort to stop cascading outages, and in doing so they effectively emulated the old telecom network world. They ended the ability for an electric grid to automatically interface with neighboring grids and the last major power outage that wasn’t due to weather happened in the west in 2011.

I’ve seen absolutely no regulatory recognition of the major telecom outages we’ve been seeing. Without the FCC pushing the big telcos, it’s highly likely nothing will change. It’s frustrating to watch the telecom networks deteriorate at the same time that electric companies got together and fixed their issues.

Exit mobile version