The Huge CenturyLink Outage

At the end of December CenturyLink had a widespread network outage that lasted over two days. The outage disrupted voice and broadband service across the company’s wide service territory.

Probably the most alarming aspect pf the outage is that it knocked out the 911 systems in parts of fourteen states. It was reported that calls to 911 might get a busy signal or a recording saying that “all circuits are busy’. In other cases, 911 calls were routed to the wrong 911 center. Some jurisdictions responded to the 911 problems by sending out emergency text messages to citizens providing alternate telephone numbers to dial during an emergency. The 911 service outages prompted FCC Chairman Ajit Pai to call CenturyLink and to open a formal investigation into the outage.

I talked last week to a resident of a small town in Montana who said that the outage was locally devasting. Credit cards wouldn’t work for most of the businesses in town including at gas stations. Businesses that rely on software in the cloud for daily operations like hotels were unable to function. Bank ATMs weren’t working. Customers with CenturyLink landlines had spotty service and mostly could not make or receive phone calls. Worse yet, cellular service in the area largely died, meaning that CenturyLink must have been supplying the broadband circuits supporting the cellular towers.

CenturyLink reported that the outage was caused by a faulty networking management card in a Colorado data center that was “propagating invalid frame packets across devices”. It took the company a long time to isolate the problem, and the final fix involved rebooting much of the network electronics.

Every engineer I’ve spoken to about this says that in today’s world it’s hard to believe that it would take 2 days to isolate and fix a network problem caused by a faulty card. Most network companies operate a system of alarms that instantly notify them when any device or card is having problems. Further, complex networks today are generally supplied with significant redundancy that allows the isolation of troubled components of a network in order to stop the kind of cascading outage that occurred in this case. The engineers all said that it’s almost inconceivable to have a single component like a card in a modern network that could cause such a huge problem. While network centralization can save money, few companies route their whole network through choke points – there are a dozen different strategies to create redundancy and protect against this kind of outage.

Obviously none of us knows any of the facts beyond the short notifications issued by CenturyLink at the end of the outage, so we can only speculate about what happened. Hopefully the FCC enquiry will uncover the facts – and it’s important that they do so, because it’s always possible that the cause of the outage is something that others in the industry need to be concerned about.

I’m only speculating, but my guess is that we are going to find that the company has not implemented best network practices in the legacy telco network. We know that CenturyLink and the other big telcos have been ignoring the legacy networks for decades. We see this all of the time when looking at the conditions of the last mile network, and we’ve always figured that the telcos were also not making the needed investments at the network core.

If this outage was caused by outdated technology and legacy network practices then such outages are likely to recur. Interestingly, CenturyLink also operates one of the more robust enterprise cloud services in the country. That business got a huge shot in the arm through the merger with Level 3, with new management saying that all of their future focus is going to be on the enterprise side of the house. I have to think that this outage didn’t much touch that network, just more likely the legacy network.

One thing for sure is that this outage is making CenturyLink customers look for an alternative. A decade ago the local government in Cook County, Minnesota – the northern-most county in the state – was so frustrated by continued prolonged CenturyLink network outages that they finally built their own fiber-to-the-home network and found alternate routing into and out of the County. I talked to one service provider in Montana who said they’ve been inundated after this recent outage by businesses looking for an alternate to CenturyLink.

We have become so reliant on the Internet that major outages are unacceptable. Much of what we do everyday relies on the cloud. The fact that this outage extended to cellular outages, a crash of 911 systems and the failure of credit card processing demonstrates how pervasive the network is in the background of our daily lives. It’s frightening to think that there are legacy telco networks that have been poorly maintained that can still cause these kinds of widespread problems.

I’m not sure what the fix is for this problem. The FCC supposedly washed their hands of the responsibility for broadband networks – so they might not be willing to tackle any meaningful solutions to prevent future network crashes. Ultimately the fix might the one found by Cook County, Minnesota – communities finding their own network solutions that bypass the legacy networks.

Our National Telecom Priorities

I recently wrote a blog that talked about the FCC’s formal goals for the next few years. I noted in that blog that some of the FCC’s actions currently seem to conflict with their stated goals. Today I present my take on what I see as the actual current priorities in our industry.

5G, 5G, 5G. The FCC and other policy makers have swallowed the 5G hype hook, line and sinker. I have no doubt that 5G will be an important part of our future telecom landscape, but the hype seems way out of proportion to the reality we are likely to see. Nothing highlights this better than a Qualcomm-sponsored article that claims that 5G technology will be as important as the introduction of electricity.

The FCC is sweeping away regulations that might interfere with 5G and already killed local say over the location of small cell electronics and towers. The FCC is well on the way towards allocating massive amounts of spectrum for 5G and ignoring other spectrum needs. The White House even held a 5G summit where politicians were repeating the talking points of the 5G carriers.

This all seems premature since engineers all say that the major benefits of mature 5G will come years from now. There will be some early 5G technology introduced into the market over the next few years, but this will not include the characteristics that make 5G an important technology. From a policy perspective, 5G seems to have won the war without having had to fight any of the battles. I’ve never seen this industry (and the politicians) go so gaga over a new technology that we aren’t even going to see for a while. The marketers at the cellular companies have clearly hit a hype home run.

The Rural Digital Divide Gets Lip Service. Talking about solving the rural digital divide is a high priority. The FCC rarely makes a presentation without mentioning how important this is to them. However, the FCC and others in Washington DC are doing almost nothing to solve the problem. The FCC even went so far as to list the rural digital divide as the first priority on their own list of goals but has done little to address the problem.

There is universal acknowledgement that the private sector is not going to invest in rural broadband without some funding help from government. Yet all of the state and federal grant programs added together are throwing millions of dollars at a problem that needs many billions of dollars to solve.

Meanwhile, the rural digital divide is widening as urban areas are seeing significantly faster broadband speeds while rural America is stuck with little or no broadband.

The Big ISPs Want to be Google. Every one of the big ISPs has made investments to try to catch-up with Google. The big ISPs want to monetize their vast troves of customer data. Big ISPs are envious of the advertising money made by Google and Facebook and want to grab a piece of those dollars. The FCC has aided the big companies by weakening consumer privacy protections.

But for whatever reason, the big ISPs haven’t yet figured this out. They have the most intimate and detailed access to customer data but have scarcely found any ways to understand it, yet alone monetize it.

Take My Residential Customers, Please. The big telcos have made it clear that they are not particularly interested in the residential market. CenturyLink made it clear this year that they will no longer invest in residential networks. Verizon has already sold vast tracts of rural networks. AT&T is constantly petitioning the FCC to let them tear down rural copper. Verizon is talking about expanding wireless local loops using 5G, but we’ll have to wait to see how serious they are about it.

Big ISPs Continue to Try to Squash Competition. The big ISPs miss no opportunity to squash competition, no matter how small. They all still rail against municipal competition, although all such competition added together is barely a blip on the national radar. They still pay for hit pieces – articles and papers that blast municipal fiber networks – even ones like Chattanooga EPB that is a paragon of competitiveness. They have been working hard to kick CLECs off of their dying copper networks, even thought the CLECs have been investing in newer DSL that can deliver decent broadband over the copper.

Broadband Statistics – 3Q 2018

As a nation we are approaching an 85% overall penetration of residential broadband. The following statistics come from the latest report from the Leichtman Group and compares broadband customers at the end of the recent 3Q of 2018 to the end of 2017.

 3Q 2018 4Q 2017 Change
Comcast 26,872,000 25,869,000 1,003,000 3.9%
Charter 24,930,000 23,903,000 1,027,000 4.3%
AT&T 15,746,000 15,719,000 27,000 0.2%
Verizon 6,958,000 6,959,000 (1,000) 0.0%
CenturyLink 5,435,000 5,662,000 (227,000) -4.0%
Cox 5,040,000 4,880,000 160,000 3.3%
Altice 4,096,300 4,046,200 50,100 1.2%
Frontier 3,802,000 3,938,000 (136,000) -3.5%
Mediacom 1,260,000 1,209,000 51,000 4.2%
Windstream 1,015,000 1,006,600 8,400 0.8%
Consolidated 781,912 783,682 (1,770) -0.2%
WOW! 755,100 730,000 25,100 3.4%
Cable ONE 660,799 524,935 135,864 25.9%
Cincinnati Bell 310,700 308,700 2,000 0.6%
97,662,811 95,056,435 2,123,694 2.2%

The large ISPs in the table control over 95% of the broadband market in the country. Not included in these numbers are the broadband customers served by the smaller ISPs – the telcos, WISPs, fiber overbuilders and municipalities. Cable companies continue to dominate the broadband market and now have 63.6 million customers compared to 34.0 million customers for the big telcos.

The 2.2% overall growth during the year is impressive since many have assumed that we are nearing the top of the market for broadband penetration. It’s worth noting that the US has had a housing construction boom and has added 1.6 million new housing units so far in 2018. If you assume those new homes share the same overall 85% market penetration as the rest of the country, the new homes would account for 1.36 million of the broadband gain. That means the rest of the market saw nearly a 1% overall increase in broadband penetration – a definite slowdown over prior years.

Much of the growth at the big cable companies continues to come at the expense of telco DSL. Overall, the big telcos lost a net of 328,370 customers for the year. This is mostly due to CenturyLink and Frontier, who are clearly bleeding DSL customers. The customer losses for these two companies is a bit surprising since by now each company should have activated big numbers of faster rural DSL customers, funded by the CAF II program. Companies are not required to report their performance for CAF II separately, and I have to wonder if many rural households are actually buying the improved rural broadband.

One thing that is clear about these numbers if that every company on the list ought to be considered now as an ISP, rather than as a telco or cable company. For this same 9-month period these same companies lost nearly 2.7 million cable customers while adding 2.1 million broadband customers. It’s clear that broadband is now the biggest and most important product for each of these companies.

Who WIll the Big ISPs Blame Now?

For the last few years the biggest ISPs have blamed regulations for reducing the amount of capital they are willing to invest. They specifically blamed Title II regulation of broadband and net neutrality rules as being a disincentive for them to invest in broadband infrastructure.

The FCC Chairman Ajit Pai adopted this same narrative and used it for justification to repeal net neutrality. He is still sticking to this story now that Title II regulation has been repealed and this week will be telling this story to the House Communications Subcommittee. In a prepared statement he claims that the repeal of the net neutrality rules is now paving the way for increased capital investment and better broadband service.

However, the whole narrative is false. There is no evidence that big ISPs held back on broadband investments before Title II authority was repealed and there is no evidence that the repeal has somehow unleashed a wave of new broadband investment. I’m not going to track the numbers in this blog, but the capital budgets of all of the big ISPs have been relatively steady for a number of years.

This particular narrative is just the latest iteration on a theme that the big ISPs have used for decades. The big ISPs have always publicly claimed that regulations were killing them, while privately admitting that they were able to successfully work around most regulations. It’s the nature of regulated industries to push back against regulation and ISPs don’t differ from the many other regulated industries in this regard.

A quick look at each of the major ISPs shows a different story than is being pushed by Chairman Pai. AT&T is a good example. They were required by an agreement from the purchase of DirecTV to pass 12 million homes and businesses with fiber. For a while it looks like they were shirking that requirement, but somewhere along the line they seem to have embraced it. They have been quietly extending fiber to apartment complexes and also to any homes or business that are located close to any of their many fiber nodes around the country. This expansion started well before the net neutrality repeal. AT&T for now has no plans to deploy 5G and says they don’t see a business case for it yet. The company is quietly walking away from rural copper and only beefing up rural cellular broadband where the FCC funded it with CAF II money.

Comcast doesn’t seem to have changed strategies for a number of years. They build fiber to shrink node sizes to relieve local network congestion. They made a decision well before the net neutrality appeal to embrace upgrades to DOCSIS 3.1. The company has entered the cellphone business, but for now resells minutes from the other cellular company networks, and only in their operating footprint. They say they plan to eventually build cellular networks to increase the profitability of the business.

Verizon has been shrinking their landline broadband networks and sold a pile of customers, including many on FiOS fiber to Frontier. The company made an announcement several years ago, and before the repeal of net neutrality that they were going to build new FiOS fiber in Boston – but it appears that project has largely been put on hold. Verizon might be the only big ISP who claims to have plans to expand residential broadband and says it will build 5G in a number of markets outside of its traditional footprint. But there is a lot of industry skepticism that this will be much larger trials of new technology and not a major capital outlay.

CenturyLink recently made it clear that they are walking away from making new broadband investments. They new CEO made it clear that the company will not be making any new capital expenditures that will earn infrastructure levels of returns. That is a 180-degree turnaround from a company that built fiber in 2017 to pass 900,000 premises and is the opposite of what Chairman Pai is claiming.

All of the big telcos have largely abandoned DSL and haven’t made new investments for years, even though there are faster DSL technologies available. To make matters worse the telcos are trying to kill the regulations from the Telecommunications Act of 1996 that allows competitors to offer faster DSL using telco copper – a move that would kick hundreds of thousands of customers nationwide off of decent broadband and force the back to the more expensive cable monopolies.

I can’t see any evidence from the big ISPs that the repeal of net neutrality made any difference in their capital spending plans. When you look at what these ISPs tell their investors the topic of regulation never arises – which it shouldn’t. The big ISPs have always invested in areas where they could foresee returns and regulation had no real negative impact on those returns. The whole false narrative has been a lobbying effort to get out from under regulation – and with this FCC the lobbying worked.

Now that Title II regulation is dead I wonder what the ISPs will blame for not investing in residential and rural broadband? They can’t point the finger any longer at regulations and I’m sure they will find a new story that sounds good. The only ISP that seems to be telling the truth is CenturyLink, and I suspect that they will soften that narrative since they are telling existing residential customers that they no longer care about them.

Who’s Pursuing Residential 5G?

I’ve seen article after article over the last year talking about how 5G is going to bring gigabit speeds to residents and give them an alternative to the cable companies. But most of the folks writing these articles are confusing the different technologies and businesses cases that are all being characterized as 5G.

For example, Verizon has announced plans to aggressively pursue 5G for commercial applications starting later this year. The technology they are talking about is a point-to-point wireless link, reminiscent of the radios that have been commonly used since MCI deployed microwave radios to disrupt Ma Bell’s monopoly. The new 5G radios use higher frequencies in the millimeter range and are promising to deliver a few gigabits of speed over distance of a mile or so.

The technology will require a base transmitter and enough height to have a clear-line-of-sight to the customer, likely sited on cell towers or tall buildings. The links are only between the transmitter and one customer. Verizon can use the technology to bring gigabit broadband to buildings not served with fiber today or to provide a second redundant broadband feed to buildings with fiber.

The press has often confused this point-to-point technology with the technology that will be used to bring gigabit broadband to residential neighborhoods. That requires a different technology that is best described as wireless local loops. The neighborhood application is going to require pole-mounted transmitters that will be able to serve homes within perhaps 1,000 feet – meaning a few homes from each transmitter. In order to deliver gigabit speeds the pole-mounted transmitters must be fiber fed, meaning that realistically fiber must be strung up each street that is going to get the technology.

Verizon says it is investigating wireless local loops and it hopes someday to eventually use the technology to target 30 million homes. The key word there is eventually, since this technology is still in the early stages of field trials.

AT&T has said that it is not pursuing wireless local loops. On a recent call with investors, CFO John Stevens said that AT&T could not see a business case for the technology. He called the business case for wireless local loops tricky and said that in order to be profitable a company would have to have a good grasp on who was going to buy service from each transmitter. He says that AT&T is going to stick to it’s current network plans which involve edging out from existing fiber and that serving customers on fiber provides the highest quality product.

That acknowledgement is the first one I’ve heard from one of the big telcos talking about the challenges of operating a widespread wireless network. We know from experience that fiber-to-the-home is an incredibly stable technology. Once installed it generally needs only minor maintenance and requires far less maintenance labor that competing technologies. We also know from many years of experience that wireless technologies require a lot more tinkering. Wireless technology is a lot more temperamental and it might take a decade or more of continuous tweaking until wireless local loop become as stable as FTTH. Whoever deploys the first big wireless local loop networks .better have a fleet of technicians ready to keep it working well.

The last of the big telcos as CenturyLink and their new CEO Jeff Storey has made it clear that the company is going to focus on high-margin enterprise business opportunities and will stop deploying slow-payback technologies like residential broadband. I think we’ve seen the end of CenturyLink investing in any last-mile residential technologies.

So who will be deploying 5G wireless local loops? We know it won’t be AT&T or CenturyLink. We know Verizon is considering it but has made no commitment. It won’t be done by the cable companies which have upgraded to DOCSIS 3.1. There are no other candidates that are willing or able to spend the billions needed to deploy the new technology.

Every new technology needs to be adopted by at least one large ISP to become successful. Vendors won’t do the needed R&D or crank up the production process until they have a customer willing to place a large order for electronics. We’ve seen promising wireless technologies like LMDS and MMDS die in the past because no large ISP embraced the technologies and ordered enough gear to push the technology into the mainstream.

I look at the industry today and I just don’t see any clear success path 5G wireless loop electronics. The big challenged faced by wireless local loops is to become less expensive than fiber-to-the-home. Until the electronics go through a few rounds of improvements that only come after field deployment, the technology is likely to require more technician time than FTTH. It’s hard to foresee anybody taking the chance on this in any grand way.

Verizon could make the leap of faith and sink big money into an untried technology, but that’s risky. We’re more likely to keep seeing press releases talking about field trials and the potential for the 5G technology. But unless Verizon or some other big ISP commits to sinking billions of dollars into the gear it’s likely that 5G local loop technology will fizzle as has happened to other wireless technologies in the past.

CenturyLink and Residential Broadband

CenturyLink is in the midst of a corporate reorganization that is going to result is a major shift in the focus of the company. The company merged with Level 3 in 2016 and the management team from Level 3 will soon be in charge of the combined business. Long-time CEO Glen Post is being pushed out of day-to-day management of the company and Jeff Storey, the former CEO of Level 3 will become the new CEO of CenturyLink. Storey was originally slated to take the top spot in 2019, but the transition has been accelerated and will happen this month.

It’s a shift that makes good financial sense for the company. Mr. Storey had huge success at Level 3 and dramatically boosted earnings and stock prices over the last four years. Mr. Storey and CenturyLink CFO Sunit Patel have both made it clear that they are going to focus on the more profitable enterprise business opportunities and that they will judge any investments in last-mile broadband in terms of the expected returns. This differs drastically from Mr. Post who comes from a background as an independent telephone company owner. As recently as a year ago Mr. Post publicly pledged to make the capital investments needed to improve CenturyLink’s last-mile broadband networks.

This is going to mean a drastic shift in the way that CenturyLink views residential broadband. The company lost 283,000 broadband customers for the year ending in December 2017, dropping them to 5.7 million broadband customers. The company blames the losses on the continued success of the cable companies to woo away DSL customers.

This size of the customer losses is a bit surprising. CenturyLink said at the end of 2017 that they were roughly 60% through their CAF II upgrades which is bringing better broadband to over 1.1 million rural households. Additionally, the company built FTTP past 900,000 potential business and residential customers in 2017. If the company was having even a modest amount of success with those two new ventures it’s hard to understand how they lost so many broadband customers.

What might all of this mean for CenturyLink broadband customers? For rural customers it means that any upgrades that are being made using CAF II funding are likely the last upgrades they will ever see. Customers in these rural areas are already used to being neglected and their copper networks are in lousy condition due to decades of neglect by former owner Qwest.

CenturyLink is required by the CAF II program to upgrade broadband speeds in the rural areas to at least 10/1 Mbps. The company says that over half of the upgraded customers are seeing speeds of at least twice that. I’ve always had a concern about any of the big telcos reaching the whole CAF II footprint, and I suspect that when the CAF II money is gone, anybody that was not upgraded as promised will never see upgrades. I’ve also always felt that the CAF II money was a waste of money –  if CenturyLink walks away from the cost of maintaining these newly upgraded DSL networks they will quickly slide back into poor condition.

There are already speculation on Wall Street that CenturyLink might try to find a buyer for their rural networks. After looking at the problems experienced by Frontier and Fairpoint after buying rural telco copper networks one has to wonder if there is a buyer for these properties. But in today’s world of big-deal corporate finance it’s not impossible to imagine some group of investors willing to tackle this. The company could also take a shot at selling rural exchanges to independent telcos – something US West did over twenty years ago.

It’s also likely that the company’s foray into building widespread FTTP in urban areas is done. This effort is capital intensive and only earns infrastructure returns that are not going to be attractive to the new management. I wouldn’t even be surprised to see the company sell off these new FTTP assets to raise cash.

The company will continue to build fiber, but with the emphasis on enterprise opportunities. They are likely to adopt a philosophy similar to AT&T’s which has been building residential fiber only to large apartment complexes and to households that are within short distances from existing fiber pops. This might bring fiber broadband to a lucky few, but mostly the new management team has made it clear they are deemphasizing residential broadband.

This management transition probably closes the book on CenturyLink as a last-mile ISP. If they are unable to find a buyer for these properties it might take a decade or more for their broadband business to quietly die. This is bad news for existing broadband customers because the company is unlikely to invest in keeping the networks in operational shape. They only ones who might perceive this as good news are those who have been thinking about overbuilding the company – they are not going to see any resistance.

When Big ISPs Fail

It’s obvious from reading the press that Frontier Communications is in trouble. The company visibly bungled the integration of the properties most recently purchased from Verizon, including some FiOS properties. The company was already experiencing customer losses, which have accelerated in the last year. Frontier is already looking to raise cash by finding a buyer for some of the properties they just purchased from Verizon.

I have no idea if Frontier is going to declare bankruptcy or fail. Watching them struggle, though, brings back memories of other big telcos that have struggled badly in the past. We’ve seen this scenario enough times to understand what poor performance will mean.

Not every telco that has struggled has gone through bankruptcy. Probably the best example of a company that almost went under, but which instead struggled for years was Qwest, which is now owned by CenturyLink. Within a few years after Qwest took over U.S. West the company fell on hard times. The company carried too much debt, and they didn’t do as well as expected in the long-line transport business that Qwest brought into the newly formed venture. The company was even fined $250 million by the Security and Exchange Commission for shady deals made with Enron’s broadband business.

We saw the consequences of Qwest’s financial struggles. They company had little money for capital and let the copper plant deteriorate a lot faster than would be expected. There were widespread reports of rural outages that were repeatedly patched rather than fixed while the company focused its limited resources on the major urban markets. Qwest lost huge numbers of broadband customers to the cable companies and also got clobbered in enterprise sales.

We saw something similar with Charter Communications. The company filed for bankruptcy protection in 2009. They pared back on capital spending and went for a number of years without making the upgrades we saw from Comcast, Cox and Mediacom. Much of the company’s footprint was stuck with first generation cable modems with slow broadband speeds.

Frontier looks to on a similar path to Fairpoint Communications after they purchased Verizon properties. Fairpoint took on massive debt to buy the New England properties from Verizon and struggled after adding 1.4 million customers to a relatively small company. Within two years after the purchase Fairpoint went through bankruptcy reorganization and continued to struggle since then due to lack of cash. They were recently purchased by Consolidated Communications.

What we’ve most learned from big ISPs that struggle is that the customers pay the price. All of these companies dealt with cash shortages by reducing staff and slashing capital expenditures. I remember Qwest staffing being reduced so much that there were entire rural counties that had only one Qwest technician. Qwest shuttered local business offices and lost the local touch in communities. Customers reported major delays in getting installations and repairs, with many reports of problems that were never solved.

We saw from Qwest and Charter that the first thing that goes in tight times is upgrades of technology. When those companies got into trouble they froze technology investment and innovation during a time when broadband speeds were climbing everywhere else.

The struggles of the big ISP invited in competition and many communities served by Qwest and Charter saw competitors build new networks. I know of some towns where the new competitors got practically every customer, showing how fed up customers were with being neglected by their big ISP. Unfortunately, the majority of communities served by such ISPs saw no competition and suffered with poor service.

Sometimes companies that struggle eventually right the ship. We see Charter now making upgrades that are a decade or more late. CenturyLink is under new management and is trying hard to make things better, but still doesn’t have enough capital to fix decades of neglect to the network. CenturyLink even got more than a billion dollar subsidy through the CAF II program to try to revitalize old rural copper. We’re going to have to wait to see if these big ISPs can make enough amends for communities to forgive them for decades of neglect.

My guess is that Frontier is not going to get the chance to reinvent themselves. They are struggling at a time when most of their rural communities are screaming for better broadband. It’s hard to imagine them somehow fixing their many problems.

Carrier-of-Last-Resort Obligations

Earlier this month the U.S. District Court for the District of Columbia upheld FCC orders that still require large telcos to be the carrier-of-last-resort provider of telephone service for at least some of their service territory. The ruling is the result of appeals made by CenturyLink and AT&T that required them to provide telephone service to new rural households.

The idea of carrier of last resort has been part of the telephone industry for nearly as long as the FCC has been regulating the industry. The concept was a key component of spreading the telephone network to all corners of the country – the Congress and the early FCC understood that the whole country was better off if everybody was connected.

Over the years the FCC and various state regulatory commissions ruled that telcos had to make a reasonable effort to connect rural customers. Telcos always had the option to petition against adding customers in really hard to reach places like mountaintops, but for the most part telcos routinely added new homes to the telephone network.

Carrier-of-last-resort started to weaken with the introduction of competition from the Telecommunications Act of 1996. Since that time the big telcos have been able to walk away from carrier-of-last-resort obligations in most of their territory. This court order ruled that in areas where the telcos are still receiving federal high cost support that the telcos are still obligated to connect homes that request service.

I worked for Ma Bell pre-divestiture and there was a real pride in the telephone industry that the network reached out to everybody. Telcos then also deployed huge numbers of pay telephones throughout the network to reach those that couldn’t afford phone service – even though they lost money on many of the payphones. The Bell company and the smaller independent telcos made it their mission to extend the network to everybody.

This order made a few comments, though, that puzzled me. They point out that many of the high-cost areas served by the big telcos are up for new funding from the upcoming CAF II auctions. Any winners of that auction are required to file to become the Eligible Telecommunications Carrier (ETC) for any areas they receive funding. The discussion in the court order implies that these new ETCs will become the carrier-of-last-resort in these areas.

That surprised me because there are plenty of carriers that have ETC status and yet are not the obligated carrier-of-last-resort. The best example is the same big telcos examined in this case who are the ETC of record for their whole footprints but now only have carrier-of-last-resort obligations for the last most rural areas covered by this case. There have been stories for years of people who built new homes, even in urban areas, and are refused service by both the telco and cable company. The cable companies have no carrier-of-last-resort obligations, but it’s clear that in many places the telcos have been able to walk away from the obligation.

I think that companies seeking the CAF II reverse auction funding might be surprised by this interpretation of the rules. Being carrier-of-last-resort means that a carrier is obligated to build to reach anybody in the covered area that requests telephone service. The reverse auction doesn’t even require total coverage of the covered census blocks and that seems to be in conflict with the court’s interpretation. The reverse auction census blocks are some of the most sparsely populated areas of the country and building to even one remote customer in some of these areas could be extremely expensive.

Unfortunately, the carrier-of-last-resort obligation only applies to telephone service and not to broadband. It would be nice to see this concept applied to broadband and the FCC missed a good opportunity to do this when they handed out billions of federal dollars in the CAF II plan. With that plan the big telcos are only required to make their best effort to reach customers with broadband in the areas that got the CAF II funding – I’m hearing from rural people all over the country that a lot of the CAF II areas aren’t seeing any upgrades. For the most part the idea of carrier-of-last resort and universal coverage are becoming quaint concepts of our past.

Convergence

Even a decade ago it was apparent that the telecom industry was headed towards convergence. By that, I mean that the various cable companies, telcos and wireless companies are expanding service lines and are starting to compete with each other in areas that were unimaginable even a few years ago.

Comcast is the best example of this. Their CEO Brian Roberts was quoted last year as saying that the company was now in all of the business lines available to it. Compare today’s Comcast with the company a decade ago. Then they were just becoming a triple play provider by adding voice to their product line-up. Since then they have added a lot more business lines.

A decade ago Comcast barely went after business customers and didn’t even own network in many business districts and industrial parks – and now they are a major provider of business services. The company recently added cellular service and it appears they are adding customers at a furious pace. They are becoming a major player in home security. The company has a thriving product line selling residential smart home services. They even started bundling home solar panels with their residential product line recently.

The company has even stepped up their traditional cable service to do better against the satellite providers. They’ve developed their own settop box that is said to be the best in the industry. And they have bought a number of the cable programmers like NBC, giving them a margin advantage over any competitor for video.

It seems to me like everybody else wants to be Comcast. Consider AT&T. A decade ago they were a traditional telco. They operated a huge copper network for residential broadband and telephone service and owned the country’s largest fiber network for providing wholesale transport and business services. They were also one of the two largest cellular companies, and with Verizon controlled the vast majority of that business.

AT&T not only added cable TV service to their product line, but they bought DirecTV and become a major video provider. They are trying hard to buy programming and content by merging with Time Warner. The company has been aggressive building fiber to large apartment complexes and has become a major player in the MDU market that used to be almost exclusively controlled by the cable incumbents. The company has also been building a lot of fiber to better compete head-to-head with Comcast and other cable companies that have faster residential broadband.

Verizon took a different path and competed head-to-head with Comcast in the northeast even a decade ago with its FiOS fiber network. The company continues to buy smaller regional fiber providers like XO to beef up its business and fiber networks. Verizon has announced that it intends to roar back into the residential market by use of small cell 5G over the next decade. And Verizon continues to thrive as a cellular carrier.

Even smaller companies like CenturyLink are looking a lot like their bigger competitors. The company had added cable to its bundle. They built fiber past almost a million passings last year to provide more robust competition for broadband speeds. And they bought Level 3 to become a major player for transport and business services.

But these big ISPs are not the only ones crossing into new product lines. Consider T-Mobile. In a move that was unthinkable even a few years ago they are making a major play to bundle video content with their cellular service – making them a direct competitor of all of the ISPs for the market segment of folks who are happy with mobile video rather than a landline connection. T-Mobile is pushing the other cellular providers to do the same.

And there are other national competitors on the horizon. For example, there are several satellite companies like SpaceX and OneWeb that are likely to compete nationally with bundles similar to the other ISPs. I also think we’ll see new competitors spring up and compete with 5G last-mile networks as that technology matures.

It’s going to be interesting to see the winners and losers over the next decade. Right now the cable companies are approaching a near monopoly in many markets for broadband. The only way these other competitors are going to survive and thrive is to chop away at Comcast and the other large cable companies. But at the same time the cable companies will be carving cellular customers. For those like me who follow the industry it’s going to be interesting to watch.

4Q 2017 National Broadband Statistics

As a nation we are approaching an 85% overall penetration of residential broadband. The following statistics come from the latest report from the Leichtman Group and compares broadband customers at the end of 2017 to the end of 2016.

 4Q 2017 4Q 2016 Change
Comcast 25,869,000 24,701,000 1,168,000 4.7%
Charter 23,903,000 22,593,000 1,310,000 5.8%
AT&T 15,719,000 15,605,000 114,000 0.7%
Verizon 6,959,000 7,038,000 (79,000) -1.1%
CenturyLink 5,662,000 5,945,000 (283,000) -4.8%
Cox 4,880,000 4,790,000 90,000 1.9%
Altice 4,046,200 3,962,500 83,700 2.1%
Frontier 3,938,000 4,271,000 (333,000) -7.8%
Mediacom 1,209,000 1,162,000 47,000 4.0%
Windstream 1,006,600 1,051,100 (44,500) -4.2%
WOW! 730,000 718,900 11,100 1.5%
Cable ONE 524,935 513,908 11,027 2.1%
Cincinnati Bell 308,700 303,200 5,500 1.8%
Fairpoint 301,000 306,624 (5,624) -1.8%
95,056,435 92,961,232 2,095,203 2.3%

These large ISPs control over 95% of the broadband market in the country – so looking at them provides a good snapshot of the industry. Not included in these numbers are the broadband customers of the smaller ISPs, the subscribers of WISPs (wireless ISPs) and customers of the various satellite services. Cable companies still dominate the broadband market and have 61.2 million customers compared to 33.9 million customers for the big telcos.

These overall numbers don’t tell the whole story since there is a lot of broadband upgrades happening around the country. Cable companies are starting to upgrade to DOCSIS 3.1 which will mean even faster download speeds. A number of telcos are building fiber to replace DSL, which they are hoping will stop the erosion of DSL customers fleeing to cable companies. There also is continued mergers in the industry, but Leichtman numbers include the impact from mergers into the prior year’s numbers – which is why the table of 2016 customer won’t match up with the same Leichtman table from a year ago.

It’s obvious that the cable companies are still taking DSL customers away from telcos in the cities of America. The cable companies added 2.7 million customers last year, and a significant percentage of these are from DSL customers converting to cable modems. But the telcos as a whole lost only 626,000 customers as a group. What accounts for this difference?

Part of the answer has to be the FCC’s CAF II program. The telcos were give money to bring broadband to over 4 million rural households and the telcos all report that they’ve finished some portion of that buildout by the end of 2017. But 2017 was only the second of a six year program and we should be seeing more broadband customers for AT&T, CenturyLink, Frontier and the other big telcos starting with the 2018 statistics.

The company that surprises me the most is AT&T. They told the FCC and stockholders that they had built fiber to reach 4 million new customers by the end of 2017, out of their goal of 12 million new potential new customers. This goal of 12 million new fiber passings was a condition of their merger with DirecTV. Perhaps they lost a mountain of DSL customers – but you would expect between CAF II and the new fiber builds to see an increase in AT&T broadband subscribers.

CenturyLink is also a bit of a puzzle. They say that they built fiber past 900,000 passings last year and I would have expected to see some benefit from that effort. Granted, they would have converted a lot of DSL customers to fiber, but you would expect them to finally have a competitive advantage with fiber compared to cable networks. And they also claim a substantial rural build from CAF II.

Probably the most important thing about broadband growth in 2017 is that is 30% less than the growth in 2016. We are finally seeing broadband reach it’s peak, and if the rate of growth continues on that curve we are only a few years away from reaching broadband equilibrium where the only broadband growth will come from new housing units entering the market. That will be the point when we will see the big ISPs panic and start struggling to find ways to satisfy Wall Street’s demand for ever-continuing profits.