Shame on the Regulators

It’s clear that even before the turn of this century that the big telcos largely walked away from maintaining and improving residential service. The evidence for this is the huge numbers of neighborhoods that are stuck with older copper technologies that haven’t been upgraded.  The telcos made huge profits over the decades in these neighborhoods and ideally should not have been allowed to walk away from their customers.

In the Cities. Many neighborhoods in urban areas still have first or second-generation DSL over copper with fastest speeds of 3 Mbps or 6 Mbps. That technology had a shelf-life of perhaps seven years and is now at least fifteen years old.

The companies that deployed the most DSL are AT&T and CenturyLink (formerly Quest). The DSL technology should have been upgraded over time by plowing profits back into the networks. This happened in some neighborhoods, but as has been shown in several detailed studies in cities like Cleveland and Dallas, the faster DSL was brought to more affluent neighborhoods, leaving poorer neighborhoods, even today, with the oldest DSL technology.

The neighborhoods that saw upgrades saw DSL speeds between 15 Mbps and 25 Mbps. Many of these neighborhoods eventually saw speeds as fast as 50 Mbps using a technology that bonded two 25 Mbps DSLs circuits. There are numerous examples of neighborhoods with 50 Mbps DSL sitting next to ones with 3 Mbps DSL.

Verizon used a different tactic and upgraded neighborhoods to FiOS fiber. But this was also done selectively although Verizon doesn’t seem to have redlined as much as AT&T, but instead built FiOS only where the construction cost was the lowest.

In Europe, the telcos decided to complete with the cable companies and have upgraded DSL over time, with the fastest DSL today offering speeds as fast as 300 Mbps. There is talk coming out of DSL vendors talking about ways to goose DSL up to gigabit speeds (but only for short distances). The telcos here basically stopped looking at better DSL technology after the introduction of VDSL2 at least fifteen years ago.

By now the telcos should have been using profits to build fiber. AT&T has done this using the strategy of building little pockets of fiber in every community near to existing fiber splice points. However, the vast majority of rural households served by AT&T are not being offered fiber, and AT&T said recently that they have no plans to build more fiber. CenturyLink built fiber to past nearly 1 million homes a few years ago, but that also seems like a dead venture going forward. But now, in 2019, each of these telcos should have been deep into urban neighborhoods in their whole service area with fiber. Had they done so they would not be getting clobbered so badly by the cable companies that are taking away millions of DSL customers every year.

Rural America. The big telcos started abandoning rural America as much as thirty years ago. They’ve stopped maintaining copper and have not voluntarily made any investments in rural America for a long time. There was a burst of rural construction recently when the FCC gave them $11 billion to improve rural broadband to 10/1 Mbps – but that doesn’t seem to be drawing many rural subscribers.

It’s always been a massive challenge to bring the same speeds to rural America that can be provided in urban America. This is particularly so with DSL since the speeds drop drastically with distance. DSL upgrades that could benefit urban neighborhoods don’t work well in farmland. But the telcos should have been expanding fiber deeper into the network over time to shorten loop lengths. Many independent telephone companies did this the right way and they were able over time to goose rural DSL speeds up to 25 Mbps.

The big telcos should have been engaging in a long-term plan to continually shorten rural copper loop lengths. That meant building fiber, and while shortening loop lengths they should have served households close to fiber routes with fiber. By now all of the small towns in rural America should have gotten fiber.

This is what regulated telcos are supposed to do. The big telcos made vast fortunes in serving residential customers for many decades. Regulated entities are supposed to roll profits back into improving the networks as technology improves – that’s the whole point of regulating the carrier of last resort.

Unfortunately, the industry got sidetracked by competition from CLECS. This competition first manifested in competition for large business customers. The big telcos used that competition to convince regulators they should be deregulated. Over time the cable companies provided real residential competition in cities, which led to the de facto total deregulation of telcos.

In Europe, the telcos never stopped competing in cities because regulators didn’t let them quit. The telcos have upgraded to copper speeds that customers still find attractive, but the telcos all admit that the next upgrade needs to be fiber. In the US, the big telcos exerted political pressure to gain deregulation at the first hint of competition. US telcos folded and walked away from their customers rather than fighting to maintain revenues.

Rural America should never have been deregulated. Shame on every regulator in every state that voted to deregulate the big telcos in rural America. Shame on every regulator that allowed companies like Verizon palm off their rural copper to companies like Frontier – a company that cannot succeed, almost by definition.

In rural America the telcos have a physical network monopoly and the regulators should have found ways to support rural copper rather than letting the telcos walk away from it. We know this can be done by looking at the different approaches taken by the smaller independent telephone companies. These small companies took care of their copper and most have now taken the next step to upgrade to fiber to be ready for the next century.

CenturyLink Embraces Fiber

CenturyLink seems to have done a 180 in terms of embracing fiber. According to Jeff Storey, the CEO of Centurylink, the company is now defining itself as the ‘go-to provider’ for fiber-based services for business customers. This is in sharp contrast to just a year ago when Storey, as the new CEO said that the company would not be pursuing low-return infrastructure investments.

Storey says that CenturyLink added 5,000 business buildings to fiber in the second quarter, following 4,500 buildings in the first quarter of this year. This contrasts to Level 3 that historically added around 500 buildings per quarter.

It appears the company may be taking a page from the AT&T storybook. AT&T has been building fiber around locations where it already has a fiber POP. This strategy has helped AT&T to now pass over 20 million locations with fiber while avoiding the high cost of large overbuilds.

CenturyLink has an extensive national fiber footprint that it’s accumulated from the purchase over the years of Level 3, Qwest, Broadwing and WilTel. Like AT&T discovered, CenturyLink is sitting close to a huge number of existing opportunities with that existing network, and perhaps that’s the new company strategy – to edge-out and take advantage of nearby low-hanging fruit.

Storey says the company has ordered 4.7 million miles of fiber to add into its urban networks. You have to take that number with a grain of salt since one mile of a 48-strand fiber counts as 48 miles of fiber when counted this way. But this is still a lot of new fiber construction. The one thing that readers of this blog will notice is that the construction is likely to be urban – it’s doubtful that the company is ever going to put another dollar into rural infrastructure. The company recently quietly searched around for the possibility of spinning off its rural business but found no takers. This is likely to mean more of the same for its rural customers – mostly neglect.

The company continues to lose broadband customers. In 2018 the company lost 262,000 broadband customers for a 4.6% drop. In the second quarter of this year, the company lost 56,000 net broadband customers but reports that it lost 78,000 customers with speeds below 20 Mbps and added 22,000 customers with speeds faster than that. Like all of the big telcos, it’s losing DSL customers converting to cable modems.

The company has a long way to go to convince Wall Street that its stock has value. In a May blog, I wrote how the company stock had dropped 43% over a year to a price of $10.89. It’s still sitting in that range with the stock price sitting at $11.56 yesterday.

The Level 3 acquisition is likely to be one of the more interesting stories in the history of our industry. Level 3 was the high-flying telecom company, with stock prices that climbed steadily. It seems that Jeff Storey could do no wrong as earnings grew faster than the rest of the sector. But that all came to a halt with the merger with CenturyLink. I’m sure that both companies thought that Storey could pull CenturyLink upward by tying it to Level 3, but just the opposite occurred. This might be one of the biggest cautionary tales ever in our industry and shows how difficult, and perhaps impossible it is for anybody to turn around one of the big incumbent telcos.

What’s most interesting in this story is that Glen Post and the crew from CenturyLink were in the process of that a slow and steady turnaround. A few years before the CenturyLink merger the company had decided to build residential fiber in its many large city markets and take back a significant piece of the broadband business that had been leaking away. In the year before the Level 3 merger the company had built fiber past nearly 1 million urban passings. We’ll never know now if a few more years of that kind of investment could have turned the company around – not to be a high-flyer like Level 3, but to earn decent long-term returns on broadband infrastructure – the kind that come from making a hundred-year investment in fiber.

What’s the Future for CenturyLink?

I don’t know how many of you watch industry stock prices. I’m certainly not a stock analyst, but I’ve always tracked the stock prices of the big ISPs as another way to try to understand the industry. The stock prices for big ISPs are hard to compare because every big ISP operates multiple lines of business these days. AT&T and Verizon are judged more as cellular companies than as ISPs. AT&T and Comcast stock prices reflect that both are major media companies.

With that said, the stock price for CenturyLink has performed far worse than other big ISPs over the last year. A year ago a share of CenturyLink stock was at $19.24. By the end of the year the stock price was down to $15.44. As I wrote this blog the price was down to $10.89. That’s a 43% drop in share price over the last year and a 30% drop since the first of the year. For comparison, following are the stock prices of the other big ISPs and also trends in broadband customers:

Stock Price 1 Year Ago Stock Price Now % Change 2018 Change in Broadband Customers
CenturyLink $19.24 $10.89 -43.4% -262,000
Comcast $32.14 $43.15 34.3% 1,353,000
Charter $272.84 $377.89 38.5% 1,271,000
AT&T $32.19 $30.62 -4.9% -18,000
Verizon $48.49 $56.91 17.4% 2,000

As a point of comparison to the overall market, the Dow Jones Industrial average was up 4% over this same 1-year period. The above chart is not trying to make a correlation between stock prices and broadband customers since that is just one of dozens of factors that affect the performance of these companies.

Again, I’ve never fully understood how Wall Street values any given company. In reading analyst reports on CenturyLink it seems that the primary reason for the drop in stock price is that all of the company’s business units are trending downward. In the recently released 1Q 2019 results the company showed a year-over-year drop in results for the international, enterprise, small and medium business, wholesale, and consumer business units. It seems that analysts had hoped that the merger with Level 3 would reverse some of the downward trends. Stock prices also dropped when the company surprised the market by cutting its dividend payment in half in February.

CenturyLink faces the same trends as all big ISPs – traditional business lines like landline telephone and cable TV are in decline. Perhaps the most important trend affecting the company is the continued migration of broadband customers from copper-based DSL to cable company broadband. CenturyLink is not replacing the DSL broadband customers it’s losing. In 2018 CenturyLink lost a lot of broadband customers with speeds under 20 Mbps, but had a net gain of customers using more than 20 Mbps. CenturyLink undertook a big fiber-to-the-home expansion in 2017 and built fiber to pass 900,000 homes and businesses – but currently almost all expansion of last-mile networks is on hold.

It’s interesting to compare CenturyLink as an ISP with the big cable companies. The obvious big difference is the trend in broadband customers and revenues. Where CenturyLink lost 262,000 broadband customers in 2018, the two biggest cable companies each added more than a million new broadband customers for the year. CenturyLink and other telcos are losing the battle of DSL versus cable modems with customers migrating to cable companies as they seek faster speeds.

It’s also interesting to compare CenturyLink to the other big telcos. From the perspective of being an ISP, AT&T and Verizon are hanging on to total broadband customers. Both companies are also losing the DSL battle with the cable companies, but each is adding fiber customers to compensate for those losses. Both big telcos are building a lot of new fiber, mostly to provide direct connectivity to their own cell sites, but secondarily to then take advantage of other fiber opportunities around each fiber node.

Verizon has converted over a hundred telephone exchanges in the northeast to fiber-only and is getting out of the copper business in urban areas. Verizon has been quietly filling in its FiOS fiber network to cover the copper it’s abandoning. While nobody knows yet if it’s real, Verizon also has been declaring big plans to to expand into new broadband markets markets using 5G wireless loops.

AT&T was late to the fiber game but has been quietly yet steadily adding residential and business fiber customers over the last few years. They have adopted a strategy of chasing pockets of customers anywhere they own fiber.

CenturyLink had started down the path to replace DSL customers when they built a lot of fiber-to-the-home in 2017. Continuing with fiber construction would have positioned the company to take back a lot of the broadband market in the many large cities it serves. It’s clear that the new CenturyLink CEO doesn’t like the slow returns from investing in last-mile infrastructure and it appears that any hopes to grow the telco part of the business are off the table.

Everything I read says that CenturyLink is facing a corporate crisis. Diving stock prices always put strain on a company. CenturyLink faces more pressure since the activist investors group Southeastern Asset Management holds more than a 6% stake in CenturyLink and made an SEC filing that that the company’s fiber assets are undervalued.

The company has underperformed compared to its peers ever since it was spun off from AT&T as US West. The company then had what turned out to be a disastrous merger with Qwest. There was hope a few years back that the merger with CenturyLink would help to right the company. Most recently has been the merger with Level 3, and at least for now that’s not made a big difference. It’s been reported that CenturyLink has hired advisors to consider if they should sell or spin off the telco business unit. That analysis has just begun, but it won’t be surprising to hear about a major restructuring of the company.

Access to Low-Price Broadband

The consumer advocate BroadbandNow recently made an analysis of broadband prices across the US and came up with several conclusions:

  • Broadband prices are higher in rural America.
  • They conclude that 45% of households don’t have access to a ‘low-priced plan’ for a wired Internet connection.

They based their research by looking at the published prices of over 2,000 ISPs. As somebody who does that same kind of research in individual markets, I can say that there is often a big difference between published rates and actual rates. Smaller ISPs tend to charge the prices they advertise, so the prices that BroadbandNow found in rural America are likely the prices most customers really pay.

However, the big ISPs in urban areas routinely negotiate rates with customers and a significant percentage of urban broadband customers pay something less than the advertised rates. But the reality is messier even than that since a majority of customers still participate in a bundle of services. It’s usually almost impossible to know the price of any one service inside a bundle and the ISP only reveals the actual rate when a customer tries to break the bundle to drop one of the bundled services. For example, a customer may think they are paying $50 for broadband in a bundle but find out their real rate is $70 if they try to drop cable TV. These issues make it hard to make any sense out of urban broadband rates.

I can affirm that rural broadband rates are generally higher. A lot of rural areas are served by smaller telcos and these companies realize that they need to charge higher rates in order to survive. As the federal subsidies to rural telcos have been reduced over the years these smaller companies have had to charge realistic rates that match their higher costs of doing business in rural America.

I think rural customers understand this. It’s a lot more expensive for an ISP to provide broadband in a place where there are only a few customers per road-mile of network than in urban areas where there might be hundreds of customers per mile. A lot of other commodities cost more in rural America for this same reason.

What this report is not highlighting is that the lower-price broadband in urban areas is DSL. The big telcos have purposefully priced DSL below the cost of cable modem broadband as their best strategy to keep customers. When you find an urban customer that’s paying $40 or $50 for broadband it’s almost always going to be somebody using DSL.

This raises the question of how much longer urban customers will continue to have the DSL option. We’ve already seen Verizon abandon copper-based products in hundreds of urban exchanges in the last few years. Customers in those exchanges can theoretically now buy FiOS on fiber – and pay more for the fiber broadband. This means for large swaths of the northeast urban centers that the DSL option will soon be gone forever. There are persistent industry rumors that CenturyLink would like to get out of the copper business, although I’ve heard no ideas of how they might do it. It’s also just a matter of time before AT&T starts walking away from copper. Will there even be any urban copper a decade from now? Realistically, as DSL disappears with the removal of copper the lowest prices in the market will disappear as well.

There is another trend that impacts the idea of affordable broadband. We know that the big cable companies now understand that their primary way to keep their bottom line growing is to raise broadband rates. We’ve already seen big broadband rate increases in the last year, such as the $5 rate increase from Charter for bundled broadband.

The expectation on Wall Street is that the cable companies will regularly increase broadband rates going into the future. One analyst a year ago advised Comcast that basic broadband ought to cost $90. The cable companies are raising broadband rates in other quieter ways. Several big cable companies have told their boards that they are going to cut back on offering sales incentives for new customers and they want to slow down on negotiating rates with existing customers. It would be a huge rate increase for most customers if they are forced to pay the ‘list’ prices for broadband.

We also see carriers like Comcast starting to collect some significant revenues for customers going over the month data caps. As household broadband volumes continue to grow the percentage of people using their monthly cap should grow rapidly. We’ve also seen ISPs jack up the cost of WiFi or other modems as a backdoor way to get more broadband revenue.

As the cable companies find way to extract more revenue out of broadband customers and as the big telcos migrate out of DSL my bet is that by a decade from now there will be very few customers with ‘affordable’ broadband. Every trend is moving in the opposite direction.

Broadband Statistics 4Q 2018

The Leichtman Research Group has published the statistics of broadband subscribers for the largest ISPs for the year ending December 31, 2018. Following compares the end of 2018 to the end of 2017.

 4Q 2018 4Q 2017 Change
Comcast 27,222,000 25,869,000 1,353,000 5.2%
Charter 25,259,000 23,988,000 1,271,000 5.3%
AT&T 15,701,000 15,719,000 (18,000) -0.1%
Verizon 6,961,000 6,959,000  2,000 0.0%
CenturyLink 5,400,000 5,662,000 (262,000) -4.6%
Cox 5,060,000 4,960,000 100,000 2.0%
Altice 4,118,100 4,046,000 71,900 1.8%
Frontier 3,735,000 3,938,000 (203,000) -5.2%
Mediacom 1,260,000 1,209,000 55,000 4.5%
Windstream 1,015,000 1,006,600 8,400 0.8%
Consolidated 778,970 780,794 (1,824) -0.2%
WOW! 759,600 732,700 26,900 3.7%
Cable ONE 663,074 643,153 19,921 3.1%
Cincinnati Bell 311,000 308,700 2,300 0.7%
98,247,744 95,822,147 2,425,597 2.5%

The large ISPs in the table control over 95% of the broadband market in the country. Not included in these numbers are the broadband customers served by the smaller ISPs – the telcos, WISPs, fiber overbuilders and municipalities.

The biggest cable companies continue to dominate the broadband market and now have 64.3 million customers compared to 33.9 million customers for the big telcos. During 2018 the big cable companies collectively added 2.9 million customers while the big telcos collectively lost 472,000 customers.

What is perhaps most astounding is that Comcast and Charter added 2.6 million customers for the year while the total broadband market for the biggest ISPs grew by only 2.5 million. For years it’s been obvious that the big cable companies are approaching monopoly status in metropolitan areas and these statistics demonstrate how Comcast and Charter, in particular, have a stranglehold over competition in their markets.

CenturyLink and Frontier are continuing to bleed DSL customers. Together the two companies lost 465,000 broadband customers in 2018, up from a loss for the two of 343,000 in 2017.

It’s always hard to understand all of the market forces behind these changes. For example, all of the big cable companies are seeing at least some competition from fiber overbuilders in some of their markets. It would be interesting to know how many customers each is losing to fiber competition.

I’d also love to know more about how the big companies are faring in different markets. I suspect that the trends for urban areas are significantly different than in smaller markets. I know that deep data analysis of the FCC’s 477 data might tell that story. (hint, hint in case anybody out there wants to do that analysis!)

I’m also curious if the cable companies are seeing enough bottom-line improvement to justify the expensive upgrades to DOCSIS 3.1. Aside from Comcast and Charter I wonder how companies like Cox, Mediacom and Cable ONE justify the upgrade costs. While those companies are seeing modest growth in broadband customers, each is also losing cable customers, and I’d love to understand if the upgrades are cost-justified.

If there is any one takeaway from these statistics it’s that we still haven’t reached the top of the broadband market. I see articles from time to time that predict that younger households are going to bail on landline broadband in favor of cellular broadband. But seeing that over 2.4 million households added broadband in the last year seems to be telling a different story.

Where’s the CAF II Success?

If you’ve read this blog you know I’ve been a big critic of the FCC’s CAF II program that gave over $10 billion in federal subsidies to the biggest telcos to improve rural broadband. My complaint is that the program set the embarrassingly low goal of improving rural broadband to speeds of at least 10/1 Mbps. My complaint is that this money could have done a huge amount of good had it been put up to reverse auction as was done with the leftover customers from this program last year – many ISPs would have used this funding to help to build rural fiber. Instead, the telcos are using the money mostly to upgrade DSL.

While I think the program was ill-conceived and was a giveaway to the big telco lobbyists, I am at least glad that it is improving rural broadband. For a household with no broadband, a 10 Mbps product might provide basic access to broadband services for the first time. We are now into the fifth year of the six-year program, so we ought to be seeing the results of these upgrades. USTelecom just published a blog saying that deployments are ahead of schedule and that CAF II is a quiet success.

The telcos have told the FCC they are largely on track – by the end of 2018 they should have upgraded broadband for at least 60% of the required households. AT&T and Windstream report that they have made at least 60% of the needed upgrades everywhere. Frontier says they are on track in 27 of the 29 states needing upgrades. CenturyLink says they are on track in only 23 of 33 states that are getting CAF II upgrades. According to USTelecom, over 2.1 million households should now be seeing faster speeds.

It’s also worth noting that the CAF II program should improve broadband for many more households that are not covered directly by the program. For example, when upgrading DSL for a CAF II area that surrounds a town, those living in the town should also see better broadband. The secondary benefit of the CAF program is that rural towns should be seeing speeds increasing from 6 Mbps or slower to as fast as 25 Mbps. By now many more millions of households should be seeing faster broadband due to CAF II.

What I find puzzling is that I would expect to see an upward burst of broadband customers for the big telcos because of CAF II. But the numbers aren’t showing that. There were four telcos that accepted more than $1 billion from the program, as follows, and three of them lost broadband customers in 2018:

Funding Households Per Household 2018 Broadband Customers
CenturyLink $3.09 B 1,190,016 $2,593 (262,000)
AT&T $2.96 B 1,265,036 $2,342 (18,000)
Frontier $1.7 B 659,587 $2,578 (203,000)
Windstream $1.07 B 413,345 $2,595 8,400
Total CAF II $10.05 B 4,075,840 $2,467

Windstream is the only telco of the four that gained customers last year. Windstream’s footprint is probably the most rural of the four telcos. We know that every telco is losing the battle for customers in towns where cable companies are increasing speeds on coaxial networks. Windstream seems to be offsetting those losses, and I can conjecture it’s because they have been selling more rural broadband.

AT&T is in a category all by itself. It’s impossible to know how AT&T is faring with CAF II. They are largely implementing CAF II using their cellular network (with the goal of tearing down rural copper). The company has also been deploying fiber past millions of homes and business in urban areas. They are clearly losing the residential broadband battle in urban markets to companies like Comcast and Charter. However, I can tell you anecdotally that AT&T hasn’t given up on urban copper. They have knocked on my door in Asheville, NC at least three times in the last year trying to sell DSL. I have to assume that they are also marketing broadband improvements in rural areas.

CenturyLink and Frontier are clearly bleeding broadband customers and each lost over 200,000 customers just in the last year. I have to wonder how hard these companies are marketing improved rural broadband. Both companies work in urban and suburban markets but also in numerous county seats situated in rural counties. Like every telco they are losing DSL customers in these markets to the cable company competitors.

Just like I have anecdotal evidence that AT&T is still pushing copper I hear stories that say the opposite for CenturyLink and Frontier. I worked in a few rural counties last year where the CAF II upgrades were reported as complete. And yet the communities seemed unaware of the improvements. Local politicians who bear the brunt of complaints from households that want better broadband weren’t aware of any upgrades – which tells me their rural constituents weren’t aware of upgrades.

I honestly don’t know what this all means. I really expected to find more positive evidence of the impact of CAF II. From what I know of rural America, households ought to leap at the opportunity to buy 10/1 Mbps DSL if they’ve had no broadband in the past. Are the upgrades being done but not being followed up with a marketing and public awareness campaign? Are actual upgraded speed not meeting the 10/1 Mbps goal? Are the upgrades really being made as reported to the FCC? We’re perhaps a year and a half away from the completion of CAF II, so I guess we’ll find out soon enough.

The Status of the CAF II Deployments

The Benton Foundation noted last month that both CenturyLink and Frontier have not met all of their milestones for deployment of CAF II. This funding from the FCC is supposed to be used to improve rural broadband to speeds of at least 10/1 Mbps. As of the end of 2018, the CAF II recipients were to have completed upgrades to at least 60% of the customers in each state covered by the funding.

CenturyLink took funding to improve broadband in 33 states covering over 1 million homes and businesses. CenturyLink claims to have met the 60% milestone in twenty-three states but didn’t make the goal in eleven states: Colorado, Idaho, Kansas, Michigan, Minnesota, Missouri, Montana, Ohio, Oregon, Washington, and Wisconsin.

Frontier received CAF II funding to improve broadband to over 774,000 locations in 29 states. Frontier says they’ve met the milestone in 27 states but haven’t reached the 60% deployment milestone in Nebraska and New Mexico.  There were a number of other large telcos that took CAF Ii funding like AT&T, Windstream, and Consolidated, and I have to assume that they’ve reported meeting the 60% milestone.

Back in 2014 when it looked like the CAF II program might be awarded by reverse auction, we helped a number of clients take a look at the CAF II service areas. In many cases, these are large rural areas that cover 50% or more of most of the rural counties in the country. Most of my clients were interested in the CAF II money as a funding mechanism to help pay for rural fiber, but all of the big telcos other than AT&T announced originally that they planned to upgrade existing DSL. AT&T announced a strategy early on to used fixed cellular wireless to satisfy their CAF II requirements. Since then a few big telcos like Frontier and Windstream have said that they are also using fixed wireless to meet their obligations.

To us, the announcement that the telcos were going to upgrade DSL set off red flag alarms. In a lot of rural counties there are only a small number of towns, and those towns are the only places where the big telcos have DSLAMs (the DSL hub). Rural telephone exchanges tend to be large and the vast majority of rural customers have always been far out of range of DSL that originates in the small towns. One only has to go a few miles – barely outside the towns – to see DSL speeds fall off to nothing.

The only way to make DSL work in the CAF II areas would be to build fiber to rural locations and establish new DSL hub sites. As any independent telco can tell you who deployed DSL the right way, this is expensive because it takes a lot of the rural DSLAMs to get within range of every customer. By electing DSL upgrades, the big telcos like CenturyLink and Frontier had essentially agreed to build a dozen or more fiber DSLAMs in each of the rural counties covered by CAF II. My back-of-the-envelope math showed that was going to cost a lot more than what the companies were receiving from the CAF fund. Since I knew these telcos didn’t want to spend their own money in rural America, I predicted execution failures for many of the planned DSL deployments.

I believe the big telcos are now facing a huge dilemma. They’ve reached 60% of customers in many places (but not all). However, it is going to cost two to three times more per home to reach the remaining 40% of homes. The remaining customers are the ones on extremely long copper loops and DSL is an expensive technology use for reaching these last customers. A DSLAM built to serve the customers at the ends of these loops might only serve a few customers – and it’s hard to justify the cost of the fiber and electronics needed to reach them.

I’ve believed from the beginning that the big telcos building DSL for the CAF II program would take the approach of covering the low hanging fruit – those customers that can be reached by the deployment of a few DSLAMs in a given rural area. If that’s true, then the big telcos aren’t going to spend the money to reach the most remote customers, meaning a huge number of CAF II customers are going to see zero improvements in broadband. The telcos mostly met their 60% targets by serving the low-hanging fruit. They are going to have a huge challenge meeting the next milestones of 80% and 100%.

Probably because I write this blog, I hear from folks at all levels of the industry about rural broadband. I’ve heard a lot of stories from technicians telling me that some of the big telcos have only tackled the low-hanging fruit in the CAF builds. I’ve heard from others that some telcos aren’t spending more than a fraction of the CAF II money they got from the FCC and are pocketing much of it. I’ve heard from rural customers who supposedly already got a CAF II upgrade and aren’t seeing speeds improved to the 10/1 threshold.

The CAF II program will be finished soon and I’m already wondering how the telcos are going to report the results to the FCC if they took shortcuts and didn’t make all of the CAF II upgrades. Will they say they’ve covered everybody when some homes saw no improvement? Will they claim 10/1 Mbps speeds when many households were upgraded to something slower? If they come clean, how will the FCC react? Will the FCC try to find the truth or sweep it under the rug?

Shrinking Competition for Transport

Bloomberg reported that CenturyLink and Alphabet are interested in buying Zayo. It’s been anticipated that Zayo would be the next fiber acquisition target since the Level 3 merger with CenturyLink since they are the largest remaining independent owner of fiber.

As you might expect, the biggest owners of fiber are the big telcos and cable companies. Consider the miles of fiber owned by the ten biggest fiber owners – I note these miles of fiber are from the end of 2017 and a few of these companies like Verizon have been building a lot of fiber since then.

AT&T 1,100 K
Verizon 520 K
CenturyLink / Level 3 450 K
Charter 233 K
Windstream 147 K
Comcast 145 K
Frontier 140 K
Zayo 113 K
Cogent 57 K
Consolidated 36 K

You might wonder why this matters? First, Zayo is the largest company on the list who’s only business is to sell transport. All of Zayo’s fiber is revenue producing. While the companies above it on the list have a lot more fiber, a lot of that fiber is in the last mile in neighborhoods where there is not a lot of opportunity to sell access to others. The biggest independent fiber owner used to be Level 3, with 200,000 miles of revenue-producing fiber before they merged with CenturyLink.

The numbers on this chart don’t tell the whole story. Companies like Zayo also swap fiber with other networks. They may trade a pair of fibers on a route they own for a route elsewhere that they want to reach. These swapping arrangements mean the transport providers like Zayo, Cogent and Level 3 control a lot more fiber than is indicated by these numbers.

It matters because as soon as you get outside of the metropolitan areas there are not many options for fiber transport. A few years ago I helped a City look for fiber transport and the three options they found that were reasonably priced were CenturyLink, Level 3 and Zayo. If CenturyLink buys Zayo they will have purchased both competitors in this region and will effectively eliminated fiber transport competition for this community. Without that competition it’s inevitable that transport prices will rise.

I think back to the early days of competition after the Telecommunications Act of 1996. I remember working with clients in the 1990s looking for fiber transport, and there were many cases where there was only one provider willing to sell transport to a community. If the sole provider was the local telco or cable company it was likely that the cost of transport was four or five times more expensive than prices in nearby communities with more choices. When I worked with rural providers in the early 2000s, one of the first question I always asked was about the availability of  transport – because lack of transport sometimes killed business plans.

Since then there has been a lot of rural fiber built by companies like statewide fiber networks and others who saw a market for rural transport. Much of the rural construction was egged on by the need to get to cellular towers.

My fear is that we’ll slide back to the bad-old-days when rural fiber was a roadblock for providing broadband. I don’t so much fear for the most rural places because those fiber networks are owned by smaller companies and they aren’t going away. I fear more for places like county seats. I worked with a city in Pennsylvania a few years ago where there was a decent number of competitors for transport – Verizon, Zayo, Level 3 and XO. Since then Verizon bought XO and CenturyLink might own the other two. That city is not going to lose transport options, but the reduction from four providers to two giant ones almost surely means higher transport costs over time.

I am intrigued that Alphabet (the parent of Google Fiber) would look at buying an extensive fiber network like Zayo. Google is one of the biggest users of bandwidth in the country due to the web traffic to Google and YouTube. Their desire for fiber might be as simple as wanting to control the fiber supply chain they use. If so, that’s almost as disconcerting as CenturyLink buying Zayo if Google wouldn’t remain as a fierce transport competitor.

The Huge CenturyLink Outage

At the end of December CenturyLink had a widespread network outage that lasted over two days. The outage disrupted voice and broadband service across the company’s wide service territory.

Probably the most alarming aspect pf the outage is that it knocked out the 911 systems in parts of fourteen states. It was reported that calls to 911 might get a busy signal or a recording saying that “all circuits are busy’. In other cases, 911 calls were routed to the wrong 911 center. Some jurisdictions responded to the 911 problems by sending out emergency text messages to citizens providing alternate telephone numbers to dial during an emergency. The 911 service outages prompted FCC Chairman Ajit Pai to call CenturyLink and to open a formal investigation into the outage.

I talked last week to a resident of a small town in Montana who said that the outage was locally devasting. Credit cards wouldn’t work for most of the businesses in town including at gas stations. Businesses that rely on software in the cloud for daily operations like hotels were unable to function. Bank ATMs weren’t working. Customers with CenturyLink landlines had spotty service and mostly could not make or receive phone calls. Worse yet, cellular service in the area largely died, meaning that CenturyLink must have been supplying the broadband circuits supporting the cellular towers.

CenturyLink reported that the outage was caused by a faulty networking management card in a Colorado data center that was “propagating invalid frame packets across devices”. It took the company a long time to isolate the problem, and the final fix involved rebooting much of the network electronics.

Every engineer I’ve spoken to about this says that in today’s world it’s hard to believe that it would take 2 days to isolate and fix a network problem caused by a faulty card. Most network companies operate a system of alarms that instantly notify them when any device or card is having problems. Further, complex networks today are generally supplied with significant redundancy that allows the isolation of troubled components of a network in order to stop the kind of cascading outage that occurred in this case. The engineers all said that it’s almost inconceivable to have a single component like a card in a modern network that could cause such a huge problem. While network centralization can save money, few companies route their whole network through choke points – there are a dozen different strategies to create redundancy and protect against this kind of outage.

Obviously none of us knows any of the facts beyond the short notifications issued by CenturyLink at the end of the outage, so we can only speculate about what happened. Hopefully the FCC enquiry will uncover the facts – and it’s important that they do so, because it’s always possible that the cause of the outage is something that others in the industry need to be concerned about.

I’m only speculating, but my guess is that we are going to find that the company has not implemented best network practices in the legacy telco network. We know that CenturyLink and the other big telcos have been ignoring the legacy networks for decades. We see this all of the time when looking at the conditions of the last mile network, and we’ve always figured that the telcos were also not making the needed investments at the network core.

If this outage was caused by outdated technology and legacy network practices then such outages are likely to recur. Interestingly, CenturyLink also operates one of the more robust enterprise cloud services in the country. That business got a huge shot in the arm through the merger with Level 3, with new management saying that all of their future focus is going to be on the enterprise side of the house. I have to think that this outage didn’t much touch that network, just more likely the legacy network.

One thing for sure is that this outage is making CenturyLink customers look for an alternative. A decade ago the local government in Cook County, Minnesota – the northern-most county in the state – was so frustrated by continued prolonged CenturyLink network outages that they finally built their own fiber-to-the-home network and found alternate routing into and out of the County. I talked to one service provider in Montana who said they’ve been inundated after this recent outage by businesses looking for an alternate to CenturyLink.

We have become so reliant on the Internet that major outages are unacceptable. Much of what we do everyday relies on the cloud. The fact that this outage extended to cellular outages, a crash of 911 systems and the failure of credit card processing demonstrates how pervasive the network is in the background of our daily lives. It’s frightening to think that there are legacy telco networks that have been poorly maintained that can still cause these kinds of widespread problems.

I’m not sure what the fix is for this problem. The FCC supposedly washed their hands of the responsibility for broadband networks – so they might not be willing to tackle any meaningful solutions to prevent future network crashes. Ultimately the fix might the one found by Cook County, Minnesota – communities finding their own network solutions that bypass the legacy networks.

Our National Telecom Priorities

I recently wrote a blog that talked about the FCC’s formal goals for the next few years. I noted in that blog that some of the FCC’s actions currently seem to conflict with their stated goals. Today I present my take on what I see as the actual current priorities in our industry.

5G, 5G, 5G. The FCC and other policy makers have swallowed the 5G hype hook, line and sinker. I have no doubt that 5G will be an important part of our future telecom landscape, but the hype seems way out of proportion to the reality we are likely to see. Nothing highlights this better than a Qualcomm-sponsored article that claims that 5G technology will be as important as the introduction of electricity.

The FCC is sweeping away regulations that might interfere with 5G and already killed local say over the location of small cell electronics and towers. The FCC is well on the way towards allocating massive amounts of spectrum for 5G and ignoring other spectrum needs. The White House even held a 5G summit where politicians were repeating the talking points of the 5G carriers.

This all seems premature since engineers all say that the major benefits of mature 5G will come years from now. There will be some early 5G technology introduced into the market over the next few years, but this will not include the characteristics that make 5G an important technology. From a policy perspective, 5G seems to have won the war without having had to fight any of the battles. I’ve never seen this industry (and the politicians) go so gaga over a new technology that we aren’t even going to see for a while. The marketers at the cellular companies have clearly hit a hype home run.

The Rural Digital Divide Gets Lip Service. Talking about solving the rural digital divide is a high priority. The FCC rarely makes a presentation without mentioning how important this is to them. However, the FCC and others in Washington DC are doing almost nothing to solve the problem. The FCC even went so far as to list the rural digital divide as the first priority on their own list of goals but has done little to address the problem.

There is universal acknowledgement that the private sector is not going to invest in rural broadband without some funding help from government. Yet all of the state and federal grant programs added together are throwing millions of dollars at a problem that needs many billions of dollars to solve.

Meanwhile, the rural digital divide is widening as urban areas are seeing significantly faster broadband speeds while rural America is stuck with little or no broadband.

The Big ISPs Want to be Google. Every one of the big ISPs has made investments to try to catch-up with Google. The big ISPs want to monetize their vast troves of customer data. Big ISPs are envious of the advertising money made by Google and Facebook and want to grab a piece of those dollars. The FCC has aided the big companies by weakening consumer privacy protections.

But for whatever reason, the big ISPs haven’t yet figured this out. They have the most intimate and detailed access to customer data but have scarcely found any ways to understand it, yet alone monetize it.

Take My Residential Customers, Please. The big telcos have made it clear that they are not particularly interested in the residential market. CenturyLink made it clear this year that they will no longer invest in residential networks. Verizon has already sold vast tracts of rural networks. AT&T is constantly petitioning the FCC to let them tear down rural copper. Verizon is talking about expanding wireless local loops using 5G, but we’ll have to wait to see how serious they are about it.

Big ISPs Continue to Try to Squash Competition. The big ISPs miss no opportunity to squash competition, no matter how small. They all still rail against municipal competition, although all such competition added together is barely a blip on the national radar. They still pay for hit pieces – articles and papers that blast municipal fiber networks – even ones like Chattanooga EPB that is a paragon of competitiveness. They have been working hard to kick CLECs off of their dying copper networks, even thought the CLECs have been investing in newer DSL that can deliver decent broadband over the copper.