Google Fiber Comes to Iowa

The City of West Des Moines recently announced a deal with Google Fiber to bring fiber to pass all 36,000 residents and businesses in the city. This is a unique business model that can best be described as open-access conduit.

The city says that the estimated cost of the construction is between $35 million and $40 million and that the construction of the network should be complete in about two-and-a-half years. The full details of the plan have not yet been released, but the press is reporting that Google Fiber will pay $2.25 per month to the city for each customer that buys service from Google Fiber.

What is most unique about this arrangement is that conduit will be built along streets and into yards and parking lots to reach every home and business. I know of many cities that lease out some empty conduit to ISPs and carriers, but the big limitation of most empty conduit is that it doesn’t provide easy access to get from the street to reach a customer. West Des Moines will be spending the money to build the conduit to serve the last hundred feet.

This business arrangement will still require Google Fiber to pull fiber throughout the entire empty conduit network – but that is far cheaper for the company than building a network from scratch. The big cost of building any fiber network is the labor needed to bring the fiber along every street – and the city has absorbed that cost. The benefit of this arrangement for Google Fiber is obvious – the company saves the cost of building a standalone fiber network in the City. It’s the cost of financing expensive networks up-front that makes ISPs hesitant to enter new markets.

From a construction perspective, I’m sure that the City is building fiber with some form of innerduct – which is a conduit with multiple interior tubes that can accommodate multiple fibers (as is shown in the picture accompanying this blog). This would allow additional ISPs to coexist in the same conduits. If the conduits built through yards also include innerduct it would make it convenient for a customer to change fiber ISPs – disconnect fiber from ISP A and connect to the fiber from ISP B.

The City is banking on other ISPs using the empty conduit because Google Fiber fees alone won’t compensate the city for the cost of the conduit. The press reported that Google Fiber has guaranteed the City a minimum payment of at least $4.5 million over 20 years. I’m sure the City is counting on Google Fiber to perform a lot better than that minimum, but even if Google Fiber connects to half of all of the customers in the City, the $2.25 monthly fee won’t repay the City’s cost of the conduit.

This business model differs significantly from the typical open-access network model. In other open-access networks, the City pays for 100% of the cost of the network and the electronics up to the side of a home or business. The typical monthly fee for an ISP to reach a customer in these open access-networks ranges between $30 and $45 per month. Those high fees invariably push ISPs into cherry-picking and only pursuing customers willing to pay high monthly rates. The $2.25 fee in West Des Moines won’t push ISPs to automatically cherry-pick or charge a lot.

Any ISP willing to come to the city has a few issues to consider. They avoid the big cost of constructing the conduit network. But a new ISP will still need to pay to blow fiber through the conduit. Any new ISP will also be competing against Google Fiber. One of the most intriguing ISPs already in the market is CenturyLink. The company has shown in Springfield, Missouri that it is willing to step outside the traditional business model and use somebody else’s network. I would have to imagine that other ISPs in the Midwest perked up at this announcement.

In announcing the network, the City said that they hoped this network would bring fiber to everybody in the City. Google Fiber doesn’t typically compete on price. Earlier this year Google Fiber discontinued its 100 Mbps broadband connection for $50. Many homes are going to find the $70 gigabit product from Google Fiber to be unaffordable. It will be interesting over time to see how the city plans on getting broadband to everybody. Even municipalities that own their own fiber network are struggling with the concept of subsidizing fiber connections below cost to make them affordable.

One thing this partnership shows is that there are still new ideas to try in the marketplace. For an open-access conduit system to be effective means attracting multiple ISPs, so this idea isn’t going to work in markets much smaller than West Des Moines. But this is another idea for cities to consider if the goal is to provide world-class broadband for citizens and businesses.

Big Regional Network Outages

T-Mobile had a major network outage last week that cut off some voice calls and most texting for nearly a whole day. The company’s explanation of the outage was provided by Neville Ray, the president of technology.

The trigger event is known to be a leased fiber circuit failure from a third party provider in the Southeast. This is something that happens on every mobile network, so we’ve worked with our vendors to build redundancy and resiliency to make sure that these types of circuit failures don’t affect customers. This redundancy failed us and resulted in an overload situation that was then compounded by other factors. This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls.

In plain English, the electronics failed on a leased circuit, and then the back-up circuit also failed. This then caused a cascade that brought down a large part of the T-Mobile network.

You may recall that something similar happened to CenturyLink about two years ago. At the time the company blamed the outage on a bad circuit card in Denver that somehow cascaded to bring down a large swatch of fiber networks in the West, including numerous 911 centers. Since that outage, there have been numerous regional outages, which is one of the reasons that Project THOR recently launched in Colorado – the cities in that region could no longer tolerate the recurring multi-hour or even day-long regional network outages,

Having electronics fail is a somewhat common event. This is particularly true on circuits provided by the big carriers which tend to push the electronics to the max and keep equipment running to the last possible moment of its useful life. Anybody visiting a major telecom hub would likely be aghast at the age of some of the electronics still being used to transmit voice and data traffic.

I can recall two of my clients that have had similar experiences in the last few years. They had a leased circuit fail and then also saw the redundant path fail as well. In both cases, it turns out that the culprit was the provider of the leased circuits, which did not provide true redundancy. Although my clients had paid for redundancy, the carrier had sold them primary and backup circuits that shared some of the same electronics at ley points in the network – and when those key points failed their whole network went down.

However, what is unusual about the two big carrier outages is that the outages somehow cascaded into big regional outages. That was largely unheard of a decade ago. This reminds more of what we saw in the past in the power grid, when power outages in one town could cascade over large areas. The power companies have been trying to remedy this situation by breaking the power grid into smaller regional networks and putting in protection so that failures can’t overwhelm the interfaces between regional networks. In essence, the power companies have been trying to introduce some of the good lessons learned over time by the big telecom companies.

But it seems that the big telecom carriers are going in the opposite direction. I talked to several retired telecom network engineers and they all made the same guess about why we are seeing big regional outages. The telecom network used to be comprised of hundreds of regional hubs. Each hub had its own staff and operations and it was physically impossible for a problem from one hub to somehow take down a neighboring hub. The worst that would happen is that routes between hubs could go dark, but the problem never moved past the original hub.

The big telcos have all had huge numbers of layoffs over the last decade, and those purges have emptied out the big companies of the technicians that built and understood the networks. Meanwhile, the companies are trying to find efficiencies to get by with smaller staffing. It appears that the efficiencies that have been found are to introduce network solutions that cover large areas or even the whole nation. This means that the identical software and technicians are now being used to control giant swaths of the network. This homogenization and central control of a network means that failure in any one place in the network might cascade into a larger problem if the centralized software and/or technicians react improperly to a local outage. It’s likely that the big outages we’re starting to routinely see are caused by a combination of the  failure of people and software systems.

A few decades ago we somewhat regular power outages that affected multiple states. At the prodding of the government, the power companies undertook a nationwide effort to stop cascading outages, and in doing so they effectively emulated the old telecom network world. They ended the ability for an electric grid to automatically interface with neighboring grids and the last major power outage that wasn’t due to weather happened in the west in 2011.

I’ve seen absolutely no regulatory recognition of the major telecom outages we’ve been seeing. Without the FCC pushing the big telcos, it’s highly likely nothing will change. It’s frustrating to watch the telecom networks deteriorate at the same time that electric companies got together and fixed their issues.

CenturyLink Expanding Fiber

CenturyLink recently announced its fiber plans for 2020 and says it will be building to pass 400,000 homes and businesses with fiber this year as a follow-up to 2019 that saw the company add 300,000 passings. Like with all big telco announcements, a bit of looking behind the scenes is needed to understand what the company is doing.

In 2017 CenturyLink was engaged in a major fiber expansion plan and built that year to pass 900,000 homes and businesses, mostly in large cities and surrounding suburbs in places like Seattle, Phoenix, and Denver. Those expansions plans were put on hold when new CEO Jeff Storey replaced the telco-minded Glen Post. One of Storey’s first announcements as CEO was that the company was no longer going to pursue capital projects with ‘infrastructure returns’ and building FTTH came to a screeching halt.

It was a lot harder than Storey might have hoped to inject the Level 3 mindset into a 100-year-old telco, and the company bogged down and stock prices dropped. Starting last year, the company started talking again about aggressively expanding its fiber network to add large buildings to the network. The company recently said it had connected to over 18,000 buildings last year. In digging deeper into things the company discussed over the last year, it seems that those buildings were a combination of multi-tenant business buildings and apartment complexes. The company also said that it was building a lot of the new fiber in 2019 to reach small cell sites.

The company also recently announced that it had added 300,000 passings as a result of the fiber expansion last year – a number that I can’t find mentioned as a goal last year. This does not mean that the company built fiber to pass 300,000 homes. Many of the passings came from the 18,000 buildings that were added to the network. CenturyLink has also entered into a contract to operate the fiber network in Springfield, MO – a network that is funded, built, and owned by the City. The 85,000 or so passings from that project seem to be included in the fiber passings claimed for 2019 and planned for 2020.

CenturyLink says it plans to add 400,000 fiber passings this year. The company is still aggressively adding buildings to the network and is also still building to small cell sites. The markets on the list for this year’s expansion include Denver, Omaha, Phoenix, Portland., Salt Lake City, Spokane, Springfield, MO, and a few others.

CenturyLink says the fiber expansion is starting to pay off. While the company lost a net of 11,000 broadband customers in the first quarter of this year, they added 60,000 subscribers with speeds of 100 Mbps or faster. Those gains were part of the industry gain of over 1.1 million broadband customers in the first quarter – at least some of which came as a result of the needs of employees and students being forced to work from home.

The company has gotten back into the infrastructure business somewhat reluctantly but now seems to have embraced some aspects of fiber expansion. CenturyLink is still bullish about adding buildings to the network and are at number four in terms of on-net buildings. I would be surprised if the fiber expansion includes any significant construction to reach single-family homes. It seems a lot more likely that the company is picking off low-hanging fruit in places where it is installing fiber to reach small cell sites or lucrative buildings. That’s the same philosophy that helped AT&T add over 12 million fiber passings over the last few years.

Jeff Storey is still adamantly painting a picture of a company that is focused on enterprise services and business applications. Any expansion into residential neighborhoods is likely a by-product of taking advantage of fiber built to pursue the primary goal.

But no matter how they are getting there, it’s good to see CenturyLink building fiber again. In 2018 it looked like they might permanently duck out of fiber construction. However, the stock market disappointment, and perhaps seeing AT&T success with limited fiber expansion convinced management some that fiber can earn more than infrastructure returns.

Enough is Enough

CenturyLink recently petitioned the FCC to allow them to be late in implementing the CAF II upgrades where the FCC doled out $11 billion to upgrade rural broadband speeds to 10/1 Mbps. The ostensible reason for the delay is the COVID-19 pandemic, but CenturyLink was already behind and notified the FCC earlier this year that they hadn’t completed their 2019 CAF II installation in 23 out of 33 states.

I say enough is enough. It’s time for the FCC to demand a reckoning of CAF II and begin handing out draconian penalties to the telcos that didn’t meet their obligations. I’m positive that if this was assessed fairly that the FCC will find that the vast majority of big telco customers have never gotten an upgrade to 10/1 Mbps.

Let’s start by looking at CenturyLink’s request. There is no reasonable explanation they can offer for not meeting their obligations in 2019. That was the fourth of a five-year buildout obligation, and the company has known for years what’s needed to be done – and they had the federal money in their pocket to make the upgrades. The claim for this year is also largely bogus. I have a lot of clients that are being cautious now about entering customer premises, but I don’t know any carrier that has stopped doing work outside of customer homes. I can’t think of any practical reason that COVID-19 would cause a delay for CenturyLink. Even if they upgrade somebody’s DSL, they could mail them a new modem – telcos have been having customers self-install DSL modems for twenty years.

It’s time to stop the pretense that CenturyLink or the other big telcos have been busy upgrading rural DSL. I don’t know anybody who thinks that’s happened. I have anecdotal evidence that it hasn’t, My company has been helping rural counties with broadband feasibility studies for many years. In the last four years, we’ve been asking rural customers to take speed tests – and I’ve never seen even one rural DSL connection that transmits at a speed of 10/1 Mbps. I’ve haven’t seen many that have tested above 5 Mbps. I’ve seen a whole lot that tested at less than 3, 2 or even 1 Mbps. Many of these tests have been in areas that are supposed to have CAF II upgrades.

I’ve also never talked to any County officials who have heard from the telcos that their county got rural broadband upgrades. One would think the telcos would brag locally when they were finished with upgrades as a pitch to get new customers. After all, customers that have only had slow DSL or satellite service should be flocking to 10/1 DSL. I’ve also not seen a marketing campaign talking about faster speeds due to CAF II. I’ve been searching the web for years to find testimonials from customers talking about their free upgrade to 10/1 Mbps, but I’ve never found anybody who has ever said that. This is not to say there have been zero upgrades in the CAF II areas, but I see no evidence of widespread upgrades.

The reality is that CenturyLink got new leadership a few years ago who immediately announced that the company was going to stop making ‘infrastructure return’ investments. We have Frontier that miraculously recently found 16,000 Census blocks that now have speeds of at least 25/3 Mbps when I’m still looking for proof that they upgraded places to 10/1 Mbps. Go interview folks in West Virginia if you think they’ve made any CAF II upgrades.

The FCC has a choice now. They can wimp out and grant the delay that CenturyLink is requesting, or the agency can come down on the side of rural broadband. There is no middle ground when it comes to CAF II. This FCC didn’t make the original CAF II decision – but they are the ones that are supposed to make sure the upgrades are done, and they are supposed to be penalizing telcos that failed to make the upgrades.

The response to CenturyLink’s request should be a giant penalty for missing the 2019 deadlines and a reminder that the company is still on the hook for 2020 unless they want more fines.

The FCC also needs to aggressively start testing in the areas that have supposedly gotten CAF II upgrades. This doesn’t have to be a big expensive testing program. We know exactly where CAF II should have been implemented – the FCC has made it easy by overlaying the CAF II footprint over Google maps. The FCC could ask County administrators across the US to solicit a speed test at CAF II locations – the Counties would be glad to oblige. If the FCC wanted to know the truth about CAF II they could get massive feedback within a few weeks about the abject failure of the CAF II program.

The ultimate penalty ought to be the return of CAF II money to the Universal Service Fund for areas that aren’t upgraded to 10/1 Mbps. Then the money could finally be given to somebody that will upgrade to real broadband. The CAF II program was ill-conceived, but the big telcos should have used that money to bring rural speeds up to 10/1 Mbps. Had they done so, we’d have millions of more homes that wouldn’t be struggling so hard during COVID-19. This FCC has a chance to do their job and set things right.

Funding Middle-mile Fiber

A decade ago, there were a lot of federal grants given to build middle mile fiber. That’s the fiber that connects communities and that provides a path between a community and a connection to the Internet. Ideally, backbone fiber also provides a diverse route with ring electronics so that if one of the fibers serving a community is cut the broadband connection to the community keeps working.

It’s not as easy to find grants for backbone fiber today. For instance, the $16.4 RDOF grant for later this year is aimed at bringing last-mile fiber to remote places in the country but doesn’t let an applicant file for money to build just backbone fiber to reach those same remote communities. It’s almost as if the FCC somehow thinks that most of America is somehow now in reach of a reliable connection to the Internet.

A new network called Project THOR recently launched in northwest Colorado that is purely a backbone project and that shows the continued need for middle-mile fiber. Project  THOR is a consortium of 14 communities that came together because they regularly suffered major broadband outages any time there was a middle-mile fiber cut in the region or an electronics problem at CenturyLink, the backbone provider for the entire section of the state. Network outages can be devastating and mean non-functional 911 centers, hospitals with no broadband, city governments that are crippled, and business districts that can no longer take credit cards or use the Internet.

The cities and towns in the region selected Mammoth Networks to create and operate a new middle-mile fiber network. The initial network is cobbled mostly with dark fiber leased from Colorado DOT, other networks like Strata, and lit fiber from CenturyLink, Comcast, and Zayo. The plan is to eventually replace lit fiber with dark fiber or constructed fiber. Mammoth oversaw the construction of lateral fibers inside of communities and also designed and implemented the electronics network. The State of Colorado Department of Local Affairs funded the lateral construction and half of the equipment purchases through a broadband grant.

The communities are free to use the network in any way they see fit. The Project THOR network terminates at a meet-me center created in each community. Several of the communities on the new network have already built fiber-to-the-home and the new network provides Internet redundancy. Other communities located the meet-me room at a hospital or other critical facility so that they’d see an immediate benefit from the network.

Project THOR brings two advantages to the region. First, the network is designed to carry up to 400 Gbps – much more capacity than any existing fiber in the region. Mammoth Networks was also able to string together routes that provide diversity for each city to protect against fiber cuts. A single fiber cut on the Project THOR routes won’t interrupt service to any of the member communities.

There was no better evidence of the effectiveness of Project THOR than when a CenturyLink fiber outage hit the region a few days after Project THOR was activated. On April 10, there was a 6.5-hour outage, and because of Project THOR, the 911 PSAP in Aspen, hospitals in Granby and Kremmling, and the city governments in Aspen and Glenwood Springs stayed operational – but would have lost broadband service without Project THOR. The Project THOR route was the only network to stay functional in the region during the outage.

It’s common knowledge that the large incumbent telcos haven’t put any money into last-mile broadband in rural areas – but the same thing is true for middle-mile fiber. What’s most amazing about Project THOR is that CenturyLink could easily be providing much of the same redundancy and quality of service that the new network offers. However, the company doesn’t seem interested in making the needed investments in diverse fiber routes or the associated electronics.

There are huge areas of the country that suffer from inadequate middle-mile fiber routes. It would be great if there was a grant program aimed specifically for middle-mile fiber. The need is there because existing middle-mile fibers are often not adequate for today’s bandwidth needs and are definitely not ready for the increased bandwidth needs of the future. Most incumbent middle mile has little redundancy, leading to regular Internet outages. It’s also not unusual to find relatively ancient electronics on middle-mile routes in rural areas.

Project THOR is an example of cities that banded together to fix a common issue – in this case, regular and extended Internet outages. In the ideal world, the incumbents would fix such issues because it’s the right thing to do. However, the lack of capital spending on rural broadband affects middle-mile fiber as much as if impacts last-mile fiber – both are inadequate in most rural areas.

Old Regulation Rears its Head

The way that we regulate telecom services is interesting. The FCC has effectively eliminated federal regulation of broadband, the service that over 90% of households now use. Meanwhile, landline telephone service, the telecom product that is used by an ever-decreasing number of homes is still heavily regulated.

The target of much of the remaining regulation are the big telephone companies that still operate large copper networks. It’s easy to bash the big telephone companies because of the poor quality of services offered on those copper networks, and I’ve done so many times in this blog.

When you stop and think about it, those companies are still using copper networks built in the 50s, 60s, and 70s. If the telcos had been good stewards of those networks and maintained them meticulously those networks would still be 50 to 70 years old, and older than the 35-40 year expected life for the networks. The big telcos largely ignored maintenance of copper for the last 30 years or more, and frankly, it’s a miracle that the old copper networks are still working.

Perhaps the oddest aspect of telephone regulation is that a regulatory body will occasionally punish a big telco for still being in the copper business. A good example is a proceeding in New Mexico last fall where CenturyLink asked to be deregulated for landline telephone services. This doesn’t mean that they would stop offering the services, but rather that many of the old regulations put in place at the heyday of the telephone monopolies would be excused. Most states have deregulated the big telcos from a lot of the old telephone rules.

The New Mexico Public Regulations Commission (NMPRC) rejected the request and said that CenturyLink had not demonstrated that there was ‘effective competition’ for residential telephone service. It’s hard to find any way to defend that decision. First, in many states, there are now more residential telephone customers using cable company telephone services than the old telephone company copper. Interestingly, cable companies face almost no regulation in providing telephone service and any cable company in New Mexico does not live under the same rules that CenturyLink must follow.

Further, the latest surveys I’ve seen show that 96% of US adults now have a cell phone. It’s hard to say with a straight face that cellular service is not a direct competitor to landline telephone service. Considering the big recent stir at the FCC where cellular 4G coverage maps were shown to be largely fictional, perhaps a lot of rural New Mexico doesn’t have cellular coverage – and perhaps that’s what drove the Commission’s decision. It’s worth noting  that cellular companies are also not as heavily regulated as landline telephone providers.

The regulation that is most relevant in this case is the obligation to be the carrier of last resort. The telcos like CenturyLink are still expected, within some regulatory exceptions, to provide service to anybody who asks for service. That obligation doesn’t extend to the cable companies, to the cellular companies, or even to rural broadband – just to telephone service.

I have no doubt that there are rural homes in the state for which CenturyLink is the only communications link to the world. In areas where there is no cellular service and where the cable companies refuse to build networks, there are rural homes that rely on CenturyLink and other telcos to keep them connected. The regulatory question that must be asked is if such homes are sufficient reason to still strongly regulate telephone service in a state. Hopefully, the number of homes without cellular service will decrease significantly when the FCC awards the $9 billion in the 5G Fund program to extend cellular service to more remote communities.

It’s not an easy question to answer. We know CenturyLink could have done a better job of taking care of their copper. In this country, the smaller independent telephone companies did the needed maintenance to keep copper in the best shape possible. We saw the same thing in Germany where the copper networks were built at the same time as US copper, but which have been maintained better.

But in this country, most of the smaller telcos have already replaced, or have plans to replace the old copper with fiber. In Germany, there are vigorous public debates on the topic, with engineers saying that the copper networks are not likely to last more than another decade. Where copper remains the Germans have invested in the fastest DSL possible – something the big telcos here inexplicably have not done.

To some degree the decision in New Mexico is meaningless. No regulatory decision can make the old copper perform better or last longer, so there are not many practical ramifications of the Commission’s decision. CenturyLink didn’t even own these networks for most of the years when the maintenance wasn’t done – although they have likely cut back further on maintenance in recent years, as have the other big telcos.

I’m not highlighting New Mexico for this issue because many other states have made similar regulatory decisions. Regulators are rightfully mad at the big telcos for neglecting copper, and even madder that there are no plans to upgrade the copper to something better. But the time for regulators to do something about this was twenty and thirty years ago. The copper wires in New Mexico are going to die, and at some future date the networks will go dark. The regulators can choose to regulate copper down to the last day of the last customer – but to a large degree, the remaining regulations don’t mean a whole lot.

CenturyLink Ready to Launch Gigabit Broadband in Springfield, MO

It’s rare to be surprised by events in the telecom world. The announcement last summer that CenturyLink will be an ISP on a city-owned fiber network in Springfield MO was one of the most surprising things I’ve heard since the announcement years ago that Google was going to become a gigabit ISP. The joint venture has been progressing and CenturyLink says it should be adding customers in the community this spring.

The partnership between the city and CenturyLink is interesting:

  • CenturyLink has agreed to lease the network over 15-years at a payment that made the city comfortable enough to build the network. The city says they won’t have to raise electric rates since the lease revenue stream justifies the cost of the new $120 million fiber expansion.
  • The city is providing dark fiber and CenturyLink will provide all of the electronics. There have been no public announcements saying which party pays for the fiber drops. Since this is being touted as an expansion of smart-grid, it would make sense that the city owns the drops.
  • The arrangement is described as non-exclusive, meaning that other ISPs are free to serve on the network. The announcements don’t say if CenturyLink gets a head-start over other ISPs through some period of exclusivity before open access kicks in. That’s been the case in similar arrangements.
  • CenturyLink is offering $65 gigabit broadband ‘for-life’ with a guarantee that the price will never be increased. Speeds are advertised ‘up to 940 Mbps. In other CenturyLink markets the gigabit product requires paperless billing and prepayment with a credit card or bank debit. CenturyLink charges $5 for an optional WiFi modem.

There are a few other similar well-known arrangements in the industry. This is similar to the Google Fiber arrangement with Huntsville, Alabama. It’s similar to the Ting arrangement in Westminster, Maryland and Charlottesville, Virginia. What’s unusual and surprising about this deal is that it’s with one of the big incumbent telcos. However, CenturyLink is not the incumbent in Springfield and enters the market purely as an outside ISP. CenturyLink will be competing side-by-side with AT&T, the first instance of two large incumbents telcos competing in a residential market. The other competitor and the incumbent cable provider in Springfield is Mediacom.

There are some in the industry touting this as a new paradigm for bringing gigabit fiber – but I’m not sure that is so. Like with any business model, all of the facts and the numbers must line up for any market to be a good target for overbuilding with fiber. It’s possible that there are unique characteristics of Springfield that might make this model hard to replicate in most other places.

Springfield owns a municipal electric utility and the utility decided years ago to build fiber to serve its own needs and to bring fiber to businesses in the city. The city started this new venture already owning 700 miles of fiber – much of which will likely be the backbone for building the last-mile for this venture. Springfield is also touting this as a smart grid initiative, meaning the electric utility is likely picking up a piece of the cost of the new fiber construction. There is a good chance that the math would not look nearly so favorable for a city without an electric utility – because in that case the venture would be starting with no existing fiber and the new fiber venture would have to absorb 100% of the cost of the new construction. I’ve looked at this lease model for cities that don’t own existing fiber or an electric utility and the math is often not pretty.

Don’t read those last statements as a criticism of the fiber lease model, but rather just as a recognition that all of the financial factors must align just right for this kind of venture to work. Any city owning an electric utility ought to do the math and consider this model. Cities with low construction costs for fiber might also be good candidates.

The surprising part of this arrangement is that this is being done by CenturyLink. This is an incumbent telco that is well known throughout rural America for operating lousy copper networks. The company has been ignoring the customers in rural markets, and CenturyLink customers living in rural Missouri can’t be thrilled to hear that the company will be offering gigabit fiber in a new market while continuing to ignore their broadband plight. CenturyLink is not going to sink a lot of capital in Springfield, but it’s paying for the cost of electronics and installation.

I have to give CenturyLink credit for tackling this venture. They were building fiber-to-the-home networks before Jeff Storey, the new CEO put a kibosh on spending capital for projects earning ‘infrastructure returns’. The FTTP businesses is an economy of scale business and CenturyLink can take advantage of the staff and platforms they already have in place to operate efficiently in Springfield. Since this is dark fiber the company can still do everything the CenturyLink way – which is an important factor for a big telco. We’ll have to wait to see if this is a new business line for CenturyLink or if Springfield is a unique case.

Will the Big Telcos Pursue RDOF Grants?

One of the most intriguing questions concerning the upcoming $16.4 billion RDOF grant program is if the big telcos are going to participate. I’ve asked the question around the industry and I’ve talked to folks who think the big telcos will fully wade into the reverse auctions, while others think they’ll barely play. We’re not likely to know until the auctions begin.

The big telcos were the full beneficiaries of the original CAF II program when the FCC surprisingly decided to unilaterally award the big telcos the full $9 billion in funding. In that grant program, CenturyLink received over $3 billion, AT&T almost $2.6 billion, Frontier nearly $2 billion, and Windstream over $1 billion. The telcos were supposed to upgrade much of their most rural properties to receive broadband speeds of at least 10/1 Mbps.

CenturyLink and Frontier both recently told the FCC that they are behind in the CAF II build out and didn’t meet their obligation at the end of 2019 to be 80% finished with the upgrades. From what I hear from rural communities, I think the problem is a lot more severe than just the telcos being late. Communities across the country have been telling me that their residents aren’t seeing faster speeds and I think we’re going to eventually find out that a lot of the upgrades aren’t being made.

Regardless of the problems with the original CAF II, the FCC is now offering the $16.4 billion RDOF grant program to cover much of the same areas covered by CAF II. The big telcos are faced with several dilemmas. If they don’t participate, then others are going to get federal assistance to overbuild the traditional big telco service territories. If the big telcos do participate, they have to promise to upgrade to meet the minimum speed obligations of the RDOF of 25/3 Mbps.

Interestingly, the upgrades needed to raise DSL speeds on copper to 25/3 Mbps are not drastically different than the upgrades needed to reach 10/1 Mbps. The upgrades require building fiber deeper into last-mile networks and installing DSL transmitters (DSLAMs) in the field to be within a few miles of subscribers. Fiber must be a little closer to the customer to achieve a speed of 25/3 Mbps rather than 10/1 Mbps – but not drastically closer.

I think the big telcos encountered two problems with the CAF II DSL upgrades. First, they needed to build a lot more fiber than was being funded by CAF II to get fiber within a few miles of every customer. Second, the condition of their rural copper is dreadful and much of it probably won’t support DSL speeds. The big telcos have ignored their rural copper for decades and found themselves unable to coax faster DSL speeds from the old and mistreated copper.

This begs the question of what it even means if the big telcos decide to chase RDOF funding. Throwing more money at their lousy copper is not going to make it perform any better. If they were unable to get 10/1 speeds out of their network, then they are surely going to be unable to get speeds upgraded to 25/3 Mbps.

We can’t ignore that the big telcos have a natural advantage in the RDOF auction. They can file for the money everywhere, and any place where a faster competitor isn’t vying for the money, the big telcos will have a good chance of winning the reverse auction. There are bound to be plenty of places where nobody else bids on RDOF funding, particularly in places like Appalachia where the cost is so high to build, even with grant funding.

It would be a travesty to see any more federal grant money spent to upgrade rural DSL particularly since the FCC already spent $9 billion trying to upgrade the same copper networks. The copper networks everywhere are past their expected useful lives, and the networks operated by the big telcos are in the worst shape. I’ve known many smaller telcos that tried in the past to upgrade to 25/3 on rural DSL and failed – and those companies had networks that were well-maintained and in good condition. It would be impossible to believe the big telcos if they say they can upgrade the most remote homes in the country to 25/3 Mbps speeds. Unfortunately, with the way I read the RDOF rules, there is nothing to stop the big telcos from joining the auction and from taking big chunks of the grant money and then failing again like they did with the original CAF II.

Broadband Still Growing – 3Q 2019

Leichtman Research Group recently released the broadband customer statistics for the third quarter of 2019 for the largest cable and telephone companies. Leichtman compiles most of these numbers from the statistics provided to stockholders other than Cox, which is estimated.

The numbers provided to investors are lower than broadband customers these same companies report to the FCC, and I think that most of the difference is due to the way many of these companies count broadband to apartment buildings. If they provide a gigabit pipe to serve an apartment building, they might that as 1 customer, whereas for FCC reporting they likely count the number of apartment units served.

Following are the broadband customer counts for the third quarter and a comparison to the second quarter of this year.

3Q 2019 Added % Change
Comcast 28,186,000 379,000 1.4%
Charter 26,325,000 380,000 1.5%
AT&T 15,575,000 (123,000) -0.8%
Verizon 6,961,000 (7,000) -0.1%
Cox 5,145,000 25,000 0.5%
CenturyLink 4,714,000 (36,000) -0.8%
Altice 4,180,300 14,900 0.4%
Frontier 3,555,000 (71,000) -2.0%
Mediacom 1,316,000 13,000 1.0%
Windstream 1,040,000 5,700 0.6%
Consolidated 784,151 1,143 0.1%
WOW 773,900 10,420 1.3%
Cable ONE 689,138 7,376 1.1%
Atlantic Broadband 446,137 2,441 0.6%
TDS 437,700 4,300 1.0%
Cincinnati Bell 425,100 (400) -0.1%
100,553,426 605,660 0.6%

Leichtman says this group of companies represents 96% of all US broadband customers. I’m not sure how they calculated that percentage. That implies that there are only about 4 million broadband customers for companies not on this list, and that feels a little low to me.

For the quarter, these companies collectively saw growth that annualizes to 2.4%. This is a significant uptick over the second quarter of 2019 that saw an annualized growth rate of 1.7%.

On an annualized basis the third quarter of 2019 added about the same number of customers that were added for the calendar year of 2018. However, the cable companies are performing better this year while the losses continue to accelerate for the big telcos. The big telco losers for the quarter are Frontier, which lost 2% of its customer base, and AT&T and CenturyLink which each lost 0.8% of their customer base. Following are the annualized changes in customers in 2018 and 2019:

‘                                          2018                2019

Cable Companies        2,987,721        3,317,904

Telcos                            ( 472,124)        ( 895,564)

Total                              2,425,597        2,422,640

Both Comcast and Charter had spectacular quarters and continue to account for most of the growth in broadband, as each company added around 380,000 customers for the quarter. It would be interesting to understand what is driving that growth. Some of that comes from providing broadband to new homes. Some comes from customers converting away from DSL. And some comes from expansion – I know of examples where both companies are building new network around the fringes of their service areas.

Shame on the Regulators

It’s clear that even before the turn of this century that the big telcos largely walked away from maintaining and improving residential service. The evidence for this is the huge numbers of neighborhoods that are stuck with older copper technologies that haven’t been upgraded.  The telcos made huge profits over the decades in these neighborhoods and ideally should not have been allowed to walk away from their customers.

In the Cities. Many neighborhoods in urban areas still have first or second-generation DSL over copper with fastest speeds of 3 Mbps or 6 Mbps. That technology had a shelf-life of perhaps seven years and is now at least fifteen years old.

The companies that deployed the most DSL are AT&T and CenturyLink (formerly Quest). The DSL technology should have been upgraded over time by plowing profits back into the networks. This happened in some neighborhoods, but as has been shown in several detailed studies in cities like Cleveland and Dallas, the faster DSL was brought to more affluent neighborhoods, leaving poorer neighborhoods, even today, with the oldest DSL technology.

The neighborhoods that saw upgrades saw DSL speeds between 15 Mbps and 25 Mbps. Many of these neighborhoods eventually saw speeds as fast as 50 Mbps using a technology that bonded two 25 Mbps DSLs circuits. There are numerous examples of neighborhoods with 50 Mbps DSL sitting next to ones with 3 Mbps DSL.

Verizon used a different tactic and upgraded neighborhoods to FiOS fiber. But this was also done selectively although Verizon doesn’t seem to have redlined as much as AT&T, but instead built FiOS only where the construction cost was the lowest.

In Europe, the telcos decided to complete with the cable companies and have upgraded DSL over time, with the fastest DSL today offering speeds as fast as 300 Mbps. There is talk coming out of DSL vendors talking about ways to goose DSL up to gigabit speeds (but only for short distances). The telcos here basically stopped looking at better DSL technology after the introduction of VDSL2 at least fifteen years ago.

By now the telcos should have been using profits to build fiber. AT&T has done this using the strategy of building little pockets of fiber in every community near to existing fiber splice points. However, the vast majority of rural households served by AT&T are not being offered fiber, and AT&T said recently that they have no plans to build more fiber. CenturyLink built fiber to past nearly 1 million homes a few years ago, but that also seems like a dead venture going forward. But now, in 2019, each of these telcos should have been deep into urban neighborhoods in their whole service area with fiber. Had they done so they would not be getting clobbered so badly by the cable companies that are taking away millions of DSL customers every year.

Rural America. The big telcos started abandoning rural America as much as thirty years ago. They’ve stopped maintaining copper and have not voluntarily made any investments in rural America for a long time. There was a burst of rural construction recently when the FCC gave them $11 billion to improve rural broadband to 10/1 Mbps – but that doesn’t seem to be drawing many rural subscribers.

It’s always been a massive challenge to bring the same speeds to rural America that can be provided in urban America. This is particularly so with DSL since the speeds drop drastically with distance. DSL upgrades that could benefit urban neighborhoods don’t work well in farmland. But the telcos should have been expanding fiber deeper into the network over time to shorten loop lengths. Many independent telephone companies did this the right way and they were able over time to goose rural DSL speeds up to 25 Mbps.

The big telcos should have been engaging in a long-term plan to continually shorten rural copper loop lengths. That meant building fiber, and while shortening loop lengths they should have served households close to fiber routes with fiber. By now all of the small towns in rural America should have gotten fiber.

This is what regulated telcos are supposed to do. The big telcos made vast fortunes in serving residential customers for many decades. Regulated entities are supposed to roll profits back into improving the networks as technology improves – that’s the whole point of regulating the carrier of last resort.

Unfortunately, the industry got sidetracked by competition from CLECS. This competition first manifested in competition for large business customers. The big telcos used that competition to convince regulators they should be deregulated. Over time the cable companies provided real residential competition in cities, which led to the de facto total deregulation of telcos.

In Europe, the telcos never stopped competing in cities because regulators didn’t let them quit. The telcos have upgraded to copper speeds that customers still find attractive, but the telcos all admit that the next upgrade needs to be fiber. In the US, the big telcos exerted political pressure to gain deregulation at the first hint of competition. US telcos folded and walked away from their customers rather than fighting to maintain revenues.

Rural America should never have been deregulated. Shame on every regulator in every state that voted to deregulate the big telcos in rural America. Shame on every regulator that allowed companies like Verizon palm off their rural copper to companies like Frontier – a company that cannot succeed, almost by definition.

In rural America the telcos have a physical network monopoly and the regulators should have found ways to support rural copper rather than letting the telcos walk away from it. We know this can be done by looking at the different approaches taken by the smaller independent telephone companies. These small companies took care of their copper and most have now taken the next step to upgrade to fiber to be ready for the next century.