Partners are Where You Find Them

An interesting new partnership has been formed between Windstream and Colquitt Electric Membership Corp. of Georgia to build a rural fiber network. Windstream is a large price-cap telco that recently emerged from an interesting bankruptcy. Colquitt is a rural electric cooperative.

Only high-level terms of the partnership have been released. Windstream will own the fiber network, will provide broadband and other services, and will own the customers. Colquitt will provide access and rights-of-ways on poles and Colquitt technicians will place the new fiber on poles. Colquitt will get access to some fibers on the new network to connect electric substations and other electric network components to fiber. The partnership is described as having a network that is ‘jointly built and jointly-owned’.

The area to be served is rural and is described as having around 7 people per square mile. It’s a little hard to put that statistic into perspective because the most commonly used metric in the industry for understanding density is the number of homes per mile of road – however, the area sounds sparsely populated.

The state of Georgia decided a few years ago to allow electric cooperatives to become ISPs – a restriction that was imposed years ago by legislation prompted by telco incumbents. Many states have recently lifted such restrictions in an attempt to find more solutions to solve the rural broadband gap.

Partnerships with larger price-cap telephone companies to provide fiber broadband is a new phenomenon. An argument can be made that decisions made by price-cap telcos over the years are one of the major reasons why much of rural America is still served by DSL broadband provided over old and poorly-maintained copper networks.

But we’ve seen several similar partnerships with price-cap telcos. CenturyLink has partnered with the City of Springfield, Missouri to provide fiber. Consolidated Communications has partnered with several villages in New Hampshire to build fiber. Cincinnati Bell has partnered with the Butler Rural Electric Cooperative in Ohio.

This announcement is a reminder to rural communities and electric cooperatives that broadband partners might be found in unexpected places. It’s easy for rural folks to assume that the telcos that built and have been operating the dreadful copper networks are not interested in providing better service. In this case, the network is being built in a rural community and it’s extremely unlikely that Windstream could justify investing the full cost to build fiber – the return and payback on investment would never meet corporate earnings metrics and would make no sense as an investment. However, sharing the costs with the electric cooperative must have reduced Windstream’s costs to the point where the project makes financial sense.

That is the power of partnerships. Investing all of the cost to build fiber in this case probably didn’t make financial sense to either the electric cooperative or to Windstream. Both parties have something to gain out of the transaction. Windstream gains customers who will like the broadband service on fiber. The cooperative gets a fiber network connecting substations. Both are contributing to an improved community that will benefit both companies in the long-term. We’ve seen that fiber can reinvigorate a rural community. Many people want to live in rural areas but need good broadband to work from home – having a fiber network should attract new residents and keep residents some local people from leaving the area to find better broadband.

During the last year, my advice to rural communities is to have a serious discussion with the incumbent providers. Historically I’ve always advised to not bother with the incumbents because over decades I had never seen a large incumbent telco respond to plea to improve service. This is still a rare occurrence, but this partnership, and the ones mentioned earlier illustrate that it’s worth having the discussion on the outside chance that you hit the right note and the right opportunity to get the attention of the incumbents.

Who’s Chasing RDOF Grants?

There is a veritable Who’s Who of big companies that have registered for the upcoming RDOF auction. All of the hundreds of small potential bidders to the auction have to be a bit nervous seeing the list of companies they could end up bidding against.

As a reminder, RDOF stands for Rural Digital Opportunity Fund and is an auction that starts in October that will award up to $16.4 billion in broadband funding. The money will be awarded by reverse auction in a process that favors faster technologies, but also favors those willing to take the lowest amount of grant per customer. The areas that are eligible for the funding are among the most remote places in the country, which is why the list of potential large bidders is puzzling.

There are some big cable companies on the list: Altice, Charter Communications, Cox Communications, Atlantic Broadband, Midco, and Mediacom Communications. These companies serve many of the county seats or other nearby towns to many of the RDOF areas. One has to wonder what these companies have in mind. The only one that has chased any significant federal grants in the past is Midco in Minnesota and North Dakota. Midco has been using grant money to extend fiber backhaul to connect its smallest markets, to build last-mile broadband in some tiny towns, and to build fixed wireless in rural areas surrounding its cable markets.

One has to wonder if the other cable companies have a similar plan. It’s incredibly inefficient to build traditional hybrid coaxial-fiber networks in rural areas, so it’s unlikely that the cable companies will be extending their existing networks. The RDOF auction is being done by Census blocks, which in rural areas can cover a large area. The winner of the auction for a given Census block must offer service to everybody in that block. I also have a hard time envisioning all of these big cable companies getting into the wireless business like Midco is doing, so their presence in the auction is a bit of a mystery.

Then there are the traditional large telcos including Frontier, Windstream, Consolidated Communications, and CenturyLink. These companies already serve many of the areas that are covered by the reverse auction. These are the rural areas where these companies have largely neglected the old copper wiring and either offer no broadband or dreadfully slow DSL. The minimum technology allowed to enter the auction must deliver 25/3 Mbps broadband. It’s almost painful to think that these companies would chase the funding and promise to upgrade DSL to 25/3 Mbps after these companies largely botched an upgrade to 10/1 Mbps DSL in the just-ending CAF II grants. The cynic in me says they are willing to pretend to upgrade DSL all over again if that means substantial grant money. I have to think that some of these companies are considering deploying fixed wireless. To the extent any of these companies is willing to take on new debt or use equity, they could also build fiber. None of these companies has built a substantial amount of fiber to truly rural places, but may these grants are the inducement they were waiting for.

Verizon and U.S. Cellular have registered for the auction. You have to think the cellular carriers will be deploying fixed cellular broadband like the 4G FWA product that Verizon just announced recently. These companies already have equipment on towers in many of the RDOF grant areas and would love to grab a subsidy to roll out a product they might be selling in these areas anyway.

Then there are the satellite companies SpaceX, Hughes Network Systems, and Viasat. Viasat has won federal grant money before for selling broadband from its high-altitude satellites. SpaceX is the wildcard since nobody knows anything about the pricing or real speeds they can provide. We know that Elon Musk has been lobbying the FCC to let him have a shot at the billions up for grabs in this auction.

There is another interesting wildcard with Starry. Their business plan is currently selling fixed wireless to large apartment buildings in center cities and they’ve developed a proprietary technology that’s perfect for that application. They must have something else in mind in chasing grant money in remote areas that are 180 degrees different than their normal business model. Starry founder Chet Kanojia is incredibly creative, so he probably has a new technology in mind if he wins auction funding.

There may be other big players in the auction as well since many of the registered bidders are participating under partnerships or corporations that are disguising their identity for now. I think one thing is clear and some of the rural ISPs and cooperative who think nobody else is interested in their markets will get a surprise early in the auction. These big companies didn’t register for the grant auction to sit on the sidelines.

Walking Away from Copper

It’s been clear for many years that the big telcos are looking for ways to walk away from legacy copper networks. Big telco copper is getting old and most was built in the 1950s and 1960s. All of this copper is far past the 40-year expected lives that the telcos claimed when they built the networks. Even old copper can be made to work if it was well-maintained, but the big telcos stopped doing routine maintenance on copper decades ago. For years, the big telco maintenance policy has been to patch problems without improving or fixing network issues.

In some cases, the big telcos have gone through the formal FCC process of formally retiring copper. This requires giving customers a 90-day notice that copper will be deactivated and providing customers an alternative to copper.  For example, Verizon posts notices of copper retirement on this web site. There have been no announced retirements this year, likely due to the COVID-19 pandemic, but Verizon was active last year, like in this September notification for Massachusetts. CenturyLink has made similar notices in parts of Arizona, Colorado, Minnesota, Nebraska, Oregon, Utah, and Washington.

In all of these cases, Verizon and CenturyLink made the announcements in communities where the carriers can provide fiber-to-the-home. It’s a natural technological progression to replace old copper technology with new fiber, and customers who lose copper to move to fiber have little room to complain.

But what about all of the places where the telcos never plan to offer fiber? There are still huge areas, including big parts of major cities where the telcos have no plans of migrating to fiber. What will happen to folks in regions where the copper is rotting past the point of usefulness, like is described in this article from last year in Fauquier County, Virginia? In that county the copper barely works for voice, which is sadly becoming the norm and not the exception.

There is nothing the big telcos can do with copper that has gone past the point of no return. No telco is going to replace bad copper and none of the big telcos are going to extend fiber into rural America or into urban neighborhoods where construction is too expensive. Verizon might be replacing copper with fiber around Boston, as indicated by the above filing, but the telco has no plans for building fiber in western Massachusetts, Cape Cod, or the many other places in the state where it never built FiOS fiber.

We might have gotten a glimpse into Verizon’s strategy when the company recently unveiled a 4G fixed wireless product. This 4G Home product promises to deliver 25 Mbps broadband using the 4G cellular network and Verizon could point to this product as a justification to abandon DSL over copper.

On paper, the 4G Home product will outproduce rural DSL, which typically has speeds well under 10 Mbps. But the Fauquier County article pointed out another ugly truth – much of rural America has poor cellular coverage to go along with outdated copper. The 4G Home product is not going to work for homes that are more than a mile or two from a cell tower. 4G Home is not going to be a reasonable substitute for DSL in communities like the towns on Cape Cod where the density is too high to support a lot of subscribers using 4G data as a landline data substitute – even a small customer penetration would swamp the 4G LTE network in populated areas.

AT&T has a similar fixed wireless product it introduced during the past year as the solution for meeting the company’s rural CAF II requirements. I’ve been tracking this product on the web and still don’t see local articles or chatter from many folks who have changed to the wireless product. AT&T has implemented this product to satisfy the FCC (and to keep the CAF II grant funding), but for some reason the company doesn’t seem to be pushing the product very hard.

The bottom line is these telcos will have to walk away from copper at some point within the next decade for the simple reason that the networks will stop functioning. From what I can see, both the FCC and many state regulatory commissions refuse to acknowledge that copper is dying and keep pretending that the telcos can somehow make this work. These networks are dying. The telcos might toss a bone to regulators by halfheartedly offering a wireless substitute for DSL. But the telcos are under no obligation to offer a replacement if the copper dies. Sadly, we’re going to look up five years from now and find a lot of rural homes without a telephone line and a cellular connection. There was a time when that was unthinkable, but it’s the coming reality.

Google Fiber Comes to Iowa

The City of West Des Moines recently announced a deal with Google Fiber to bring fiber to pass all 36,000 residents and businesses in the city. This is a unique business model that can best be described as open-access conduit.

The city says that the estimated cost of the construction is between $35 million and $40 million and that the construction of the network should be complete in about two-and-a-half years. The full details of the plan have not yet been released, but the press is reporting that Google Fiber will pay $2.25 per month to the city for each customer that buys service from Google Fiber.

What is most unique about this arrangement is that conduit will be built along streets and into yards and parking lots to reach every home and business. I know of many cities that lease out some empty conduit to ISPs and carriers, but the big limitation of most empty conduit is that it doesn’t provide easy access to get from the street to reach a customer. West Des Moines will be spending the money to build the conduit to serve the last hundred feet.

This business arrangement will still require Google Fiber to pull fiber throughout the entire empty conduit network – but that is far cheaper for the company than building a network from scratch. The big cost of building any fiber network is the labor needed to bring the fiber along every street – and the city has absorbed that cost. The benefit of this arrangement for Google Fiber is obvious – the company saves the cost of building a standalone fiber network in the City. It’s the cost of financing expensive networks up-front that makes ISPs hesitant to enter new markets.

From a construction perspective, I’m sure that the City is building fiber with some form of innerduct – which is a conduit with multiple interior tubes that can accommodate multiple fibers (as is shown in the picture accompanying this blog). This would allow additional ISPs to coexist in the same conduits. If the conduits built through yards also include innerduct it would make it convenient for a customer to change fiber ISPs – disconnect fiber from ISP A and connect to the fiber from ISP B.

The City is banking on other ISPs using the empty conduit because Google Fiber fees alone won’t compensate the city for the cost of the conduit. The press reported that Google Fiber has guaranteed the City a minimum payment of at least $4.5 million over 20 years. I’m sure the City is counting on Google Fiber to perform a lot better than that minimum, but even if Google Fiber connects to half of all of the customers in the City, the $2.25 monthly fee won’t repay the City’s cost of the conduit.

This business model differs significantly from the typical open-access network model. In other open-access networks, the City pays for 100% of the cost of the network and the electronics up to the side of a home or business. The typical monthly fee for an ISP to reach a customer in these open access-networks ranges between $30 and $45 per month. Those high fees invariably push ISPs into cherry-picking and only pursuing customers willing to pay high monthly rates. The $2.25 fee in West Des Moines won’t push ISPs to automatically cherry-pick or charge a lot.

Any ISP willing to come to the city has a few issues to consider. They avoid the big cost of constructing the conduit network. But a new ISP will still need to pay to blow fiber through the conduit. Any new ISP will also be competing against Google Fiber. One of the most intriguing ISPs already in the market is CenturyLink. The company has shown in Springfield, Missouri that it is willing to step outside the traditional business model and use somebody else’s network. I would have to imagine that other ISPs in the Midwest perked up at this announcement.

In announcing the network, the City said that they hoped this network would bring fiber to everybody in the City. Google Fiber doesn’t typically compete on price. Earlier this year Google Fiber discontinued its 100 Mbps broadband connection for $50. Many homes are going to find the $70 gigabit product from Google Fiber to be unaffordable. It will be interesting over time to see how the city plans on getting broadband to everybody. Even municipalities that own their own fiber network are struggling with the concept of subsidizing fiber connections below cost to make them affordable.

One thing this partnership shows is that there are still new ideas to try in the marketplace. For an open-access conduit system to be effective means attracting multiple ISPs, so this idea isn’t going to work in markets much smaller than West Des Moines. But this is another idea for cities to consider if the goal is to provide world-class broadband for citizens and businesses.

Big Regional Network Outages

T-Mobile had a major network outage last week that cut off some voice calls and most texting for nearly a whole day. The company’s explanation of the outage was provided by Neville Ray, the president of technology.

The trigger event is known to be a leased fiber circuit failure from a third party provider in the Southeast. This is something that happens on every mobile network, so we’ve worked with our vendors to build redundancy and resiliency to make sure that these types of circuit failures don’t affect customers. This redundancy failed us and resulted in an overload situation that was then compounded by other factors. This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls.

In plain English, the electronics failed on a leased circuit, and then the back-up circuit also failed. This then caused a cascade that brought down a large part of the T-Mobile network.

You may recall that something similar happened to CenturyLink about two years ago. At the time the company blamed the outage on a bad circuit card in Denver that somehow cascaded to bring down a large swatch of fiber networks in the West, including numerous 911 centers. Since that outage, there have been numerous regional outages, which is one of the reasons that Project THOR recently launched in Colorado – the cities in that region could no longer tolerate the recurring multi-hour or even day-long regional network outages,

Having electronics fail is a somewhat common event. This is particularly true on circuits provided by the big carriers which tend to push the electronics to the max and keep equipment running to the last possible moment of its useful life. Anybody visiting a major telecom hub would likely be aghast at the age of some of the electronics still being used to transmit voice and data traffic.

I can recall two of my clients that have had similar experiences in the last few years. They had a leased circuit fail and then also saw the redundant path fail as well. In both cases, it turns out that the culprit was the provider of the leased circuits, which did not provide true redundancy. Although my clients had paid for redundancy, the carrier had sold them primary and backup circuits that shared some of the same electronics at ley points in the network – and when those key points failed their whole network went down.

However, what is unusual about the two big carrier outages is that the outages somehow cascaded into big regional outages. That was largely unheard of a decade ago. This reminds more of what we saw in the past in the power grid, when power outages in one town could cascade over large areas. The power companies have been trying to remedy this situation by breaking the power grid into smaller regional networks and putting in protection so that failures can’t overwhelm the interfaces between regional networks. In essence, the power companies have been trying to introduce some of the good lessons learned over time by the big telecom companies.

But it seems that the big telecom carriers are going in the opposite direction. I talked to several retired telecom network engineers and they all made the same guess about why we are seeing big regional outages. The telecom network used to be comprised of hundreds of regional hubs. Each hub had its own staff and operations and it was physically impossible for a problem from one hub to somehow take down a neighboring hub. The worst that would happen is that routes between hubs could go dark, but the problem never moved past the original hub.

The big telcos have all had huge numbers of layoffs over the last decade, and those purges have emptied out the big companies of the technicians that built and understood the networks. Meanwhile, the companies are trying to find efficiencies to get by with smaller staffing. It appears that the efficiencies that have been found are to introduce network solutions that cover large areas or even the whole nation. This means that the identical software and technicians are now being used to control giant swaths of the network. This homogenization and central control of a network means that failure in any one place in the network might cascade into a larger problem if the centralized software and/or technicians react improperly to a local outage. It’s likely that the big outages we’re starting to routinely see are caused by a combination of the  failure of people and software systems.

A few decades ago we somewhat regular power outages that affected multiple states. At the prodding of the government, the power companies undertook a nationwide effort to stop cascading outages, and in doing so they effectively emulated the old telecom network world. They ended the ability for an electric grid to automatically interface with neighboring grids and the last major power outage that wasn’t due to weather happened in the west in 2011.

I’ve seen absolutely no regulatory recognition of the major telecom outages we’ve been seeing. Without the FCC pushing the big telcos, it’s highly likely nothing will change. It’s frustrating to watch the telecom networks deteriorate at the same time that electric companies got together and fixed their issues.

CenturyLink Expanding Fiber

CenturyLink recently announced its fiber plans for 2020 and says it will be building to pass 400,000 homes and businesses with fiber this year as a follow-up to 2019 that saw the company add 300,000 passings. Like with all big telco announcements, a bit of looking behind the scenes is needed to understand what the company is doing.

In 2017 CenturyLink was engaged in a major fiber expansion plan and built that year to pass 900,000 homes and businesses, mostly in large cities and surrounding suburbs in places like Seattle, Phoenix, and Denver. Those expansions plans were put on hold when new CEO Jeff Storey replaced the telco-minded Glen Post. One of Storey’s first announcements as CEO was that the company was no longer going to pursue capital projects with ‘infrastructure returns’ and building FTTH came to a screeching halt.

It was a lot harder than Storey might have hoped to inject the Level 3 mindset into a 100-year-old telco, and the company bogged down and stock prices dropped. Starting last year, the company started talking again about aggressively expanding its fiber network to add large buildings to the network. The company recently said it had connected to over 18,000 buildings last year. In digging deeper into things the company discussed over the last year, it seems that those buildings were a combination of multi-tenant business buildings and apartment complexes. The company also said that it was building a lot of the new fiber in 2019 to reach small cell sites.

The company also recently announced that it had added 300,000 passings as a result of the fiber expansion last year – a number that I can’t find mentioned as a goal last year. This does not mean that the company built fiber to pass 300,000 homes. Many of the passings came from the 18,000 buildings that were added to the network. CenturyLink has also entered into a contract to operate the fiber network in Springfield, MO – a network that is funded, built, and owned by the City. The 85,000 or so passings from that project seem to be included in the fiber passings claimed for 2019 and planned for 2020.

CenturyLink says it plans to add 400,000 fiber passings this year. The company is still aggressively adding buildings to the network and is also still building to small cell sites. The markets on the list for this year’s expansion include Denver, Omaha, Phoenix, Portland., Salt Lake City, Spokane, Springfield, MO, and a few others.

CenturyLink says the fiber expansion is starting to pay off. While the company lost a net of 11,000 broadband customers in the first quarter of this year, they added 60,000 subscribers with speeds of 100 Mbps or faster. Those gains were part of the industry gain of over 1.1 million broadband customers in the first quarter – at least some of which came as a result of the needs of employees and students being forced to work from home.

The company has gotten back into the infrastructure business somewhat reluctantly but now seems to have embraced some aspects of fiber expansion. CenturyLink is still bullish about adding buildings to the network and are at number four in terms of on-net buildings. I would be surprised if the fiber expansion includes any significant construction to reach single-family homes. It seems a lot more likely that the company is picking off low-hanging fruit in places where it is installing fiber to reach small cell sites or lucrative buildings. That’s the same philosophy that helped AT&T add over 12 million fiber passings over the last few years.

Jeff Storey is still adamantly painting a picture of a company that is focused on enterprise services and business applications. Any expansion into residential neighborhoods is likely a by-product of taking advantage of fiber built to pursue the primary goal.

But no matter how they are getting there, it’s good to see CenturyLink building fiber again. In 2018 it looked like they might permanently duck out of fiber construction. However, the stock market disappointment, and perhaps seeing AT&T success with limited fiber expansion convinced management some that fiber can earn more than infrastructure returns.

Enough is Enough

CenturyLink recently petitioned the FCC to allow them to be late in implementing the CAF II upgrades where the FCC doled out $11 billion to upgrade rural broadband speeds to 10/1 Mbps. The ostensible reason for the delay is the COVID-19 pandemic, but CenturyLink was already behind and notified the FCC earlier this year that they hadn’t completed their 2019 CAF II installation in 23 out of 33 states.

I say enough is enough. It’s time for the FCC to demand a reckoning of CAF II and begin handing out draconian penalties to the telcos that didn’t meet their obligations. I’m positive that if this was assessed fairly that the FCC will find that the vast majority of big telco customers have never gotten an upgrade to 10/1 Mbps.

Let’s start by looking at CenturyLink’s request. There is no reasonable explanation they can offer for not meeting their obligations in 2019. That was the fourth of a five-year buildout obligation, and the company has known for years what’s needed to be done – and they had the federal money in their pocket to make the upgrades. The claim for this year is also largely bogus. I have a lot of clients that are being cautious now about entering customer premises, but I don’t know any carrier that has stopped doing work outside of customer homes. I can’t think of any practical reason that COVID-19 would cause a delay for CenturyLink. Even if they upgrade somebody’s DSL, they could mail them a new modem – telcos have been having customers self-install DSL modems for twenty years.

It’s time to stop the pretense that CenturyLink or the other big telcos have been busy upgrading rural DSL. I don’t know anybody who thinks that’s happened. I have anecdotal evidence that it hasn’t, My company has been helping rural counties with broadband feasibility studies for many years. In the last four years, we’ve been asking rural customers to take speed tests – and I’ve never seen even one rural DSL connection that transmits at a speed of 10/1 Mbps. I’ve haven’t seen many that have tested above 5 Mbps. I’ve seen a whole lot that tested at less than 3, 2 or even 1 Mbps. Many of these tests have been in areas that are supposed to have CAF II upgrades.

I’ve also never talked to any County officials who have heard from the telcos that their county got rural broadband upgrades. One would think the telcos would brag locally when they were finished with upgrades as a pitch to get new customers. After all, customers that have only had slow DSL or satellite service should be flocking to 10/1 DSL. I’ve also not seen a marketing campaign talking about faster speeds due to CAF II. I’ve been searching the web for years to find testimonials from customers talking about their free upgrade to 10/1 Mbps, but I’ve never found anybody who has ever said that. This is not to say there have been zero upgrades in the CAF II areas, but I see no evidence of widespread upgrades.

The reality is that CenturyLink got new leadership a few years ago who immediately announced that the company was going to stop making ‘infrastructure return’ investments. We have Frontier that miraculously recently found 16,000 Census blocks that now have speeds of at least 25/3 Mbps when I’m still looking for proof that they upgraded places to 10/1 Mbps. Go interview folks in West Virginia if you think they’ve made any CAF II upgrades.

The FCC has a choice now. They can wimp out and grant the delay that CenturyLink is requesting, or the agency can come down on the side of rural broadband. There is no middle ground when it comes to CAF II. This FCC didn’t make the original CAF II decision – but they are the ones that are supposed to make sure the upgrades are done, and they are supposed to be penalizing telcos that failed to make the upgrades.

The response to CenturyLink’s request should be a giant penalty for missing the 2019 deadlines and a reminder that the company is still on the hook for 2020 unless they want more fines.

The FCC also needs to aggressively start testing in the areas that have supposedly gotten CAF II upgrades. This doesn’t have to be a big expensive testing program. We know exactly where CAF II should have been implemented – the FCC has made it easy by overlaying the CAF II footprint over Google maps. The FCC could ask County administrators across the US to solicit a speed test at CAF II locations – the Counties would be glad to oblige. If the FCC wanted to know the truth about CAF II they could get massive feedback within a few weeks about the abject failure of the CAF II program.

The ultimate penalty ought to be the return of CAF II money to the Universal Service Fund for areas that aren’t upgraded to 10/1 Mbps. Then the money could finally be given to somebody that will upgrade to real broadband. The CAF II program was ill-conceived, but the big telcos should have used that money to bring rural speeds up to 10/1 Mbps. Had they done so, we’d have millions of more homes that wouldn’t be struggling so hard during COVID-19. This FCC has a chance to do their job and set things right.

Funding Middle-mile Fiber

A decade ago, there were a lot of federal grants given to build middle mile fiber. That’s the fiber that connects communities and that provides a path between a community and a connection to the Internet. Ideally, backbone fiber also provides a diverse route with ring electronics so that if one of the fibers serving a community is cut the broadband connection to the community keeps working.

It’s not as easy to find grants for backbone fiber today. For instance, the $16.4 RDOF grant for later this year is aimed at bringing last-mile fiber to remote places in the country but doesn’t let an applicant file for money to build just backbone fiber to reach those same remote communities. It’s almost as if the FCC somehow thinks that most of America is somehow now in reach of a reliable connection to the Internet.

A new network called Project THOR recently launched in northwest Colorado that is purely a backbone project and that shows the continued need for middle-mile fiber. Project  THOR is a consortium of 14 communities that came together because they regularly suffered major broadband outages any time there was a middle-mile fiber cut in the region or an electronics problem at CenturyLink, the backbone provider for the entire section of the state. Network outages can be devastating and mean non-functional 911 centers, hospitals with no broadband, city governments that are crippled, and business districts that can no longer take credit cards or use the Internet.

The cities and towns in the region selected Mammoth Networks to create and operate a new middle-mile fiber network. The initial network is cobbled mostly with dark fiber leased from Colorado DOT, other networks like Strata, and lit fiber from CenturyLink, Comcast, and Zayo. The plan is to eventually replace lit fiber with dark fiber or constructed fiber. Mammoth oversaw the construction of lateral fibers inside of communities and also designed and implemented the electronics network. The State of Colorado Department of Local Affairs funded the lateral construction and half of the equipment purchases through a broadband grant.

The communities are free to use the network in any way they see fit. The Project THOR network terminates at a meet-me center created in each community. Several of the communities on the new network have already built fiber-to-the-home and the new network provides Internet redundancy. Other communities located the meet-me room at a hospital or other critical facility so that they’d see an immediate benefit from the network.

Project THOR brings two advantages to the region. First, the network is designed to carry up to 400 Gbps – much more capacity than any existing fiber in the region. Mammoth Networks was also able to string together routes that provide diversity for each city to protect against fiber cuts. A single fiber cut on the Project THOR routes won’t interrupt service to any of the member communities.

There was no better evidence of the effectiveness of Project THOR than when a CenturyLink fiber outage hit the region a few days after Project THOR was activated. On April 10, there was a 6.5-hour outage, and because of Project THOR, the 911 PSAP in Aspen, hospitals in Granby and Kremmling, and the city governments in Aspen and Glenwood Springs stayed operational – but would have lost broadband service without Project THOR. The Project THOR route was the only network to stay functional in the region during the outage.

It’s common knowledge that the large incumbent telcos haven’t put any money into last-mile broadband in rural areas – but the same thing is true for middle-mile fiber. What’s most amazing about Project THOR is that CenturyLink could easily be providing much of the same redundancy and quality of service that the new network offers. However, the company doesn’t seem interested in making the needed investments in diverse fiber routes or the associated electronics.

There are huge areas of the country that suffer from inadequate middle-mile fiber routes. It would be great if there was a grant program aimed specifically for middle-mile fiber. The need is there because existing middle-mile fibers are often not adequate for today’s bandwidth needs and are definitely not ready for the increased bandwidth needs of the future. Most incumbent middle mile has little redundancy, leading to regular Internet outages. It’s also not unusual to find relatively ancient electronics on middle-mile routes in rural areas.

Project THOR is an example of cities that banded together to fix a common issue – in this case, regular and extended Internet outages. In the ideal world, the incumbents would fix such issues because it’s the right thing to do. However, the lack of capital spending on rural broadband affects middle-mile fiber as much as if impacts last-mile fiber – both are inadequate in most rural areas.

Old Regulation Rears its Head

The way that we regulate telecom services is interesting. The FCC has effectively eliminated federal regulation of broadband, the service that over 90% of households now use. Meanwhile, landline telephone service, the telecom product that is used by an ever-decreasing number of homes is still heavily regulated.

The target of much of the remaining regulation are the big telephone companies that still operate large copper networks. It’s easy to bash the big telephone companies because of the poor quality of services offered on those copper networks, and I’ve done so many times in this blog.

When you stop and think about it, those companies are still using copper networks built in the 50s, 60s, and 70s. If the telcos had been good stewards of those networks and maintained them meticulously those networks would still be 50 to 70 years old, and older than the 35-40 year expected life for the networks. The big telcos largely ignored maintenance of copper for the last 30 years or more, and frankly, it’s a miracle that the old copper networks are still working.

Perhaps the oddest aspect of telephone regulation is that a regulatory body will occasionally punish a big telco for still being in the copper business. A good example is a proceeding in New Mexico last fall where CenturyLink asked to be deregulated for landline telephone services. This doesn’t mean that they would stop offering the services, but rather that many of the old regulations put in place at the heyday of the telephone monopolies would be excused. Most states have deregulated the big telcos from a lot of the old telephone rules.

The New Mexico Public Regulations Commission (NMPRC) rejected the request and said that CenturyLink had not demonstrated that there was ‘effective competition’ for residential telephone service. It’s hard to find any way to defend that decision. First, in many states, there are now more residential telephone customers using cable company telephone services than the old telephone company copper. Interestingly, cable companies face almost no regulation in providing telephone service and any cable company in New Mexico does not live under the same rules that CenturyLink must follow.

Further, the latest surveys I’ve seen show that 96% of US adults now have a cell phone. It’s hard to say with a straight face that cellular service is not a direct competitor to landline telephone service. Considering the big recent stir at the FCC where cellular 4G coverage maps were shown to be largely fictional, perhaps a lot of rural New Mexico doesn’t have cellular coverage – and perhaps that’s what drove the Commission’s decision. It’s worth noting  that cellular companies are also not as heavily regulated as landline telephone providers.

The regulation that is most relevant in this case is the obligation to be the carrier of last resort. The telcos like CenturyLink are still expected, within some regulatory exceptions, to provide service to anybody who asks for service. That obligation doesn’t extend to the cable companies, to the cellular companies, or even to rural broadband – just to telephone service.

I have no doubt that there are rural homes in the state for which CenturyLink is the only communications link to the world. In areas where there is no cellular service and where the cable companies refuse to build networks, there are rural homes that rely on CenturyLink and other telcos to keep them connected. The regulatory question that must be asked is if such homes are sufficient reason to still strongly regulate telephone service in a state. Hopefully, the number of homes without cellular service will decrease significantly when the FCC awards the $9 billion in the 5G Fund program to extend cellular service to more remote communities.

It’s not an easy question to answer. We know CenturyLink could have done a better job of taking care of their copper. In this country, the smaller independent telephone companies did the needed maintenance to keep copper in the best shape possible. We saw the same thing in Germany where the copper networks were built at the same time as US copper, but which have been maintained better.

But in this country, most of the smaller telcos have already replaced, or have plans to replace the old copper with fiber. In Germany, there are vigorous public debates on the topic, with engineers saying that the copper networks are not likely to last more than another decade. Where copper remains the Germans have invested in the fastest DSL possible – something the big telcos here inexplicably have not done.

To some degree the decision in New Mexico is meaningless. No regulatory decision can make the old copper perform better or last longer, so there are not many practical ramifications of the Commission’s decision. CenturyLink didn’t even own these networks for most of the years when the maintenance wasn’t done – although they have likely cut back further on maintenance in recent years, as have the other big telcos.

I’m not highlighting New Mexico for this issue because many other states have made similar regulatory decisions. Regulators are rightfully mad at the big telcos for neglecting copper, and even madder that there are no plans to upgrade the copper to something better. But the time for regulators to do something about this was twenty and thirty years ago. The copper wires in New Mexico are going to die, and at some future date the networks will go dark. The regulators can choose to regulate copper down to the last day of the last customer – but to a large degree, the remaining regulations don’t mean a whole lot.

CenturyLink Ready to Launch Gigabit Broadband in Springfield, MO

It’s rare to be surprised by events in the telecom world. The announcement last summer that CenturyLink will be an ISP on a city-owned fiber network in Springfield MO was one of the most surprising things I’ve heard since the announcement years ago that Google was going to become a gigabit ISP. The joint venture has been progressing and CenturyLink says it should be adding customers in the community this spring.

The partnership between the city and CenturyLink is interesting:

  • CenturyLink has agreed to lease the network over 15-years at a payment that made the city comfortable enough to build the network. The city says they won’t have to raise electric rates since the lease revenue stream justifies the cost of the new $120 million fiber expansion.
  • The city is providing dark fiber and CenturyLink will provide all of the electronics. There have been no public announcements saying which party pays for the fiber drops. Since this is being touted as an expansion of smart-grid, it would make sense that the city owns the drops.
  • The arrangement is described as non-exclusive, meaning that other ISPs are free to serve on the network. The announcements don’t say if CenturyLink gets a head-start over other ISPs through some period of exclusivity before open access kicks in. That’s been the case in similar arrangements.
  • CenturyLink is offering $65 gigabit broadband ‘for-life’ with a guarantee that the price will never be increased. Speeds are advertised ‘up to 940 Mbps. In other CenturyLink markets the gigabit product requires paperless billing and prepayment with a credit card or bank debit. CenturyLink charges $5 for an optional WiFi modem.

There are a few other similar well-known arrangements in the industry. This is similar to the Google Fiber arrangement with Huntsville, Alabama. It’s similar to the Ting arrangement in Westminster, Maryland and Charlottesville, Virginia. What’s unusual and surprising about this deal is that it’s with one of the big incumbent telcos. However, CenturyLink is not the incumbent in Springfield and enters the market purely as an outside ISP. CenturyLink will be competing side-by-side with AT&T, the first instance of two large incumbents telcos competing in a residential market. The other competitor and the incumbent cable provider in Springfield is Mediacom.

There are some in the industry touting this as a new paradigm for bringing gigabit fiber – but I’m not sure that is so. Like with any business model, all of the facts and the numbers must line up for any market to be a good target for overbuilding with fiber. It’s possible that there are unique characteristics of Springfield that might make this model hard to replicate in most other places.

Springfield owns a municipal electric utility and the utility decided years ago to build fiber to serve its own needs and to bring fiber to businesses in the city. The city started this new venture already owning 700 miles of fiber – much of which will likely be the backbone for building the last-mile for this venture. Springfield is also touting this as a smart grid initiative, meaning the electric utility is likely picking up a piece of the cost of the new fiber construction. There is a good chance that the math would not look nearly so favorable for a city without an electric utility – because in that case the venture would be starting with no existing fiber and the new fiber venture would have to absorb 100% of the cost of the new construction. I’ve looked at this lease model for cities that don’t own existing fiber or an electric utility and the math is often not pretty.

Don’t read those last statements as a criticism of the fiber lease model, but rather just as a recognition that all of the financial factors must align just right for this kind of venture to work. Any city owning an electric utility ought to do the math and consider this model. Cities with low construction costs for fiber might also be good candidates.

The surprising part of this arrangement is that this is being done by CenturyLink. This is an incumbent telco that is well known throughout rural America for operating lousy copper networks. The company has been ignoring the customers in rural markets, and CenturyLink customers living in rural Missouri can’t be thrilled to hear that the company will be offering gigabit fiber in a new market while continuing to ignore their broadband plight. CenturyLink is not going to sink a lot of capital in Springfield, but it’s paying for the cost of electronics and installation.

I have to give CenturyLink credit for tackling this venture. They were building fiber-to-the-home networks before Jeff Storey, the new CEO put a kibosh on spending capital for projects earning ‘infrastructure returns’. The FTTP businesses is an economy of scale business and CenturyLink can take advantage of the staff and platforms they already have in place to operate efficiently in Springfield. Since this is dark fiber the company can still do everything the CenturyLink way – which is an important factor for a big telco. We’ll have to wait to see if this is a new business line for CenturyLink or if Springfield is a unique case.