Google Fiber Leaving Louisville

Most readers have probably heard by now that Google Fiber is leaving Louisville because of failures with their fiber network. They are giving customers two months of free service and sending them back to the incumbent ISPs in the city. The company used a construction technique called micro-trenching where they cut a tiny slit in the road, one inch wide and few inches deep to carry the fiber. Only a year after construction the fiber is popping out of the micro-trenches all over the city.

Everybody I’ve talked to is guessing that it’s a simple case of ice heaving. While a micro-trench is sealed, it’s likely that small amounts of moisture seep into the sealed micro-trench and freezes when it gets cold. The first freeze would create tiny cracks, and with each subsequent freeze the cracks would get a little larger until the trench finally fills up with water, fully freezes and ejects the fill material. The only way to stop this would be to find a permanent seal that never lets in moisture. That sounds like a tall task in a city like Louisville that might freeze and thaw practically every night during the winter.

Nobody other than AT&T or Charter can be happy about this. The reason that Google Fiber elected to use micro-trenching is that both big ISPs fought tooth and nail to block Google Fiber from putting fiber on the utility poles in the city. The AT&T suit was resolved in Google’s favor, with the Charter one is still in court. Perhaps Google Fiber should have just waited out the lawsuits – but the business pressure was there to get something done. Unfortunately, the big ISPs are being rewarded for their intransigence.

One obvious lesson learned is not to launch a new network using an untried and untested construction technique. In this case, the micro-trenches didn’t just fail, they failed spectacularly, in the worst way imaginable. Google Fiber says the only fix for the problem would be to build the network again from scratch, which makes no financial sense.

Certainly, the whole industry is going to now be extremely leery about micro-trenching, but there is a larger lesson to be learned from this. For example, I’ve heard from several small ISPs who are ready to leap into the 5G game and build networks using millimeter wave radios installed on poles. This is every bit a new and untested technology like micro-trenching. I’m not predicting that anybody pursuing that business plan will fail – but I can assuredly promise that they will run into unanticipated problems.

Over my career, I can’t think of a single example where an ISP that took a chance on a cutting-edge technology didn’t have big problems – and some of those problems were just as catastrophic as what Google Fiber just ran into. For example, I can remember half a dozen companies that tried to deploy broadband networks using the LMDS spectrum. I remember one case where the radios literally never worked and the venture lost their $2 million investment. I remember several others where the radios had glitches that caused major customer outages and were largely a market disaster.

One thing that I’ve seen over and over is that telecom vendors take shortcuts. When they introduce a new technology they are under extreme pressure to get it to market and drive new revenues. Ideally, a vendor would hold small field trials of new technology for a few years to work out the bugs. But if a vendor finds an ISP willing to take a chance on a beta technology, they are happy to let the customers of that ISP be the real guinea pigs for the technology, and for the ISP to take the hit for the ensuing problems.

I can cite similar stories for the first generation of other technologies including the first generation of DSL, WiFi mesh networks, PON fiber-to-the-home and IPTV. The companies that were the first pioneers deploying these technologies had costly and sometimes deadly problems. So perhaps the lesson learned is that pioneers pay a price. I’m sure that this failure of micro-trenching will result in changing or abandoning the technique. Perhaps we’ll learn to not use micro-trenches in certain climates. Or perhaps they’ll find a way to seal the micro-trenches against humidity. But none of those future solutions will make up for Google Fiber’s spectacular failure.

The real victims of this situation are the households in Louisville who had changed to Google Fiber – and everybody else in the City. Because of Google Fiber’s lower prices, both Charter and AT&T lowered prices everywhere in the city. You can bet it’s not going to take long to get the market back to full prices. Any customers crawling back to the incumbents from Google Fiber can probably expect to pay full price immediately – there is no real incentive to give them a low-price deal. As a whole, every household in the City is going to be spending $10 or $20 more per month for broadband – which is a significant penalty on the local economy.

The Louisville Pole Attachment Lawsuit

There has been a major legislative push lately to make it easier for wireless companies to get onto poles in order to deploy the small cell sites needed for 5G deployment. AT&T and Verizon have been leading the fight for easier access and there have been attempts at both the federal and state level to enact ‘one-touch’ rules. Proposed legislation not only sets a low price for compensating pole owners, but proposed legislation also removes the ability for pole owners or municipalities to slow down wireless deployments.

There is a lot of debate in the industry about the one-touch issue. As I have discussed in various blogs, issues with getting onto poles is still one of the major roadblocks to many fiber deployments. And from the examples cited by the cellular carriers they are seeing huge delays in deploying urban small cell sites.

Like any debate there are legitimate issues to be considered on both sides of the issues. Proponents of one-touch cite the extraordinary costs of wading through the paperwork-heavy pole attachment process as well as the dollar and cents costs of delaying construction projects.

But on the other side are pole owners and current networks hung on wires. Carriers are legitimately worried about safety issues for their technicians if large boxes the size of refrigerators are hung on poles without constraint. They legitimately worry about how such devices could cause problems during repairs from storm damage. And carriers are also worried about network outages if a new attacher is allowed and able to move their wires without their knowledge or permission.

A court decision a few weeks ago might be a first step into putting some clarity to the issue. In that suit AT&T had sued the City of Louisville in order to stop them from passing a one-touch make-ready ordinance. The ordinance was aimed at making it easier for Google Fiber and other competitive providers to get onto poles in the City. The City of Louisville owns most of the poles in the city and the City has been working with Google Fiber to deploy a fiber network to everybody in the City.

You have to let the irony of AT&T’s lawsuit sink in for a minute. This is a company that is spending millions right now lobbying for one-touch rules. AT&T not only wants to deploy small cell sites, but they are also in the process of building a huge amount of fiber to support those sites. And yet AT&T felt compelled to fight against the very kind of ordinance they are promoting because it would help one of their competitors.

It turns out that not all one-touch ordinances are the same. The ordinances that AT&T and Verizon are pushing are crafted very carefully to help them while still not making it quite so easy for their competitors. The Louisville ordinance made it easier for any new attacher to get onto poles, including AT&T.

The US District Court Judge of Kentucky completely rejected all of AT&T’s claims and tossed the lawsuit. The court basically said that all of AT&T’s claims in the suit were false. It’s ironic that many of the issues raised by the City in defense of the suit sound the same as the claims that AT&T makes elsewhere when lobbying for one-touch legislation.

I’ve always said that being in the regulatory department at AT&T has to be the hardest job in our industry. It’s a company that wears too many hats. AT&T owns a huge monopoly landline network and wants to protect itself from competitors. In some markets AT&T is a major pole owner. AT&T is also a huge wireless company that now wants access to poles. And AT&T is a huge builder of fiber, much of it now outside of its monopoly telco territory.

Any regulatory position the company takes to benefit one of these business lines is likely to not be in the best interest of other parts of the company. When looking at the big picture one has to think that AT&T will get far more benefit than harm from one-touch rules. Such rules will make it a lot easier to build more fiber and to deploy cell sites. And yet, a company with this many tentacles in the industry could not restrain itself from filing a lawsuit that probably was not in its own best long-term interest. The monopoly side of the company felt it could not sit back and let a competitor like Google Fiber build without the company taking steps to slow them down.

Who Controls Access to Poles?

telephone cablesAT&T has sued the City of Louisville, KY over a recent ordinance that amends the rules about providing access to poles to a carrier that wants to build fiber. Louisville is hoping to attract Google or some other fiber overbuilder to the city.

But there has been no announcement that any such deal is in place. It seems the city is trying to make it more attractive for a fiber overbuilder to come to the city and so they passed an ordinance that allows a new fiber builder relatively fast access to poles. The ordinance gives a new fiber builder the right to rearrange or relocate existing wires on poles if the other wire owners on the poles don’t act to do so within 30 days.

AT&T opposes the measure, and their court case says, “The Ordinance thus purports to permit a third party… to temporarily seize AT&T’s property, and to alter or relocate AT&T’s property, without AT&T’s consent and, in most circumstances, without prior notice to AT&T.” They argue that a new attacher will cause service outages and create other problems with their network.

The real issue at hand in the case is if a City has the right to make rules concerning poles. Today there are basic pole rules issued by the FCC that lays forth the fact that a competitive telecom provider must be given access to existing poles, ducts and conduits. Such rights were provided by the Telecommunications Act of 1996. In reading the FCC rules you might think that a new attacher already has the rights that are being granted by Louisville. The FCC rules allow a new attacher to go ahead and put their wires on poles if the pole owners don’t act quickly enough to process the needed paperwork to allow this.

But the rub comes in when there is not a clear space on an existing pole. There are FCC and national electrical standards that require that there be certain spacing between different kinds of cables on poles, mostly to protect the safety of those that have to work in that space. If you’ve ever looked up at poles much you’ll notice that it’s not unusual for the distances between the different utilities to vary widely from pole to pole, meaning that whoever hung the cables was not paying a lot of attention to the spacing.

In the industry, when there is not enough of a gap to accommodate a new attacher, the existing wire owners have to move their wires to create the needed space. If there is not enough space after such a rearrangement then a new taller pole must be erected and the wires all moved to the new pole. The new attacher is on the hook for all rearrangement costs. This process is called ‘make ready’ work and is one of the major costs of getting onto poles in busy urban environments.

The FCC has granted states the right to make additional rules concerning pole attachments, and many states have done so. This lawsuit asks if a city has the same right to make pole attachment rules as is granted to the states – and so this is basically a jurisdiction issue. It’s the kind of issue that probably is going to have to eventually go to the Supreme Court if the loser of this first suit doesn’t like the court’s answer.

To put all of this into perspective, pole issues have often been one of the biggest problems for new telecom providers. Back in the late 1990s I had one client that wanted to get on about 10,000 poles and was told by the local electric company that they were only willing to process paperwork for about a hundred poles per week. I had another client back in that same time frame that was told by a rural electric company that they just didn’t have the time to process any pole attachment requests.

And as you can imagine, when getting on poles bogs down, a new fiber project also bogs down. This can be extremely costly for the company making the expansion because they will have already begun spending the money to build the new network and they will have a pressing need to start generating revenues to pay for it.

Across the country the conditions of poles vary widely. In some cities the poles are relatively short and they are crammed full of wires. In other cities the poles are taller and do not require much make ready work for a new attacher. But when the poles are not ready for a new attacher this can be a costly and time-consuming process. It’s going to be interesting to see if the courts allow a city to get involved in this issue in the same way that states can.