Decommissioning Rural Copper, Part 2

In the last blog I wrote about my belief that AT&T and Verizon want out of the rural wireline business. They both have plans to largely walk away from their rural copper networks and replace landline copper services with cellular service. Today I want to talk about what regulators ought to do with those networks.

When these two giant telcos walk away from rural copper they will inevitably harm rural America. While many homes will get the ‘privilege’ of now buying highly-priced cellular-based broadband, other homes are going to find themselves without telephone service if they happen to live in one of the many cellular dead zones. Such homes will not only be unable to benefit from cellular broadband, but if they have poor cell service they will find themselves cut off from voice communications as well.

As somebody who has traveled extensively in rural America I can tell you that there are a lot more cellular dead zones than people realize. And it’s not only farms, and there are county seats in rural America where it’s difficult to get a working cellphone signal inside of buildings.

As part of this transition both companies are going to walk away from a huge amount of existing copper cable. I think this copper cable is an incredibly valuable asset and that regulators ought not to allow them to tear it down.

The copper wire network today goes almost everywhere in rural America. Congressional laws and FCC policies led to most homes in the country getting access the the copper network. These copper wires occupy a valuable space on existing telephone poles – on the majority of rural poles the only two wires are the power lines at the top and the telephone wires at the bottom.

If these copper wires are kept in place they could greatly reduce the cost of building rural fiber. It is far cheaper when building fiber to ‘lash’ the fiber onto an existing set of cables than to hang fiber from scratch. It was this construction technique that allowed Verizon to build a lot of its FiOS fiber network – they lashed fiber onto existing telephone wires. And my guess is that when Verizon decommissions urban copper they are still going to leave a lot of the copper wires in place as a guidewire for their fiber.

If these telcos are going to walk away from these copper wires, then they ought to be required to keep them in place for use by somebody else to hang fiber. Many states might force the big telcos to tear down the copper wires since they will eventually create safety hazards as they break away from poles if they aren’t maintained. But if somebody else is willing to take over that maintenance then it shouldn’t be an issue.

I can picture a regulatory process whereby some other carrier is allowed to come in and ‘claim’ the abandoned wires once they are empty of customers. That would provide fiber overbuilders or rural communities to claim this copper as an asset.

There is some salvage value to copper wires and and it’s possible, but not probable that the value of the copper could exceed the cost to tear it down. So I can see the telcos fighting such an idea as a confiscation of their assets. But these rural wires have been fully depreciated for decades and the telcos have earned back the cost of these copper lines many times over. I believe that by the act of abandoning the wires and depriving some homes of wireline service that the big telcos will have forfeited any rights they might have to the remaining assets.

Anybody claiming the abandoned copper could use it in two ways. First, in many cases there is still existing life left in the copper, as witnessed by Frontier and CenturyLink rehabbing old rural copper with upgraded DSL. Local communities or small carriers could use the copper to bring the better services that the big telcos have refused to do over the last few decades.

But more importantly these wires represent the cheapest path forward for building rural fiber. Anybody taking over the old copper can save a lot of fiber construction costs by lashing fiber onto the existing copper. If our nationwide goal is really to get better broadband to rural America, then offering abandoned copper to fiber builders might be one of the easiest tools available to help the process along.

The big telcos abandoned rural America dacades ago. They stopped doing routine maintenance on rural copper and slashed the number of rural technicians. They now want to walk away from that copper and instead force rural America to buy cellular services at inflated prices. We owe it to the folks who paid for this copper many times over to get some benefit from it and to offer an alternative to the new rural cellular monopolies.

Decommissioning Rural Copper

I’ve been watching AT&T and Verizon since I’ve been in the industry (including a short stint at Southwestern Bell in the early 80s). We are about to see both of these companies unravel their rural telco properties.

Verizon got ahead of the curve and has been selling off rural properties for a few decades, many of which ending up with Frontier. Verizon still serves some rural areas and probably has shed  half of their rural customers. But there are still big swaths or rural Verizon customers in Pennsylvania, New York, Maryland and other northeastern states. Verizon benefitted from these sell-offs by selling completely depreciated and poorly maintained networks at high prices – as can be evidenced by how much Frontier is struggling to cover their massive debts. AT&T has sold almost no rural properties and still serves gigantic rural areas in dozens of states.

Both companies are clearly on a path to tear down the remaining rural copper networks and replace them with cellular wireless networks. There are both pros and cons for these transitions for rural customers.

On the plus side, many of these rural areas have never had broadband since these big telcos never extended their DSL to their rural service areas. We know that they could have extended DSL, because we have hundreds of examples of independent telephone companies that brought DSL to all of their customers, no matter how remote. But the big companies stopped spending money on rural properties decades ago. The remaining copper is now in terrible shape and one has to imagine that cellular voice is probably often as good or better than voice over these old copper lines.

There will now many customers who can buy fixed cellular broadband. This uses the same frequencies as the broadband for smartphones, but the cellular companies are pricing it to be a little less expensive. For many households the fixed-cellular broadband will be the first real broadband alternative they have ever had.

But there are also big downsides to this shift from old copper to cellular networks. First, cellular networks are effective for only a few miles from any given cell site. Anybody who has driven in rural America knows that there are cellular dead spaces everywhere. Any customers living in the cellular dead spaces are going to be left with no communications to the outside world. They’ll lose their copper and they won’t have cellular voice or data. This will be a huge step backwards for many homes.

The big telcos will be taking advantage of the fact that, as a cellular provider, they have no obligations to try to serve everybody. One of the reasons that we had nearly ubiquitous telephone coverage in the country is that telcos were the carriers of last resort in their service areas. They were required by law to extend telephone service to all but extremely remote customers. But that obligation doesn’t apply to a cellular carrier. We already have tons of evidence that the cellular carriers make no apologies to homes that happen to live out of range of their cellular towers. With no copper landlines left we will now have rural communications dead zones. It will be hard for anybody living in these dead zones to stay there and certainly nobody is going to build new homes in a place that doesn’t have cellular service.

There is a downside even for those households that get fixed-cellular broadband. The speeds on this service are going to be slow by today’s standards, in the range of 10 – 15 Mbps for those that live relatively close to a cellular tower, but considerably slower for customers at greater distances. The real downside to getting cellular data is that the speeds are not likely to get better in rural America for many years, even decades. The whole industry is abuzz with talk about 5G cellular making a big difference, but it’s hard to see that technology making much impact in rural areas.

I think this transition away from copper is going to catch a lot of rural people by surprise. These two big telcos have already started the process of decommissioning copper and once that gets full FCC approval the speed of decommissioning copper is likely to soon accelerate. I think a lot of homes are going to be surprised when they find out that the telcos no longer have an obligation to serve them.

What’s the Next FTTP Technology?

There is a lot of debate within the industry about the direction of the next generation of last mile fiber technology. There are three possible technologies that might be adopted as the preferred next generation of electronics – NG-PON2, XGS-PON or active Ethernet. All of these technologies are capable of delivering 10 Gbps streams to customers.

Everybody agrees that the current widely deployed GPON is starting to get a little frayed around the edges. That technology delivers 2.4 Gbps downstream and 1 Gbps upstream for up to 32 customers, although most networks I work with are configured to serve 16 customers at most. All the engineers I talk to think this is still adequate technology for residential customers and I’ve never heard of a neighborhood PON being maxed out for bandwidth. But many ISPs already use something different for larger business customers that demand more bandwidth than a PON can deliver.

The GPON technology is over a decade old, which generally is a signal to the industry to look for the next generation replacement. This pressure usually starts with vendors who want to make money pushing the latest and greatest new technology – and this time it’s no different. But after taking all of the vendor hype out of the equation it’s always been the case that any new technology is only going to be accepted once that new technology achieves and industry-wide economy of scale. And that almost always means being accepted by at least one large ISP. There are a few exceptions to this, like what happened with the first generation of telephone smart switches that found success with small telcos and CLECs first – but most technologies go nowhere until a vendor is able to mass manufacture units to get the costs down.

The most talked about technology is NG-PON2 (next generation passive optical network). This technology works by having tunable lasers that can function at several different light frequencies. This would allow more than one PON to be transmitted simultaneously over the same fiber, but at different wavelengths. But that makes this a complex technology and the key issue is if this can ever be manufactured at price points that can match other alternatives.

The only major proponent of NG-PON2 today is Verizon which recently did a field trial to test the interoperability of several different vendors including Adtran, Calix, Broadcom, Cortina Access and Ericsson. Verizon seems to be touting the technology, but there is some doubt if they alone can drag the rest of the industry along. Verizon seems enamored with the idea of using the technology to provide bandwidth for the small cell sites needed for a 5G network. But the company is not building much new residential fiber. They announced they would be building a broadband network in Boston, which would be their first new construction in years, but there is speculation that a lot of that deployment will use wireless 60 GHz radios instead of fiber for the last mile.

The big question is if Verizon can create an economy of scale to get prices down for NG-PON2. The whole industry agrees that NG-PON2 is the best technical solution because it can deliver 40 Gbps to a PON while also allowing for great flexibility in assigning different customers to different wavelengths. But the best technological solution is not always the winning solution and the concern for most of the industry is cost. Today the early NG-PON2 electronics is being priced at 3 – 4 times the cost of GPON, due in part to the complexity of the technology, but also due to the lack of economy of scale without any major purchaser of the technology.

Some of the other big fiber ISPs like AT&T and Vodafone have been evaluating XGS-PON. This technology can deliver 10 Gbps downstream and 2.5 Gbps upstream – a big step up in bandwidth over GPON. The major advantage of the technology is that is uses a fixed laser which is far less complex and costly. And unlike Verizon, these two companies are building a lot more FTTH networks that Verizon.

And while all of this technology is being discussed, ISPs today are already delivering 10 Gbps data pipes to customers using active Ethernet (AON) technology. For example, US Internet in Minneapolis has been offering 10 Gbps residential service for several years. The active Ethernet technology uses lower cost electronics than most PON technologies, but still can have higher costs than GPON due to the fact that there is a dedicated pair of lasers – one at the core and one at the customer site – for each customer. A PON network instead uses one core laser to serve multiple customers.

It may be a number of years until this is resolved because most ISPs building FTTH networks are still happily buying and installing GPON. One ISP client told me that they are not worried about GPON becoming obsolete because they could double the capacity of their network at any time by simply cutting the number of customers on a neighborhood PON in half. That would mean installing more cards in the core without having to upgrade customer electronics.

From what everybody tells me GPON networks are not experiencing any serious problems. But it’s obvious as the household demand for broadband keeps doubling every three years that the day will come when these networks will experience blockages. But creative solutions like splitting the PON could keep GPON working great for a decade or two. And that might make GPON the preferred technology for a long time, regardless of the vendors strong desire to get everybody to pay to upgrade existing networks.

More Pressure on WiFi

As if we really needed more pressure put onto our public WiFi spectrum, both Verizon and AT&T are now launching Licensed Assisted Access (LAA) broadband for smartphones. This is the technology that allows cellular carriers to mix LTE spectrum with the unlicensed 5 GHz spectrum for providing cellular broadband. The LAA technology allows for the creation of ‘fatter’ data pipes by combining multiple frequencies, and the wider the data pipe the more data that makes it to the end-user customer.

When carriers combine frequencies using LAA they can theoretically create a data pipe as large as a gigabit while only using 20 MHz of licensed frequency. The extra bandwidth for this application comes mostly from the unlicensed 5 GHz band and is similar to the fastest speeds that we can experience at home using this same frequency with 802.11AC. However, such high-speed bandwidth is only useful for a short distance of perhaps 150 feet and the most practical use of LAA is to boost cellphone data signals for customers closest to a cell tower. That’s going to make LAA technology most beneficial in dense customer environments like busy downtown areas, stadiums, etc. LAA isn’t going to provide much benefit to rural cellphone towers or those along interstate highways.

Verizon recently did a demonstration of the LAA technology that achieved a data speed of 953 Mbps. They did this using three 5 GHz channels combined with one 20 megahertz channel of AWS spectrum. Verizon used a 4X4 MIMO (multiple input / multiple output) antenna array and 256 QAM modulation to achieve this speed. The industry has coined the new term of four-carrier aggregation for the technology since it combines 4 separate bands of bandwidth into one data pipe. A customer would need a specialized MIMO antenna to receive the signal and also would need to be close to the transmitter to receive this kind of speed.

Verizon is starting to update selected cell sites with the technology this month. AT&T has announced that they are going to start introducing LAA technology along with 4-way carrier aggregation by the end of this year. It’s important to note that there is a big difference between the Verizon test with 953 Mbps speeds and what customers will really achieve in the real world. There are numerous factors that will limit the benefits of the technology. First, there aren’t yet any handsets with the right antenna arrays and it’s going to take a while to introduce them. These antennas look like they will be big power eaters, meaning that handsets that try to use this bandwidth all of the time will have short battery lives. But there are more practical limitations. First is the distance limitation and many customers will be out of range of the strongest LAA signals. A cellular company is also not going to try to make this full data connection using all 4 channels to one customer for several reasons, the primary one being the availability of the 5 GHz frequency.

And that’s where the real rub comes in with this technology. The FCC approved the use of this new technology last year. They essentially gave the carriers access to the WiFi spectrum for free. The whole point of licensed spectrum is to provide data pipes for all of the many uses not made by licensed wireless carriers. WiFi is clearly the most successful achievement of the FCC over the last few decades and providing big data pipes for public use has spawned gigantic industries and it’s hard to find a house these days without a WiFi router.

The cellular carriers have paid billions of dollars for spectrum that only they can use. The rest of the public uses a few bands of ‘free’ spectrum, and uses it very effectively. To allow the cellular carriers to dip into the WiFi spectrum runs the risk of killing that spectrum for all of the other uses. The FCC supposedly is requiring that the cellular carriers not grab the 5 GHz spectrum when it’s already busy in use. But to anybody that understands how WiFi works that seems like an inadequate protection, because any of the use of this spectrum causes interference by definition.

In practical use if a user can see three or more WiFi networks they experience interference, meaning that more than one network is trying to use the same channel at the same time. It is the nature of this interference that causes the most problems with WiFi performance. When two signals are both trying to use the same channel, the WiFi standard causes all competing devices to go quiet for a short period of time, and then both restart and try to grab an open channel. If the two signals continue to interfere with each other, the delay time between restarts increases exponentially in a phenomenon called backoff. As there are more and more collisions between competing networks, the backoff increases and the performance of all devices trying to use the spectrum decays. In a network experiencing backoff the data is transmitted in short bursts between the times that the connection starts and stops from the interference.

And this means that when the cellular companies use the 5 GHz spectrum they will be interfering with the other users of that frequency. That’s what WiFi was designed to do and so the interference is unavoidable. This means other WiFi users in the immediate area around an LAA transmitter will experience more interference and it also means a degraded WiFi signal for the cellular users of the technology – and they reason they won’t get speeds even remotely close to Verizon’s demo speeds. But the spectrum is free for the cellular companies and they are going to use it, to the detriment of all of the other uses of the 5 GHz spectrum. With this decision the FCC might well have nullified the tremendous benefits that we’ve seen from the 5 GHz WiFi band.

Merger Madness

The last year was a busy one for mergers in the industry. We saw Charter gobble up Time Warner Cable and Bright House Networks. We saw CenturyLink buy Level 3 Communications. But those mergers were nothing like we see on the horizon right now. I can barely read industry news these days without reading about some rumored gigantic acquisitions.

There have always been mergers in the industry, but I can’t remember a time when there was this level of merger talk happening. This might be due in part to an administration that says it won’t oppose megamergers. It’s also being driven by Wall Street that makes a lot of money when they find the financing for a big merger. Here are just a few of the mergers being talked discussed seriously in the financial press:

Crown Castle and Lightower. This merger is already underway with Crown Castle paying $7.1 billion for Lightower. It matches up two huge fiber networks along with tower assets to make the new company the major player in the small cell deployment space, particularly in the northeast.

Discovery and Scripps. Discovery Communications announced a deal to buy Scripps Networks for about $11.9 billion. This reduces the already-small number of major programmers and Discovery will be picking up networks like the Food Network, HGTV, Travel Channel, the Cooking Channel and Great American Country.

Comcast, Altice and Charter. Citigroup issued a report that speculates that Comcast and Altice would together buy Charter and split the assets. Comcast would gain the former Time Warner cable systems with the rest going to Altice. There is also talk of Altice trying to finance the purchase of Charter on its own. But with Charter valued at about $120 billion while also carrying around $63 billion in debt that seems like a huge number to finance. This would be an amazing merger with the ink not yet dry on Charter’s merger with Time Warner.

Amazon and Dish Network. This makes sense because Amazon could finally help Dish capitalize on its 700 E-block and AWS-4 spectrum licenses. This network could be leveraged by Amazon to track trucks and packages, monitor the IoT and to control drones.

T-Mobile and Sprint. Deutsche Telecom currently owns 63% of T-Mobile and Softbank owns 82% of Sprint. A straight cashless merger would create an instantly larger company and gain major operational advantages. The FCC and the Justice Department nixed a merger between T-Mobile and AT&T a few years back, but in an environment where the cellular companies are getting into the wireless business this might sail through a lot easier today. Sprint has also been having negotiations for either a merger or some sort of partnership with Comcast and Charter.

Comcast and Verizon. There is also Wall Street speculation about Comcast buying Verizon. The big advantage would be to merge the Comcast networks with the Verizon Wireless assets. Comcast has a history of buying companies in distress and Verizon’s stock price has dipped 17% already this year. But this would still be a gigantic merger worth as much as $215 billion. There are also some major regulatory hurdles to overcome with the big overlap in the northeast between Comcast and the Verizon FiOS networks.

FirstNet – A Boon or Boondoggle?

The federal program FirstNet was born out of the tragedy of the 9/11 terrorist attacks. At the time there was a lot of negative press when it was realized that first responders from New Jersey were unable to communicate with those from New York. And the idea was born to create a nationwide platform so that all first responders could easily communicate with each other.

The FirstNet concept first tackled the concept of interoperability. There were a number of jurisdictions where interoperability was an issue then. But since 9/11 most metropolitan areas have solved the interoperability issue on their own. The fire and police departments in regions got together in the years after 9/11 and made sure they could communicate with each other. One of the easiest fixes was for first responders to add cellphones to complement the first responder radios that were the major method of communications in 2001. So the concept morphed into a discussion of finding cellular bandwidth for first responders. We’ve seen repeatedly that local cellular networks instantly get clogged during any kind of major emergency, and this means that first responders have trouble making cellphone connections just like everybody else.

Congress stepped into the discussion in 2012 and created FirstNet (First Responder Network Authority). As part of that action Congress set aside Band 14 of the 700 MHz spectrum for the exclusive use of first responders nationwide. After several reboots of the RFP process the new agency finally chose AT&T to provide a nationwide LTE network for first responders. The company was given $7 billion as the first payment towards creating the nationwide cellular network. The GAO had estimated that the final network could cost as much as $47 billion.

States were given the right to opt-in to FirstNet with zero cost to the states. In the last month or so thirteen states have chosen to be part of the effort. That means that AT&T will provide the network in those states using federal dollars.

But there is a huge question, mostly technical, of whether this network makes any sense. A lot of engineers say that FirstNet is overkill and that there are now other ways to solve the same problem. A hint of how easily this can be done came from a press release from Kansas, which just bought into FirstNet. In that release AT&T said that until FirstNet is built in the state that first responders would immediately get priority access to cell towers and by the end of this year would have preemptive access – meaning that a call attempt made by a first responder would shove somebody else off the cellular network. Providing preemptive access is a far less costly way of solving the problem. If first responders can be given preemptive access that easily, then there really is no longer a need for FirstNet.

To add fuel to the fire, Verizon just announced at the end of the next week that they would offer these same services to first responders everywhere – and with zero federal dollars. Verizon will immediately offer preemptive access to cell towers to all first responders.

Any topic having to do with first responders is always an emotional one and much of the first responder community has bought into the concept of having interference-free spectrum. But the Verizon announcement shows that the FirstNet solution is obsolete before the first piece of network is constructed.

And the FirstNet implementation comes with a big national cost. It’s clear that we need a huge amount of bandwidth to satisfy customer demands for cellular data. It seems wasteful to use a slice of prime spectrum in Band 14 of 700 MHz when it’s not needed. That spectrum is worth more to the country for providing cellular data than for handling calls from first responders. This would not be true if first responders really needed this spectrum to communicate – but the cellular companies can give them preemptive access using existing cellular spectrum. For the vast majority of time the FirstNet spectrum will sit virtually unused – at any given time in a city it might be handling hundreds of transmissions from first responders when it could instead be handling hundreds of thousands of transmissions for everybody.

There is also the rural issue to deal with. FirstNet is supposed to provide nationwide first responder access. But as somebody who travels widely in rural America, I can tell you that a lot of the AT&T LTE coverage map is bosh. There is a whole lot of rural America where cell coverage is either spotty or non-existent. When you get to a rural place you quickly come to understand the short distance that a cell signal travels from any given cellular tower. There are gaps everywhere in rural America between widely-spaced cell towers.

First responders in rural America are not going to rely on the FirstNet spectrum even if it’s freely available to them. They are more likely going to keep their current radio networks that work today, using spectrum that travels farther than the 700 MHz spectrum. I can’t help but picture a rural tragedy, such as a downed-plane, where first responders from outside the area will have no communication ability if the FirstNet signal to the needed area is weak or nonexistent.

I see this as another giant government handout to the huge carriers. You can be assured that a lot of the money going to AT&T will go to their bottom line. I hope, at least, that some of the money they are getting for FirstNet will at least improve normal cellular coverage in rural America – but I’m not holding my breath. To me this seems like another big federal program that is being spent to fix a problem that no longer exists. Local jurisdictions solved the interoperability problem in the first few years after 9/11. And the ability of cellular companies to give preemptive access to first responders means there is no reason to set aside a huge valuable slice of spectrum.

CHECK Comments for correction and update.

Consolidation of Fiber Networks

I’ve written a few recent blogs discussing the amount of fiber that’s going to be needed to support the 5G networks envisioned by Verizon and AT&T. This blog in particular cited a recent Deloitte study that estimates that the cost to build the fiber needed to support a ubiquitous 5G network nationwide would be $130 billion.

We know that adding fiber is now a high priority for Verizon. They announced in April a deal to buy over $1 billion of fiber from Corning over 3 years. (As an aside, all of the press releases and articles about that purchase say that amount buys 12.4 million miles of fiber per year, or 37.2 million miles of fiber. There are only a little over 4 million miles of roads in the US, so that obviously means miles of individual fiber strands. Pardon the interruption, but misleading statistics drive me up the wall.)

We can almost be certain that Verizon plans to build fiber for backhaul to cell sites. There are around 250,000 current cell towers in the country, but the deployment of small neighborhood cell sites is going to explode that number potentially by millions. Years ago both Verizon and AT&T elected to let other companies build and own cell towers, which spun off a major new industry. And in that process both companies largely agreed to lease fiber transport to reach those towers. But as the cell industry margins are tightening the companies are now looking to directly own as many of those fiber routes as possible to hold down lease expenses.

While Verizon plans to build a lot of fiber, they are also on an obvious path to buy existing fiber networks that supply transport to cell towers. Last year they purchased XO Communications and just last week announced they were buying a Chicago-area fiber network from Wide Open West.

I have seen several analysts speculate that Verizon will be considering more fiber purchases. Interestingly the analysts focus on the potential purchase of large ILECs like Consolidated or Cincinnati Bell, which both own a lot of fiber. But much of the fiber in these companies is last-mile fiber to reach customers, and it would be curious to see Verizon buy back into that business. Just last year they sold off a significant chunk of their FiOS fiber network to Frontier and it would be a major reversal of that strategy to turn around and invest this soon in last mile fiber. We’ve seen big companies pivot before, but this would be possibly the biggest such change of mind our industry would ever have seen.

I think it’s more likely that they will consider buying transport fiber networks rather than last-mile networks. The problem the company faces is that there are not that many big fiber providers left. CenturyLink recently purchased the largest such network from Level 3, which owns over 55,000 miles of fiber. The only other fiber transport networks left that own over 10,000 miles of fiber are Birch, Zayo, EarthLink, Cogent and Lightower/Fibertech. There are only another half dozen companies that own fiber transport networks of between 5,000 and 10,000 miles. I have to think that Verizon and AT&T have considered buying many of these companies over the last year or two.

There is one other set of big fiber networks that don’t get as much national attention. These are fiber transport networks built largely by consortiums of independent telephone companies. Most of these networks were constructed as a way for the telcos to gain cheap fiber transport to the world outside of their operating territories. Many of these smaller telcos were held hostage to incredibly expensive special access transport from the RBOCs which made it difficult for them to buy affordable Internet access. Since these networks were originally built a lot of them now have expanded throughout their operating regions and are now connected to cell towers, large businesses, governments, universities and other customers needing fiber transport.

Most of these ILEC-owned networks have joined together to form INDATEL. Here is a map showing the wide-spread footprint of INDATEL-member networks. Through this consortium many of these networks are now interconnected, providing a nearly nationwide fiber footprint. The various members have POPs in all of the biggest cities in their region but then also go to all of the smaller communities that have largely been ignored by most of the other fiber providers, with perhaps the exception of Level 3.

I have no idea if either Verizon or AT&T has considered buying these networks. For a company like Verizon these fiber routes would provide transport into many areas where they don’t have fiber today. The owners of these networks might want to explore the possibility of selling their networks. Now that the networks are in place the ILECs that built these networks are no longer isolated from the rest of the world. A sale would let them capitalize on their investment in fiber at a time when fiber networks have an all-time high valuation.

Of course, the downside to all of this is that if Verizon, AT&T and a few others like CenturyLink gobble up the few remaining independent fiber networks they will have a virtual monopoly on fiber transport. During the XO and Level 3 purchases there were a lot comments filed with regulators expressing concern about the negative impact on competition from fiber consolidation. I’d hate to see us go back to the bad old days where the only option for transport was a handful of the big telcos.

Broken Promises by Big ISPs

One of the most frustrating things for regulators has to be when giant ISPs renege on regulatory deals they’ve negotiated and don’t follow through with their promises. Books could be written listing all of the times when big ISPs have promised to do something and then never did it.

I am reminded of one such deal when I read that New York City is suing Verizon over its broken promise to bring FiOS fiber to the city. The lawsuit states that almost a million households are still unable to get FiOS, although the company had promised full coverage when they got a franchise from the city in 2008. In that agreement Verizon promised to bring fiber service to the whole city by 2014. The agreement with the city required that Verizon bring fiber, in conduit, directly in front of, behind, or otherwise adjacent to every residential building in the City.

Verizon had a similar longstanding dispute with the State of Pennsylvania. Back in 2002 the company made a promise to bring DSL service to cover 80% of the state as a prerequisite for the company being relieved of a lot of regulatory oversight by the state. But Verizon never completed a lot of the needed upgrades and huge parts of rural Pennsylvania still didn’t have DSL a decade later.

I wrote a blog a few months back about Charter in New York. There the state had found that the cable modems deployed by the company were not technically capable of delivering anything close to the speeds that the company was advertising. Charter agreed to fix the problem, but five years later had made almost no upgrades and was recently sued by the State.

I could list more examples all day long and there have been disputes all across the country with major telcos and cable companies that have made deals with regulators and then either ignored the agreements or only implemented them in a half-hearted manner.

The problem is that there are really no regulatory penalties that are big enough to penalize an ISP for not doing what it promised. There have been fines levied, but those fines are never nearly as big as the profits or savings realized by the ISPs for ignoring the agreements with regulators. For example, it’s unlikely that lawsuits or penalties will be able to force Verizon to finish the FiOS build in New York City. I am sure the company built to the parts of NYC that made economic sense and decided, for whatever reason, that there is not sufficient payback to justify building to the remaining parts of the city.

And that’s what regulators fail to recognize – big ISPs make decisions based upon the anticipated return for stockholders. I think it’s likely that in many of these cases that the big ISPs had no intention of complying with their agreements from the start. The cynical side of me says that they are often willing to take the upsides associated with these kinds of deals – be that decreased regulation or the ability to complete a merger – while knowing up front that they are unlikely to ever complete whatever they have agreed to do.

I think we are likely to see another round of broken promises in a few years as we start moving towards the end of the FCC’s CAF II program. The big telcos accepted over $9 billion over six years to improve rural broadband to speeds of at least 10 Mbps. I’ve been getting feedback from a lot of areas in the country that those deployments seem to be behind schedule. It will certainly come as no surprise if one or more of the big telcos spends the CAF II funding without bringing broadband to the promised households, or else will deliver speeds under the promised levels. The FCC recently issued a warning to carriers telling them that it expects them to fulfill the CAF II commitments – and I suspect that warning is due to the same kind of rumblings I’ve been hearing.

But ultimately the FCC doesn’t really have any way to make these telcos complete the builds. They might withhold future funding from the telcos, but as the FCC keeps eliminating regulation it is going to have very little ability to enforce the original CAF II agreements or to take any steps to really penalize the telcos.

The saddest part of these various broken promises is that millions of real people get hurt. It’s been reported that there are significant pockets of residents in urban areas like New York City that still don’t have even one broadband provider. There are huge rural swaths of the country that are desperate for any kind of broadband, which is what CAF II is supposed to deliver for the first time. But I think we need to be realistic in that big ISPs often do not meet their promises – whether deliberately or not. And perhaps it’s finally time to stop making these big deals with companies that have a history of broken promises.

The Economics of Tower Transport

Many of my clients lease towers and/or fiber transport to reach towers to wireless companies. Since most of my clients operate last-mile networks this is not usually a major source of revenue for them, but it is a significant one, and one of the more profitable things they sell.

I have been advising clients that we are in the midst big changes in the cellular industry and that they should expect payments for cell tower connectivity to start dropping. Transport providers and cell tower owners that won’t renegotiate lower prices could risk losing the business entirely.

Let’s look at AT&T as an example of this. AT&T has been aggressively pushing its vendors to lower prices. At an investor meeting last year AT&T’s president of technology operations told investors that the current industry model is not sustainable. And he is right. As I wrote in a recent blog the entire cellular industry seems to have crossed the threshold where cellular service is becoming a commodity, and that is putting huge pressure on the cellular companies to reduce costs.

Last year FierceWireless posted a letter that AT&T sent to many of vendors telling them to expect to renegotiate rates and terms. In that letter AT&T said that they would pushing for early termination of existing contracts with the expectations of lowering fees. They said they would be looking for the ability to modify or upgrade existing towers for free. And they want to eliminate any automatic price increases and instead have “rents reduced to competitive rates”.

There are two major costs for a cellular company to use somebody else’s tower. First they must lease space on towers including paying for power and space underneath to house equipment. Where AT&T doesn’t own the fiber connecting to the towers they also have to pay for fiber transport to reach the towers. And that transport is not cheap because the bandwidth they need at towers is growing at a torrid pace. Where just five years ago there were very few towers that needed more than a gigabit of bandwidth, I’ve seen rural towers where the carriers are now asking for the right over time to grow to five gigabits. And everything I read about cellular data usage tells me that demand for bandwidth at towers will continue to grow rapidly.

Many of my clients operate in rural areas and some think that their physical isolation makes them immune from any price negotiations with the wireless companies. But I think they are wrong for several reasons.

  • First, I think a lot of the billions being spend by the FCC’s CAF II program is being used to construct fiber to rural towers. AT&T is spending a most of the $2.5 billion from that program to extend fiber into rural areas. And where they build fiber they won’t need to lease it from anybody else.
  • I also suspect that the cellular companies are working with Frontier and CenturyLink, the other two big recipients of CAF II money to piggyback on their fiber expansion to reach cellular towers at a lower cost.
  • Both AT&T and Verizon are also undertaking significant fiber expansion, with one of the goals of that program to cut transport costs. I believe they are doing the math and that they will build fiber to the towers that save them money over the long-run – with those places with the most savings at the top of the list. If they sustain this kind of construction for five or ten years they will eventually be able to bypass most of the towers that they lease today. And the cellular companies should be doing this. If there are going to be lower margins in the cellular business then they ought to use their capital, while they have it, to permanently reduce operating costs.
  • I also suspect that, while AT&T and Verizon are competitors that they are cooperating to reach the more rural cell sites and have transport swap plans in place that save them both money.
  • Finally, these companies have been buying fiber network providers, like Verizon’s purchase last year of XO Communications. It would not be surprising to see them continue to buy companies that provide cell site transport.

The cellular companies and their partners don’t communicate well with smaller transport and cell tower owners. I suspect that many of clients will only get an inkling that a cellular company is going to bypass them when they get the cancellation notice of their contracts. So I have been encouraging folks to reach out to the cellular companies to renegotiate terms and prices. I think that those willing to so might be able to keep this as a long-term revenue stream, but that those that want to stick with higher historical prices will eventually get bypassed and will lose the revenue stream altogether. It’s a tough call, because some places are remote enough that they may never be bypassed – but it’s a crap shoot to guess if your own region is on the fiber-expansion list.

Our Aging Fiber Infrastructure

One thing that I rarely hear talked about is how many of our long-haul fiber networks are aging. The fiber routes that connect our largest cities were mostly built in the 1990s in a very different bandwidth environment. I have a number of clients that rely on long-haul fiber routes and the stories they tell me scare me about our future ability to move bandwidth where it’s needed.

In order to understand the problems of the long-haul networks it’s important to look back at how these fiber routes were built. Many were built by the big telcos. I can remember the ads from AT&T thirty years ago bragging how they had built the first coast-to-coast fiber network. A lot of other fiber networks were built by competitive fiber providers like MCI and Qwest, which saw an opportunity for competing against the pricing of the big telco monopolies.

A lot of the original fibers built on intercity routes were small by today’s standards. The original networks were built to carry voice and much smaller volumes of data than today and many of the fibers contain only 48 pairs of fiber.

To a large degree the big intercity fiber routes follow the same physical paths, either following interstate highways, but to an even greater extent following the railroad tracks that go between markets. Most companies that move big amounts of data want route diversity to protect against fiber cuts or disasters, yet a significant percentage of the routes between many cities are located next to fibers of rival carriers.

It’s also important to understand how the money works in these routes. The owners of the large fibers have found it to be lucrative to lease pairs of fiber to other carriers on long-term leases called IRUs (indefeasible rights to use). It’s not unusual to be able to shop for a broadband connection between primary and secondary markets, say Philadelphia and Harrisburg, and find a half-dozen different carriers. But deeper examination often shows they all share leased pairs in the same fiber sheath.

Our long-haul fiber network infrastructure is physically aging and I’ve seen a lot of evidence of network failures. There are a number of reasons for these failures. First, the quality of fiber glass today has improved by several magnitudes over glass that was made in the 1980s and 1990s. Some fiber routes are starting to show signs of cloudiness from age which kills a given fiber pair. Probably even more significant is the fact that fiber installation techniques have improved over the years. We’ve learned that if a fiber cable is stretched or stressed during installation that microscopic cracks can be formed that slowly spread over time until a fiber becomes unusable. And finally, we are seeing the expected wear and tear on networks. Poles get knocked down by weather or accidents. Contractors occasionally cut buried fibers. Every time a long-haul fiber is cut it loses a little efficiency, and over time splices can add up to become problems.

Probably the parts of the network that are in the worst shape are the electronics. It’s an expensive proposition to upgrade the bandwidth on a long-haul fiber network because that means not only changing lasers at the end points of a fiber, but at all of the repeater huts along a fiber route. Unless a fiber route is completely utilized the companies operating these routes don’t want to spend the capital dollars needed to improve bandwidth. And so they keep operating old electronics that are often many years past their expected functional lives.

Construction of new long-haul fiber networks is incredibly expensive and it’s rare to hear of any major initiative to build fiber on the big established intercity routes. Interestingly, the fiber to smaller markets is in much better shape than the fiber between NFL cities. These secondary fiber routes were often built by groups like consortiums of independent telephone companies. There were also some significant new fiber routes built using the stimulus funding in 2008.

Today a big percentage of the old intercity fiber network is owned by AT&T, Verizon and CenturyLink. They built a lot of the original network but over the years have also gobbled up many of the other companies that built fiber – and are still doing so, like with Verizon’s purchase last year of XO and CenturyLink’s purchase of Level3. I know a lot of my clients worry every time one of these mergers happens because it removes another of a small handful of actual fiber owners from the market. They are fearful that we are going to go back to the old days of monopoly pricing and poor response to service issues – the two issues that prompted most of the construction of competitive fiber routes in the first place.

A lot of the infrastructure of all types in this country is aging. Sadly, I think we need to put a lot of our long-haul fiber backbone network into the aging category.