Regulation and Capital Spending

At the recent Mobile World Congress, FCC Chairman Ajit Pai said that one of his reasons he wants to reverse Title II regulation is that it has had a drastic impact on capital spending by ISPs. He says that the new regulations have been a disincentive for the ISPs to invest in broadband.

The Chairman bases that position on statistics provided by USTelecom which are based upon work done by Hal Singer, a Senior Fellow at GW Institute for Public Policy. Mr. Singer created the following table that shows the domestic capital spending for the big ISPs for 2014 through 2016. And indeed, this table shows a 5.6% drop, or $3.6 billion a year from 2014 to 2016 – which Mr. Singer attributes to Title II regulation.

2014

2015

2016

AT&T $21.1 $17.3 $17.8
Verizon $17.2 $17.8 $17.1
Comcast $6.4 $7.1 $7.7
Sprint $3.8 $3.9 $1.4
Time Warner Cable $4.1 $4.4 $3.8
T-Mobile $4.3 $4.7 $4.7
CenturyLink $3.0 $2.9 $3.0
Charter $2.2 $1.9 $3.1
Cablevision $0.9 $0.8 $0.6
Frontier $0.6 $0.7 $1.3
US Cellular $0.6 $0.5 $0.5
Suddenlink $0.3 $0.4 $0.3
   Total $64.6 $62.4 $61.0

But like with all statistics, it’s not hard to draw different conclusions from the same set of numbers. For example, all of the drop in capital spending can be attributed to AT&T and Sprint. Taking those companies out of the table shows that capital spending for the other big ISPs is up $2.1 billion or 5% from 2014 to 2016.

So what’s going on with AT&T? There are a number of reasons for their change in capital spending:

·         During these same years the company made massive capital investments in DirecTV ($3 billion over the last few years) and also on the company’s purchase and expansion of its cellular network into Mexico ($3 billion over 4 years). Those numbers are not included in the above table and it’s easy to argue that the company just set different priorities and diverted normal domestic capital to these two giant ventures. If you add those capital expenditures into the table then AT&T’s capital spending has grown – just not their ‘domestic’ spending on traditional broadband.

·         AT&T has been making a huge effort to update its cellular network using software defined networking (SDN) as described at this AT&T website. They have been decommissioning traditional hardware at cell sites and installing much less expensive, off-the-shelf routers that can now control the cell sites from centralized data centers. They have already converted over half of their cell sites and this upgrade means vastly reduced spending on traditional cell site electronics. The company has been bragging about this shift to investors for several years.

·         AT&T has also retracted from expanding traditional big tower cell sites. For a number of years AT&T has been spending money to get fiber to its more remote cell sites, and that upgrade is largely done.

Sprint can also be easily explained. This is a company in trouble and that has been well documented over the last few years. A number of attempts to find a buyer has fallen through. What’s not shown on this table is that in 2013 (the year before the table begins) Sprint spent $6.4 billion on capital in a massive system-wide upgrade to LTE. Since then the company has very publicly stated that they are cutting capital spending to conserve cash. The company is only expanding now with carefully selected small cell deployments. But the company is clearly in network maintenance mode and is spending only what is needed to keep the cell sites functioning. Also included in the drop in spending is a change in the way that Sprint treats leased cellphones – they used to capitalize the phones and they now expense them.

There are going to be further decreases in future telecom capital spending across the industry. I expect capital spending for all four big wireless companies to keep decreasing due to efficiencies from SDN. We are now seeing a burst of spending from cable companies due to upgrades to DOCSIS 3.1, but when that’s done I would expect a significant decline in their capital spending as well. We are entering a time when improvements in software will lower the need for new hardware – not just in telecom, but in many other sectors as well.

I have always been annoyed when statistics are used to falsely justify public policy. There is no evidence that the big ISPS have changed their spending habits in any drastic way due to Title II regulations. The argument that Title II has affected capital spending comes directly from constant press releases from USTelecom, and the FCC Chairman should be above repeating arguments from lobbyists. If the FCC wants to undo Title II then it should just do it – there are a number of valid reasons why this might be good policy. But it’s disingenuous to cook up false reasons for why the change is needed.

Technology and Telecom Jobs

PoleIn case you haven’t noticed, the big companies in the industry are cutting a lot of jobs – maybe the biggest job cuts ever in the industry. These cuts are due to a variety of reasons, but technology change is a big contributor.

There have been a number of announced staff cuts by the big telecom vendors. Cisco recently announced it would cut back as many as 5,500 jobs, or about 7% of its global workforce. Cisco’s job cuts are mostly due to the Open Compute Project where the big data center owners like Facebook, Amazon, Google, Microsoft and others have turned to a model of developing and directly manufacturing their own routers and switches and data center gear. Cloud data services are meanwhile wiping out the need for corporate data centers as companies are moving most of their computing processes to the much more efficient cloud. Even customers that are still buying Cisco boxes are cutting back since the technology now provides a huge increase of capacity over older technology and they need fewer routers and switches.

Ericsson has laid off around 3,000 employees due to falling business. The biggest culprit for them is SDNs (Software Defined Networks). Most of the layoffs are related to cell site electronics. The big cellular companies are actively converting their cell sites to centralized control with the brains in the core. This will enable these companies to make one change and have it instantly implemented in tens of thousands of cell sites. Today that process requires upgrading the brains at each cell site and also involves a horde of technicians to travel to and update each site.

Nokia plans to layoff at least 3,000 employees and maybe more. Part of these layoffs are due to final integration with the purchase of Alcatel-Lucent, but the layoffs also have to do with the technology changes that are affecting every vendor.

Cuts at operating carriers are likely to be a lot larger. A recent article published in the New York Times reported that internal projections from inside AT&T had the company planning to eliminate as many as 30% of their jobs over the next few years, which would be 80,000 people and the biggest telco layoff ever. The company has never officially mentioned a number but top AT&T officials have been warning all year that many of the job functions at the company are going to disappear and that only nimble employees willing to retrain have any hope of retaining a long-term job.

AT&T will be shedding jobs for several reasons. One is the big reduction is technicians needed to upgrade cell sites. But an even bigger reason is the company’s plans to decommission and walk away from huge amounts of its copper network. There is no way to know if the 80,000 number is valid, but even a reduction half that size would be gigantic.

And vendor and carrier cuts are only a small piece of the cuts that are going to be seen across the industry. Consider some of the following trends:

  • Corporate IT staffs are downsizing quickly from the move of computer functions to the cloud. There have been huge number of technicians with Cisco certifications, for example, that are finding themselves out of work as their companies eliminate the data centers at their companies.
  • On the flip side of that, huge data centers are being built to take over these same IT functions with only a tiny handful of technicians. I’ve seen reports where cities and counties gave big tax breaks to data centers because they expected them to bring jobs, but instead a lot of huge data centers are operating with fewer than ten employees.
  • In addition to employees there are fleets full of contractor technicians that do things like updating cell sites and these opportunities are going to dry up over the next few years. There will always be opportunities for technicians brave enough to climb cell towers, but that is not a giant work demand.

It looks like over the next few years that there are going to be a whole lot of unemployed technicians. Technology companies have always been cyclical and it’s never been unusual for engineers and technicians to have worked for a number of different vendors or carriers during a career, yet mostly in the past when there was a downsizing in one part of the industry the slack was picked up somewhere else. But we might be looking at a permanent downsizing this time. Once SDN networks are in place the jobs for those networks are not coming back. Once most IT functions are in the cloud those jobs aren’t coming back. And once the rural copper networks are replaced with 5G cellular those jobs aren’t coming back.

The Future of the Fiber Edge Device

Settop boxEvery once in a while I see a new technology idea that really makes me pause. I recently ran across Virtual Gateway Labs, a group out of San Jose that has come up with a radically different way of looking at fiber edge devices. The company has developed a fiber ONT (the edge device for a PON fiber network) that puts a lot of processing power into the ONT.

There is tremendous potential for this approach. They have created an ONT with multiple powerful processors that can be programmed as virtual machines to emulate all sorts of other virtual edge devices. With a software download you could use the ONT to emulate a settop box or an IoT hub. This would allow a fiber provider to avoid having to deal with the plethora of expensive boxes they have to deal with today. Instead a home only needs  this one device that can act as an ONT, a WiFi router, a settop box and an IoT home automation hub.

The company has created this box as their vision of how a software defined network might best be implemented in a fiber network. Since the ONT processors are remotely programmable and configurable they can be changed or updated as needed to keep up with innovations or customer needs.

The company is taking a very different approach from much of the rest of the industry. In general the SDN movement envisions creating dumber edge devices with the brains in the cloud. And so we see companies starting to develop a settop box in the cloud where the brains are at a central hub and a much cheaper device like a dongle is placed at the TV. Under that concept of SDN, the edge devices get simpler and less expensive, but rely on the Internet for all services.

Virtual Labs’ concept creates a single box at the home which can be segmented into multiple virtual machines that can be programmed for a myriad of purposes. Their version means that the single intelligent-box-at-the-home will only communicate sparely with the cloud. This gives the customer more autonomy with their services while giving the operator control over the services, without completely requiring the cloud for all activities.

Virtual Gateway Labs also envisions that their device would enable a truly open access network. With their device there could be different ISPs offering different services at a home using the same ONT. That would enable the ultimate version of open access where service providers wouldn’t just compete for customers but would compete for each service.

The problem with that idea is that it’s a solution for an industry that doesn’t exist. Talk to the handful of open access network operators and they will tell you that they have a very difficult time finding quality ISPs to compete on their networks even when those ISPs can sell bundles of products. It’s extremely unlikely in the current environment in the US that anybody is going to get any traction trying to operate an open access network with service level competition. But this idea might be of interest in Europe where there are already some very large open access networks.

But the Virtual Gateway Labs ONT might be a good solution for getting rid of the settop box. The FCC is currently considering an order that would try to force settop boxes to be standardized and generic so that customers can own them. But this new technology can eliminate the settop box completely and instead perhaps only need a dongle or small device at each TV to talk to the smart ONT hub.

Like with any new technology there is no telling if the idea might catch on. The company has a long way to go from concept to sellable unit. But if they could develop the software to emulate edge devices like settop boxes and IoT hubs then I think they might get some traction. They are designing the ONT to work with the major brands of PON equipment. I don’t know any service provider that wouldn’t like to get rid of settop boxes. They are one of the biggest hassles of being in the residential triple play business.

If these devices get traction there is no end to the range of future possible uses. The ONT processors could be programmed to act as a virtual reality or enhanced reality hub. They could be used to finally create the health monitoring systems that we have been hearing about for a decade but have yet to see. This is an intriguing concept and I hope they can make this happen.

Software Defined Networks

The InternetAT&T announced last week that they are going to implement software defined networking (SDN) in their network and that over a few years they will replace other kind of telecom gear. They say that over time this is going to save them billions on hardware costs. This announcement probably is a watershed moment for the telecom industry and is going to have huge implications for the way we build our networks and the vendors we use for routers and switches.

For those who are not familiar with the term, SDN is an idea that got started at UC Berkeley in 2008 and is now starting to hit the market. Its core concept is to use generic low cost routers, switches and other network hardware and to control them with specialized and centralized software. Today the routers that operate our networks come as packages of combined hardware and software, of which software is the more expensive component. Each vendor has their own way of doing things and you will find networks that are Cisco centric or Juniper centric, and network technicians become proficient with a specific brand of equipment.

But SDN is going to change all of that. With SDN a company like AT&T will be able to buy one set of centralized software and control their devices all over the network. The equipment becomes secondary in this configuration and AT&T could mix and match different brands of equipment. The biggest obvious savings will come in that they are not having to buy the software again each time they buy a router.

But there are even bigger savings promised with SDN over time. The promise of the technology is that companies can tailor their networks on the fly by making a software change rather than swapping or upgrading hardware systems. For a company that is as decentralized and huge as AT&T this could be transformational. I am sure many of you have waited before for AT&T to make facilities available because they were in the middle of a network upgrade. AT&T says that it is not unusual today for them to take 18 months to effectuate complex network changes. With SDN they could do it on the fly, and even after taking time with testing and double checks, they will be able to effectuate major changes in weeks instead of many months. And if circumstances dictate it, such as in an emergency, they could make changes on the fly.

SDN will give a whole new set of tools to network engineers. Today traffic is forwarded using industry standards such as MPLS, BGP or OSPF. With SDN a network engineer will be able to get extremely granular with traffic. For example, they might shuttle all traffic that is experiencing jitter to a specific place in the network. Since an SDN network is programmable it is going to give them flexibility they never have had.

This announcement has to be putting fear into the large telecom vendors like Cisco, Juniper and Alcatel. These companies supply the majority of the gear to the large network providers and the companies who are pioneering SDN are much smaller start-ups. Cisco and others are already climbing onto the SDN bandwagon and developing products, but there is no doubt that SDN will hurt these vendors. The billions of dollars of savings envisioned by AT&T has to come from somewhere. Carriers will be buy cheap generic switches and routers, will be able to keep them longer and are not likely to be as loyal to specific vendors as they were in the past.

This announcement should not send you out quite yet to change your own network to SDN. The industry is still in its infancy and the cost of the master SDN software is really steep today. But like every change of this magnitude the product will eventually get cheaper and work its way down into the rest of the industry. Let’s let AT&T figure out the bugs and at some point this will become the industry norm.