SDN Finally Comes to Telecom

For years we’ve heard that Software Defined Networking (SDN) is coming to telecom. There have been some movement in that area in routing on long-haul fiber routes, but mostly this network concept is not being used in telecom networks.

AT&T just announced the first major deployment of SDN. They will be introducing more than 60,000 ‘white box’ routers into their cellular networks. White box means that the routers are essentially blank generic hardware that comes with no software or operating systems. This differs from the normal routers from companies like Cisco that come with a full suite of software that defines how the box will function. In fact, from a cost perspective the software costs a lot more than the software in a traditional router.

AT&T will now be buying low-cost hardware and will load their own software onto the boxes. This is not a new concept and the big data center companies like Facebook and Google have been doing this for several years. SDN let’s a provider load only the software they need to support just the functions they need. The data center providers say that simplifying the software saves them a fortune in power costs and air conditioning since the routers are far more efficient.

AT&T is a little late to the game compared to the big web companies, and it’s probably taken them a lot longer to develop their own proprietary suite of cell site software since it’s a lot more complicated than switches in a big data center. They wouldn’t want to hand their cell sites over to new software until it’s been tested hard in a variety of environments.

This move will save AT&T a lot of money over time. There’s the obvious savings on the white box routers. But the real savings is in efficiency. AT&T has a fleet of employees and contractors whose sole function is to upgrade cell sites. If you’ve followed the company you’ve seen that it takes them a while to introduce upgrades into their networks as technicians often have to visit every cell site, each with different generics of operating hardware and software.

The company will still need to visit cell sites to make hardware changes, but the promise of SDN is that software changes can be implemented across their whole network in a short period of time. This means they can fix security flaws or introduce new features quickly. They will have a far more homogeneous network where cell sites use the same generics of hardware and software, which should reduce glitches and local problems. The company will save a lot on labor and contractor costs.

This isn’t good news for the rest of the industry. This means that Cisco and other router makers are going to sell far fewer telecom-specific routers. The smaller companies in the country have always ridden the coattails of AT&T and Verizon, whose purchase of switches and routers pulled down the cost of these boxes for everybody else. These big companies also pushed the switch manufacturers to constantly improve their equipment, and the volume of boxes sold justified the router manufacturers to do the needed R&D.

You might think that smaller carriers could also buy their own white box routers to also save money. This looks particularly attractive since AT&T is developing some of the software collaboratively with other carriers and making the generic software available to everybody. But the generic base software is not the same software that will run AT&T’s new boxes. They’ve undoubtedly sunken tens of millions into customizing the software further. Smaller carriers won’t have the resources to customize this software to make it fully functional.

This change will ripple through the industry in other ways. For years companies often hired technicians who had Cisco certification on various types of equipment, knowing that they understood the basics of how the software could be operated. But as Cisco and other routers are edged out of the industry there are going to be far fewer jobs for those who are Cisco certified. I saw an article a few years ago that predicted that SDN would decimate the technician work force by eliminating a huge percentage of jobs over time. AT&T will need surprisingly few engineers and techs at a central hub now to update their whole network.

We’ve known this change has been coming for five years, but now the first wave of it is here. SDN will be one of the biggest transformational technologies we’ve seen in years – it will make the big carriers nimble, something they have never been. And they are going to make it harder over time for all of the smaller carriers that compete with them – something AT&T doesn’t mind in the least.

The Shift To Proprietary Hardware

OLYMPUS DIGITAL CAMERA

There is a trend in the industry that is not good for smaller carriers. More and more I see the big companies designing proprietary hardware just for themselves. While that is undoubtably good for the big companies, and I am sure that it saves them a lot of money, it is not good for anybody else.

I first started noticing this a few years ago with settop boxes. It used to be that Comcast and the other large cable companies used the same settop boxes as everybody else. And their buying power is so huge that it drove down the cost of the settop boxes for everybody in the industry. It was standard for large companies to put their own name tag on the front of the boxes, but for the most part they were the same boxes that everybody else could buy, from the same handful of manufacturers.

But then I started seeing news releases and stories indicating that the largest cable companies had developed proprietary settop boxes of their own. One driver for this change is that the carriers are choosing different ways to bring broadband to the settop box. Another change is that the big companies are adding different features, and are modifying the hardware to go along with custom software. Cable companies are even experimenting with very non-traditional settop box platforms like Roku or the various game consoles.

I see this same thing going on all over the industry. The cable modems and customer gateways that the large cable companies and the large telcos use are proprietary and designed just for them. I recently learned that the WiFi units that Comcast and other large cable companies are deploying outdoors are proprietary to them. Google has designed its own fiber-the-the-premise equipment. And many companies including Amazon, Facebook, Google, Microsoft, and others are designing their own proprietary routers to use in their cloud data centers.

In all of these cases (and many other that I haven’t listed here), the big companies used to buy off-the-shelf equipment. They might have had a slightly different version of some of the hardware, but not different enough that it made a difference to the manufacturers. Telco has always been an industry where only a handful of companies make any given kind of electronics. Generally, smaller companies bought from whichever vendors the big companies chose, since those vendors had the economy of scale.

But now the big carriers are not only using proprietary hardware, but a lot of them are getting it manufactured for themselves directly, without one of the big vendors in the middle. You can’t blame a large company for this; I am sure they save a lot of money by cutting Alcatel/Lucent, Cisco, and Motorola out of the supply chain. But this tendency is putting a hurt on these traditional vendors and making it harder for vendors to survive.

It’s going to get worse. Currently there is a huge push in many parts of the telecom business to use software-defined networking (SDN) to simplify field hardware and control everything from the cloud. Since the large carriers will shift to SDN networks long before smaller carriers, the big companies will be using very different gear at the edges of the network – and those are the parts of the network that cost the most.

This is a problem for smaller carriers since they often no longer benefit from being able to buy the same devices that the large companies buy to take advantage of their huge economy of scale. Over time this is going to mean the prices for the basic components smaller carriers buy are going to go up. And in the worst case there might not be any vendor that can make a business case for manufacturing a given component for the small carriers. One of the advantages of having healthy large manufacturers in the industry was that they could take a loss on some product lines as long as the whole suite of products they sold made a good profit. That will probably no longer be the case.

I hate to think about where this trend is going to take the industry in five to ten years, and I add it to the list of things that small carriers need to worry about.

The Battle of the Network Switches

Cisco_Media_Convergence_ServersYesterday Facebook announced that it has successfully built an open-source network switch. This is really big news in an industry where Cisco and Juniper together have more or less cornered the switch market. The Facebook switch has been named Wedge and is operated by an open-source software platform they called FBOSS. This has been created as part of the Open Compute Project (OCP) started by Facebook but now involving many other companies. The goal of this project was to radically change the way companies buy hardware and software, and it is starting to achieve those goals.

 

This announcement is going to shake up the $23 billion Ethernet switch market in the same way that the introduction of the softswitch killed the duopoly on voice switches once held by Nortel and Lucent. I’ve written earlier about how the Ethernet switch industry is moving towards software-defined networking (SDN). The goal of SDN is to take features that have baked into hardware, such as security and device management and make those functions software controlled.

 

Cisco has already introduced their own version of SDN and they now have software that will control their various devices. But honestly this is only a modest change for them, because at the end of the day all of their hardware and software is proprietary. We are all very familiar with network engineers who need multiple Cisco certifications just to be able to operate the Cisco gear. Cisco’s SDN doesn’t really change that need for network engineers or lower the cost. It just layers a new software over top of the old platform.

 

The industry was ripe for this change because Cisco has grown into the same kind of company that we saw in Lucent and Nortel at their peak. The Cisco pricing model now includes a permanent 15% annual fee on top of any hardware you buy from them. This fee is ostensibly for upgrades and maintenance, but the people who write the checks for this don’t feel like they are getting much value from these annual checks. This sounds exactly like the kinds of pricing practice we saw in the voice industry when it was a duopoly of Nortel and Lucent.

 

Cisco has been reported to have a 60% profit margin, and so they are ripe for a challenge. Cisco is not going to go away easily and they have been very clever in the way they have shaped the network switch market. That market is operated by and decisions made by switch engineers, all of whom Cisco has made certain have a long list of Cisco certifications. And frankly, the OCP initiative is aimed directly at getting rid of those network engineers, in the same way that cloud computing is doing away with server engineers.

 

Certainly Cisco has already lost the largest customers in the market. Facebook will be going with their own new technology. It’s been reported that Amazon, Microsoft and Google all are working on their own versions of SDN servers as well, although none of them are reported to be headed towards open-sourcing like the OCP initiative. But one would think that this is going to put a massive amount of price pressure on Cisco in a few years, as ought to happen with any company that has gigantic profit margins. There are still going to be a number of network operators who are going to go with traditional Cisco for a while simply because it works and is comfortable for them. But as the OCP hardware becomes readily available and proves able to work in the market it’s going to get harder and harder to justify buying expensive and proprietary servers.

 

It took a full decade for the traditional voice switch manufacturers to fail after the introduction of the softswitch. And Cisco is probably better equipped to fight back against this change than were Nortel and Lucent. But in the early days of the softswitch I saw some of my clients cut their hardware and maintenance costs in half by going with a softswitch and it was obvious then that the newer technology would eventually win. This Facebook announcement is the first day of the decade that is going to transform the way we buy and use network switches.

 

Software Defined Networks

The InternetAT&T announced last week that they are going to implement software defined networking (SDN) in their network and that over a few years they will replace other kind of telecom gear. They say that over time this is going to save them billions on hardware costs. This announcement probably is a watershed moment for the telecom industry and is going to have huge implications for the way we build our networks and the vendors we use for routers and switches.

For those who are not familiar with the term, SDN is an idea that got started at UC Berkeley in 2008 and is now starting to hit the market. Its core concept is to use generic low cost routers, switches and other network hardware and to control them with specialized and centralized software. Today the routers that operate our networks come as packages of combined hardware and software, of which software is the more expensive component. Each vendor has their own way of doing things and you will find networks that are Cisco centric or Juniper centric, and network technicians become proficient with a specific brand of equipment.

But SDN is going to change all of that. With SDN a company like AT&T will be able to buy one set of centralized software and control their devices all over the network. The equipment becomes secondary in this configuration and AT&T could mix and match different brands of equipment. The biggest obvious savings will come in that they are not having to buy the software again each time they buy a router.

But there are even bigger savings promised with SDN over time. The promise of the technology is that companies can tailor their networks on the fly by making a software change rather than swapping or upgrading hardware systems. For a company that is as decentralized and huge as AT&T this could be transformational. I am sure many of you have waited before for AT&T to make facilities available because they were in the middle of a network upgrade. AT&T says that it is not unusual today for them to take 18 months to effectuate complex network changes. With SDN they could do it on the fly, and even after taking time with testing and double checks, they will be able to effectuate major changes in weeks instead of many months. And if circumstances dictate it, such as in an emergency, they could make changes on the fly.

SDN will give a whole new set of tools to network engineers. Today traffic is forwarded using industry standards such as MPLS, BGP or OSPF. With SDN a network engineer will be able to get extremely granular with traffic. For example, they might shuttle all traffic that is experiencing jitter to a specific place in the network. Since an SDN network is programmable it is going to give them flexibility they never have had.

This announcement has to be putting fear into the large telecom vendors like Cisco, Juniper and Alcatel. These companies supply the majority of the gear to the large network providers and the companies who are pioneering SDN are much smaller start-ups. Cisco and others are already climbing onto the SDN bandwagon and developing products, but there is no doubt that SDN will hurt these vendors. The billions of dollars of savings envisioned by AT&T has to come from somewhere. Carriers will be buy cheap generic switches and routers, will be able to keep them longer and are not likely to be as loyal to specific vendors as they were in the past.

This announcement should not send you out quite yet to change your own network to SDN. The industry is still in its infancy and the cost of the master SDN software is really steep today. But like every change of this magnitude the product will eventually get cheaper and work its way down into the rest of the industry. Let’s let AT&T figure out the bugs and at some point this will become the industry norm.