Explaining SDN

Somebody asked me to explain software defined networking (SDN), and I thought a good way to answer the question was to send them to an article that explains the concept. I couldn’t find anything on the web that explains SDN in plain English. This is not unusual for technical topics since tech guys generally have problems explaining what they do to laypeople. They hate boiling things down to simple language because a simple description doesn’t capture the nuances of the technology. I’ve always challenged engineers I work with to explain what they do in a way that their mother could understand – and most look at me like I’m an alien. I won’t promise that this is in plain English, but here is my shot at explaining SDN to a non-technical person.

The basis for SDN is that it is a technology that allows networks to be centrally and intelligently controlled or programmed. What does that mean?

There was a time in early computing when a network owner purchased all of the network gear from one vendor. Doing so made it possible to control the network with one set of software as long as the network owner could master the protocols used by the vendor. This sent a whole generation of IT technicians to become Cisco certified to prove that they had mastered Cisco network gear.

But it’s no longer reasonable today to have a complex network provisioned from one vendor. For one thing, most networks now use the cloud to some extent as part of the network – meaning they use computing power that is outside the direct control of the network owner. The pandemic has also forced most companies into allowing their network to communicate with remote employees – something that many companies refused to consider in the past. Networks have also gotten more complex due to the need to control Internet of Things devices – networks don’t just communicate with computers anymore.

The first goal of SDN is to bring everything under one big software umbrella. SDN provides a software platform that lets a network owner visualize the entire network. What does that mean in plain English? The goal of a network owner is to efficiently flow data to where it needs to go and to do so safely. It’s incredibly challenging to understand the flow of data in a network comprised of multiple devices, multiple feeds to and from the outside world, and constantly shifting demand from users on how they want to use the data.

SDN is a software platform that enables the network owner to see the data flow between different parts of the platform. Modern SDN technology has evolved from the OpenFlow protocol developed in 2008 in a collaboration between Stanford University and the University of California at Berkeley. The original platform enabled a network owner to measure and manage data traffic between routers and switches, regardless of the brand of equipment.

Over time, SDN has grown in sophistication and can do much more. As an example, with SDN, a network owner can set different levels of security for different parts of the network. A network operator might wall off traffic between remote employees and core data storage so that somebody working remotely can’t get access to some parts of the network. SDN software provides a way to break a network into subsets and treat each of them differently in terms of security protocols, the priority of routing, and access to other parts of the network. This is something that can’t easily be done by tinkering with the software settings of each individual router and switch – which is what network operators tried to do before SDN.

There have been huge benefits from SDN. Probably the biggest is that SDN allows a network owner to use generic white-box devices in the network – inexpensive routers and switches that are not pre-loaded with expensive vendor software. The SDN software can direct the generic devices to perform a needed function without the box needing to be pre-programmed. That’s the second big benefit of SDN – the whole network can be programmed as if every device came from the same vendor. The SDN software can tell each part of the network what to do and can even override preset functions from vendors.

It’s not hard to see why this is hard for a network engineer to explain because they don’t want to explain the primary goals of SDN without dipping into how it does all of this – and that is something that is incredibly hard to explain without using technical language and jargon. For that, I’d send you to the many articles written on the topic.

Technology The Industry

SDN Finally Comes to Telecom

For years we’ve heard that Software Defined Networking (SDN) is coming to telecom. There have been some movement in that area in routing on long-haul fiber routes, but mostly this network concept is not being used in telecom networks.

AT&T just announced the first major deployment of SDN. They will be introducing more than 60,000 ‘white box’ routers into their cellular networks. White box means that the routers are essentially blank generic hardware that comes with no software or operating systems. This differs from the normal routers from companies like Cisco that come with a full suite of software that defines how the box will function. In fact, from a cost perspective the software costs a lot more than the software in a traditional router.

AT&T will now be buying low-cost hardware and will load their own software onto the boxes. This is not a new concept and the big data center companies like Facebook and Google have been doing this for several years. SDN let’s a provider load only the software they need to support just the functions they need. The data center providers say that simplifying the software saves them a fortune in power costs and air conditioning since the routers are far more efficient.

AT&T is a little late to the game compared to the big web companies, and it’s probably taken them a lot longer to develop their own proprietary suite of cell site software since it’s a lot more complicated than switches in a big data center. They wouldn’t want to hand their cell sites over to new software until it’s been tested hard in a variety of environments.

This move will save AT&T a lot of money over time. There’s the obvious savings on the white box routers. But the real savings is in efficiency. AT&T has a fleet of employees and contractors whose sole function is to upgrade cell sites. If you’ve followed the company you’ve seen that it takes them a while to introduce upgrades into their networks as technicians often have to visit every cell site, each with different generics of operating hardware and software.

The company will still need to visit cell sites to make hardware changes, but the promise of SDN is that software changes can be implemented across their whole network in a short period of time. This means they can fix security flaws or introduce new features quickly. They will have a far more homogeneous network where cell sites use the same generics of hardware and software, which should reduce glitches and local problems. The company will save a lot on labor and contractor costs.

This isn’t good news for the rest of the industry. This means that Cisco and other router makers are going to sell far fewer telecom-specific routers. The smaller companies in the country have always ridden the coattails of AT&T and Verizon, whose purchase of switches and routers pulled down the cost of these boxes for everybody else. These big companies also pushed the switch manufacturers to constantly improve their equipment, and the volume of boxes sold justified the router manufacturers to do the needed R&D.

You might think that smaller carriers could also buy their own white box routers to also save money. This looks particularly attractive since AT&T is developing some of the software collaboratively with other carriers and making the generic software available to everybody. But the generic base software is not the same software that will run AT&T’s new boxes. They’ve undoubtedly sunken tens of millions into customizing the software further. Smaller carriers won’t have the resources to customize this software to make it fully functional.

This change will ripple through the industry in other ways. For years companies often hired technicians who had Cisco certification on various types of equipment, knowing that they understood the basics of how the software could be operated. But as Cisco and other routers are edged out of the industry there are going to be far fewer jobs for those who are Cisco certified. I saw an article a few years ago that predicted that SDN would decimate the technician work force by eliminating a huge percentage of jobs over time. AT&T will need surprisingly few engineers and techs at a central hub now to update their whole network.

We’ve known this change has been coming for five years, but now the first wave of it is here. SDN will be one of the biggest transformational technologies we’ve seen in years – it will make the big carriers nimble, something they have never been. And they are going to make it harder over time for all of the smaller carriers that compete with them – something AT&T doesn’t mind in the least.

Regulation - What is it Good For? The Industry

Regulation and Capital Spending

At the recent Mobile World Congress, FCC Chairman Ajit Pai said that one of his reasons he wants to reverse Title II regulation is that it has had a drastic impact on capital spending by ISPs. He says that the new regulations have been a disincentive for the ISPs to invest in broadband.

The Chairman bases that position on statistics provided by USTelecom which are based upon work done by Hal Singer, a Senior Fellow at GW Institute for Public Policy. Mr. Singer created the following table that shows the domestic capital spending for the big ISPs for 2014 through 2016. And indeed, this table shows a 5.6% drop, or $3.6 billion a year from 2014 to 2016 – which Mr. Singer attributes to Title II regulation.




AT&T $21.1 $17.3 $17.8
Verizon $17.2 $17.8 $17.1
Comcast $6.4 $7.1 $7.7
Sprint $3.8 $3.9 $1.4
Time Warner Cable $4.1 $4.4 $3.8
T-Mobile $4.3 $4.7 $4.7
CenturyLink $3.0 $2.9 $3.0
Charter $2.2 $1.9 $3.1
Cablevision $0.9 $0.8 $0.6
Frontier $0.6 $0.7 $1.3
US Cellular $0.6 $0.5 $0.5
Suddenlink $0.3 $0.4 $0.3
   Total $64.6 $62.4 $61.0

But like with all statistics, it’s not hard to draw different conclusions from the same set of numbers. For example, all of the drop in capital spending can be attributed to AT&T and Sprint. Taking those companies out of the table shows that capital spending for the other big ISPs is up $2.1 billion or 5% from 2014 to 2016.

So what’s going on with AT&T? There are a number of reasons for their change in capital spending:

·         During these same years the company made massive capital investments in DirecTV ($3 billion over the last few years) and also on the company’s purchase and expansion of its cellular network into Mexico ($3 billion over 4 years). Those numbers are not included in the above table and it’s easy to argue that the company just set different priorities and diverted normal domestic capital to these two giant ventures. If you add those capital expenditures into the table then AT&T’s capital spending has grown – just not their ‘domestic’ spending on traditional broadband.

·         AT&T has been making a huge effort to update its cellular network using software defined networking (SDN) as described at this AT&T website. They have been decommissioning traditional hardware at cell sites and installing much less expensive, off-the-shelf routers that can now control the cell sites from centralized data centers. They have already converted over half of their cell sites and this upgrade means vastly reduced spending on traditional cell site electronics. The company has been bragging about this shift to investors for several years.

·         AT&T has also retracted from expanding traditional big tower cell sites. For a number of years AT&T has been spending money to get fiber to its more remote cell sites, and that upgrade is largely done.

Sprint can also be easily explained. This is a company in trouble and that has been well documented over the last few years. A number of attempts to find a buyer has fallen through. What’s not shown on this table is that in 2013 (the year before the table begins) Sprint spent $6.4 billion on capital in a massive system-wide upgrade to LTE. Since then the company has very publicly stated that they are cutting capital spending to conserve cash. The company is only expanding now with carefully selected small cell deployments. But the company is clearly in network maintenance mode and is spending only what is needed to keep the cell sites functioning. Also included in the drop in spending is a change in the way that Sprint treats leased cellphones – they used to capitalize the phones and they now expense them.

There are going to be further decreases in future telecom capital spending across the industry. I expect capital spending for all four big wireless companies to keep decreasing due to efficiencies from SDN. We are now seeing a burst of spending from cable companies due to upgrades to DOCSIS 3.1, but when that’s done I would expect a significant decline in their capital spending as well. We are entering a time when improvements in software will lower the need for new hardware – not just in telecom, but in many other sectors as well.

I have always been annoyed when statistics are used to falsely justify public policy. There is no evidence that the big ISPS have changed their spending habits in any drastic way due to Title II regulations. The argument that Title II has affected capital spending comes directly from constant press releases from USTelecom, and the FCC Chairman should be above repeating arguments from lobbyists. If the FCC wants to undo Title II then it should just do it – there are a number of valid reasons why this might be good policy. But it’s disingenuous to cook up false reasons for why the change is needed.

Technology Uncategorized

Technology and Telecom Jobs

In case you haven’t noticed, the big companies in the industry are cutting a lot of jobs – maybe the biggest job cuts ever in the industry. These cuts are due to a variety of reasons, but technology change is a big contributor.

There have been a number of announced staff cuts by the big telecom vendors. Cisco recently announced it would cut back as many as 5,500 jobs, or about 7% of its global workforce. Cisco’s job cuts are mostly due to the Open Compute Project where the big data center owners like Facebook, Amazon, Google, Microsoft and others have turned to a model of developing and directly manufacturing their own routers and switches and data center gear. Cloud data services are meanwhile wiping out the need for corporate data centers as companies are moving most of their computing processes to the much more efficient cloud. Even customers that are still buying Cisco boxes are cutting back since the technology now provides a huge increase of capacity over older technology and they need fewer routers and switches.

Ericsson has laid off around 3,000 employees due to falling business. The biggest culprit for them is SDNs (Software Defined Networks). Most of the layoffs are related to cell site electronics. The big cellular companies are actively converting their cell sites to centralized control with the brains in the core. This will enable these companies to make one change and have it instantly implemented in tens of thousands of cell sites. Today that process requires upgrading the brains at each cell site and also involves a horde of technicians to travel to and update each site.

Nokia plans to layoff at least 3,000 employees and maybe more. Part of these layoffs are due to final integration with the purchase of Alcatel-Lucent, but the layoffs also have to do with the technology changes that are affecting every vendor.

Cuts at operating carriers are likely to be a lot larger. A recent article published in the New York Times reported that internal projections from inside AT&T had the company planning to eliminate as many as 30% of their jobs over the next few years, which would be 80,000 people and the biggest telco layoff ever. The company has never officially mentioned a number but top AT&T officials have been warning all year that many of the job functions at the company are going to disappear and that only nimble employees willing to retrain have any hope of retaining a long-term job.

AT&T will be shedding jobs for several reasons. One is the big reduction is technicians needed to upgrade cell sites. But an even bigger reason is the company’s plans to decommission and walk away from huge amounts of its copper network. There is no way to know if the 80,000 number is valid, but even a reduction half that size would be gigantic.

And vendor and carrier cuts are only a small piece of the cuts that are going to be seen across the industry. Consider some of the following trends:

  • Corporate IT staffs are downsizing quickly from the move of computer functions to the cloud. There have been huge number of technicians with Cisco certifications, for example, that are finding themselves out of work as their companies eliminate the data centers at their companies.
  • On the flip side of that, huge data centers are being built to take over these same IT functions with only a tiny handful of technicians. I’ve seen reports where cities and counties gave big tax breaks to data centers because they expected them to bring jobs, but instead a lot of huge data centers are operating with fewer than ten employees.
  • In addition to employees there are fleets full of contractor technicians that do things like updating cell sites and these opportunities are going to dry up over the next few years. There will always be opportunities for technicians brave enough to climb cell towers, but that is not a giant work demand.

It looks like over the next few years that there are going to be a whole lot of unemployed technicians. Technology companies have always been cyclical and it’s never been unusual for engineers and technicians to have worked for a number of different vendors or carriers during a career, yet mostly in the past when there was a downsizing in one part of the industry the slack was picked up somewhere else. But we might be looking at a permanent downsizing this time. Once SDN networks are in place the jobs for those networks are not coming back. Once most IT functions are in the cloud those jobs aren’t coming back. And once the rural copper networks are replaced with 5G cellular those jobs aren’t coming back.


The Future of the Fiber Edge Device

Every once in a while I see a new technology idea that really makes me pause. I recently ran across Virtual Gateway Labs, a group out of San Jose that has come up with a radically different way of looking at fiber edge devices. The company has developed a fiber ONT (the edge device for a PON fiber network) that puts a lot of processing power into the ONT.

There is tremendous potential for this approach. They have created an ONT with multiple powerful processors that can be programmed as virtual machines to emulate all sorts of other virtual edge devices. With a software download you could use the ONT to emulate a settop box or an IoT hub. This would allow a fiber provider to avoid having to deal with the plethora of expensive boxes they have to deal with today. Instead a home only needs  this one device that can act as an ONT, a WiFi router, a settop box and an IoT home automation hub.

The company has created this box as their vision of how a software defined network might best be implemented in a fiber network. Since the ONT processors are remotely programmable and configurable they can be changed or updated as needed to keep up with innovations or customer needs.

The company is taking a very different approach from much of the rest of the industry. In general the SDN movement envisions creating dumber edge devices with the brains in the cloud. And so we see companies starting to develop a settop box in the cloud where the brains are at a central hub and a much cheaper device like a dongle is placed at the TV. Under that concept of SDN, the edge devices get simpler and less expensive, but rely on the Internet for all services.

Virtual Labs’ concept creates a single box at the home which can be segmented into multiple virtual machines that can be programmed for a myriad of purposes. Their version means that the single intelligent-box-at-the-home will only communicate sparely with the cloud. This gives the customer more autonomy with their services while giving the operator control over the services, without completely requiring the cloud for all activities.

Virtual Gateway Labs also envisions that their device would enable a truly open access network. With their device there could be different ISPs offering different services at a home using the same ONT. That would enable the ultimate version of open access where service providers wouldn’t just compete for customers but would compete for each service.

The problem with that idea is that it’s a solution for an industry that doesn’t exist. Talk to the handful of open access network operators and they will tell you that they have a very difficult time finding quality ISPs to compete on their networks even when those ISPs can sell bundles of products. It’s extremely unlikely in the current environment in the US that anybody is going to get any traction trying to operate an open access network with service level competition. But this idea might be of interest in Europe where there are already some very large open access networks.

But the Virtual Gateway Labs ONT might be a good solution for getting rid of the settop box. The FCC is currently considering an order that would try to force settop boxes to be standardized and generic so that customers can own them. But this new technology can eliminate the settop box completely and instead perhaps only need a dongle or small device at each TV to talk to the smart ONT hub.

Like with any new technology there is no telling if the idea might catch on. The company has a long way to go from concept to sellable unit. But if they could develop the software to emulate edge devices like settop boxes and IoT hubs then I think they might get some traction. They are designing the ONT to work with the major brands of PON equipment. I don’t know any service provider that wouldn’t like to get rid of settop boxes. They are one of the biggest hassles of being in the residential triple play business.

If these devices get traction there is no end to the range of future possible uses. The ONT processors could be programmed to act as a virtual reality or enhanced reality hub. They could be used to finally create the health monitoring systems that we have been hearing about for a decade but have yet to see. This is an intriguing concept and I hope they can make this happen.


Software Defined Networks

AT&T announced last week that they are going to implement software defined networking (SDN) in their network and that over a few years they will replace other kind of telecom gear. They say that over time this is going to save them billions on hardware costs. This announcement probably is a watershed moment for the telecom industry and is going to have huge implications for the way we build our networks and the vendors we use for routers and switches.

For those who are not familiar with the term, SDN is an idea that got started at UC Berkeley in 2008 and is now starting to hit the market. Its core concept is to use generic low cost routers, switches and other network hardware and to control them with specialized and centralized software. Today the routers that operate our networks come as packages of combined hardware and software, of which software is the more expensive component. Each vendor has their own way of doing things and you will find networks that are Cisco centric or Juniper centric, and network technicians become proficient with a specific brand of equipment.

But SDN is going to change all of that. With SDN a company like AT&T will be able to buy one set of centralized software and control their devices all over the network. The equipment becomes secondary in this configuration and AT&T could mix and match different brands of equipment. The biggest obvious savings will come in that they are not having to buy the software again each time they buy a router.

But there are even bigger savings promised with SDN over time. The promise of the technology is that companies can tailor their networks on the fly by making a software change rather than swapping or upgrading hardware systems. For a company that is as decentralized and huge as AT&T this could be transformational. I am sure many of you have waited before for AT&T to make facilities available because they were in the middle of a network upgrade. AT&T says that it is not unusual today for them to take 18 months to effectuate complex network changes. With SDN they could do it on the fly, and even after taking time with testing and double checks, they will be able to effectuate major changes in weeks instead of many months. And if circumstances dictate it, such as in an emergency, they could make changes on the fly.

SDN will give a whole new set of tools to network engineers. Today traffic is forwarded using industry standards such as MPLS, BGP or OSPF. With SDN a network engineer will be able to get extremely granular with traffic. For example, they might shuttle all traffic that is experiencing jitter to a specific place in the network. Since an SDN network is programmable it is going to give them flexibility they never have had.

This announcement has to be putting fear into the large telecom vendors like Cisco, Juniper and Alcatel. These companies supply the majority of the gear to the large network providers and the companies who are pioneering SDN are much smaller start-ups. Cisco and others are already climbing onto the SDN bandwagon and developing products, but there is no doubt that SDN will hurt these vendors. The billions of dollars of savings envisioned by AT&T has to come from somewhere. Carriers will be buy cheap generic switches and routers, will be able to keep them longer and are not likely to be as loyal to specific vendors as they were in the past.

This announcement should not send you out quite yet to change your own network to SDN. The industry is still in its infancy and the cost of the master SDN software is really steep today. But like every change of this magnitude the product will eventually get cheaper and work its way down into the rest of the industry. Let’s let AT&T figure out the bugs and at some point this will become the industry norm.

Exit mobile version