Network Function Virtualization

Comcast recently did a trial of DOCSIS 4.0 at a home in Jacksonville, Florida, and was able to combine various new techniques and technologies to achieve a symmetrical 1.25 Gbps connection. Comcast says this was achieved using DOCSIS 4.0 technology coupled with network function virtualization (NFV), and distributed access architecture (DAA). Today I’m going to talk about the NFV concept.

The simplest way to explain network function virtualization is that it brings the lessons learned in creating efficient data centers to the edge of the network. Consider a typical data center application that is to provide computing to a large business customer. Before the conversion to the cloud, the large business network likely contained a host of different devices such as firewalls, routers, load balancers, VPN servers, and WAN accelerators. In a fully realized cloud application, all of these devices would be replaced with software that would mimic the functions of each device, all operated remotely in a data center consisting of banks of super-fast computer chips.

There are big benefits from a conversion to the cloud. Each of the various devices used in the business IT environment  is expensive and proprietary. The host of expensive devices, likely from different vendors are replaced with lower-cost generic servers that run on fast chips. A host of expensive electronics sitting at each large business is replaced by much cheaper servers sitting in a data center in the cloud.

There is also a big efficiency gain from the conversion because inevitably the existing devices in the historic network operated with different software systems that were never 100% compatible. Everything was cobbled together and made to work, but the average IT department at a large corporation never fully understood everything going on inside the network. There were always unexplained glitches when software systems of different devices interacted in the work network.

In this trial, Comcast used this same concept in the cable TV broadband network. Network function virtualization was used to replace the various electronic devices in the Comcast traditional network including the CMTS (cable modem termination system), various network routers, transport electronics for sending a broadband signal to neighborhood nodes, and likely the whole way down to the settop box. All of these electronic components were virtualized and performed in the data center or nearer to the edge in devices using the same generic chips that are used in the data center.

There are some major repercussions for the industry if the future is network function virtualization. First, all of the historic telecom vendors in the industry disappear. Comcast would operate a big data center composed of generic servers, as is done today in other data centers all over the country. Gone would be different brands of servers, transport electronics, and CMTS servers – all replaced by sophisticated software that will mimic the performance of each function performed by the former network gear. The current electronics vendors are replaced by one software vendor and cheap generic servers that can be custom built by Comcast without the need for an external vendor.

This also means a drastically reduced need for electronics technicians at Comcast, replaced by a handful of folks operating the data center. We’ve seen this same transition roll through the IT world as IT staffs have been downsized due to the conversion to the cloud. There is no longer a need for technicians that understand proprietary hardware such as Cisco servers, because those devices no longer exist in the virtualized network.

NFV should mean that a cable company becomes more nimble in that it can introduce a new feature for a settop box or a new efficiency into data traffic routing instantly by upgrading the software system that now operates the cable network.

But there are also two downsides for a cable company. First, conversion to a cloud-based network means an expensive rip and replacement of every electronics component in the network. There is no slow migration into DOCSIS 4.0 if it means a drastic redo of the underlying way the network functions.

There is also the new danger that comes from reliance on one set of software to do everything in the network. Inevitably there are going to be software problems that arise – and a software glitch in an NFV network could mean a crash of the entire Comcast network everywhere. That may sound extreme, and companies operating in the cloud will work hard to minimize such risks – but we’ve already seen a foreshadowing of what this might look like in recent years. The big fiber providers have centralized network functions across their national fiber networks, and we’ve seen network outages in recent years that have knocked out broadband networks in half of the US. When a cloud-based network crashes, it’s likely to crash dramatically.

Will Cable Companies Ignore Pleas for Faster Uploads?

One of the biggest impacts of the pandemic on broadband networks has been that homes suddenly care about upload speeds. Homes that largely lived off of downloading video transitioned to having adults and students at home and simultaneously trying to connect to remote work and school servers. People who were happy with their broadband speeds pre-pandemic suddenly found their broadband connection to be inadequate. Industry statistics show that huge numbers of people have upgraded to faster broadband products hoping to improve the home broadband performance without realizing that their performance bottleneck is due to inadequate upload speeds.

The cable industry has largely ignored upload bandwidth in the past. DOCSIS technology that operates the cable broadband networks allows as much as one-eight of total bandwidth to be dedicated to uploading. However, many cable broadband connections are configured with something less than that, because very few homeowners, other than perhaps photographers or others professionals who routinely send big files have ever cared about upload speeds. To make matters worse, the cable industry generally has squeezed the upload data stream into the portion of a cable network spectrum that has the most noise and interference. That never mattered to most people when sending files, but it matters when trying to maintain a steady connection to a work or school WAN.

But suddenly upload speeds matter to a lot of households. Some of the current pressure on upload speeds will be mitigated as K12 students eventually return to the classroom, but there seems to be widespread consensus that we’re going to see more adults permanently working from home.

It’s going to be interesting to see how the big cable companies react to the upload crisis. I’ve not seen many of them talking about the issue publicly and I suspect they are hoping this will go away when the pandemic ends. The cable companies have to know that they will be competing against technologies that offer faster upload data speeds. AT&T built fiber in the last few years to pass over 12 million homes. Telcos like CenturyLink and Windstream are quietly building fiber in some communities. Verizon says it’s going to pass 30 million homes with its fiber-to-the-curb technology using millimeter wave spectrum. And private ISPs are edging fiber into cable markets all over the country.

The cable companies have possible solutions on the horizon. Cable Labs recently announced the release of the DOCSIS 4.0 standard that will allow cable companies to offer symmetrical bandwidth. The gear for this technology ought to hit the market starting next year, but industry tech writers who interview cable company management seem to agree that the big cable companies have no appetite for paying for a new round of upgrades.

The cable companies all upgraded to DOSCSIS 3.1 in the last few years that added the capability for a gigabit download product. The web is full of pronouncements from the CTOs of the big cable companies saying that they hope to get a decade out of this last upgrade. Are any of these companies going to be willing to make a major new investment in new technology so soon after the last upgrade? In many markets the cable companies have become de facto monopolies, and that inevitably leads to milking technology upgrades for as long as possible.

There are a few other technology upgrades on the horizon that could provide relief for upload speeds. There has been a move by several vendors to explore expanding the bandwidth used inside a coaxial cable. A coaxial cable network acts like a captive radio network that uses a big range of different frequencies. Cable systems historically used as much as 1 GHz of total spectrum. In recent years with the advent of DOCSIS 4.0 that’s been expanded to 1.2 GHz of total spectrum. The technology now exists to upgrade cable networks to 1.8 GHz. That would provide a huge additional pile of spectrum that could be dedicated to bandwidth. But such an upgrade would require changing out a lot of network components such as amplifiers, power taps, and modems. Such an upgrade might be nearly as expensive as a shift to DOCSIS 4.0.

The bottom line is that any significant changes to create more upload bandwidth inside cable networks will cost a lot of money. I bet that the big cable companies will stick with the current technology migration plan that would keep DOCSIS 3.1 for the rest of this decade. Likely the only thing that might prompt cable companies to upgrade sooner would be competitors mass deploying technologies that are marketed for having faster upload speeds. The most likely future is that the big cable companies will introduce DOCSIS 4.0 selectively in the few markets where they are feeling competitive pressure, but that most of households are not going to see the upload speeds that people now want.

Can 5G Compete with Cable Broadband?

One of the recurring themes used to promote 5G is that wireless broadband is going to become a serious competitor to wireline broadband. There are two primary types of broadband competition – competition by price or performance. Cable companies have largely won the broadband battle in cities and suburbs and I’ve been thinking about the competition that cable companies might see from 5G.

Cable broadband is an interesting product. In most cities and suburbs today, the basic broadband product has a download speed between 100 Mbps to 200 Mbps with upload speeds in the range of 10 Mbps to 15 Mbps. The cable companies decided over a decade ago that they were going to stay in front of market demand and have periodically increased speeds, with the most recent speed increases introduced around two years ago. Cable systems can offer speeds up to a gigabit, but the ugly secret that cable companies don’t want to talk about is that it would be incredibly expensive if too many people bought and used gigabit speeds. CCG does market surveys and the primary complaints that customers have about urban cable broadband is inconsistency – networks have periodic slowdowns and outages that customers find frustrating. As much as one third of cable customers also poll as hating the customer service of the larger cable companies.

The biggest weakness of cable broadband is the upload speed. This wasn’t an issue for most homes until the recent pandemic sent students and parents home. Many homes that were satisfied with cable broadband have found that the upload streams are inadequate to allow multiple people in a home to connect to servers and video conferencing services. Cable companies can probably tweak upload speeds upward by 50% more, but that will still feel slow to many homes. Cable companies are faced with an expensive upload to DOCSIS 4.0 to create symmetrical speeds.

There are two products being marketed as 5G. The first is Verizon’s fixed wireless access product. This is not 5G and is best described as fiber-to-the-curb, because it requires a fiber network built close to homes to provide this product. This is a fiber technology that happens to use a wireless drop. As such, it is technologically superior to cable broadband in that speeds can be symmetrical. Verizon says speeds can be as fast as a gigabit, but speeds will vary by customer and will likely slow down during heavy rain or get slower in summer when shrubs and trees are in full leaf. From a price perspective, Verizon is using this product to reduce cellular churn and is pricing it at $50 for a Verizon wireless customer and $70 for everybody else.  The $70 price is not going to push Comcast and Charter to lower prices, but it might force them to hesitate with future rate increases for neighborhoods that are competing with the Verizon product.

The FCC and the industry have implied for years that 5G cellular will be a competitor for landline broadband. I still can’t see many homes accepting 5G cellular as a replacement for landline broadband. I can think of a number of important ways to compare and contrast the two broadband products:

Speed. Forget the millimeter-wave product that cellular companies are touting as delivering cellular speeds over a gigabit. It’s a gimmick product used  to try to promote the idea that 5G is fast. The millimeter-wave technology is only good outdoors, and even then only travels a few hundred feet from a cell site. It delivers gigabit speeds to cellphones – when cellphones aren’t designed to run multiple apps that require fast broadband. The 5G download speeds on regular cellphones should creep up 100 Mbps over the next 5 to 7 years, and would rival the base speeds on cable company networks – but by that time the cable companies are likely to upgrade all of their customers to 250 Mbps. Cellular upload speeds don’t matter, because no family is going to conduct multiple upload sessions over a single cellphone.

Overall Capacity. Cellular networks today carry less than 5% of all US broadband. Even the majority of data passed through cellphones is handed off to landline networks through WiFi. In North America this year, Cisco predicts that in 2020 there will be 77 exabytes per month carried by landline networks compared to 3.4 exabytes carried by cellular networks. By 2022 that will grow to 109 exabytes for landline networks and 6 exabytes for cellular networks – the gap between the two technologies is rapidly widening. There is no scenario where cellular networks can somehow steal away a lot of the traffic carried by landlines. When cellular companies make this claim they are arguing against the realities of physics.

Household Usage. Household usage of broadband has exploded. In the first quarter of 2018, the average US home used 215 gigabytes of data per month. At the end of the recent first quarter of 2020 that had grown to over 400 gigabytes per month. By 2024 the average home might be using more than 700 gigabytes per month.

Data Caps. The above statistics show the absurdity of the claim that cellular will somehow overtake landline broadband. Even the ‘unlimited’ cellular data plans today are capped or heavily throttled after 20 or so gigabytes of data used in a month. Cellular companies are not likely to raise the data caps much because they don’t want heavy data users sucking all of the capacity out of the cellular networks.

Pricing. US cellular data is the most expensive broadband in developed countries. For 5G to compete with landline broadband, the cellular companies would have to kill the paradigm of selling an extra gigabyte of data for $10. 5G can only compete with landline broadband if the cellular carriers can increase wireless network capacity by a factor of ten and are willing to lower prices by more than a factor of ten. The first is not possible due to the limitations of physics and there are no indications that cellular carriers are willing to consider the second.

How Will Cable Companies Cope with COVID-19?

A majority of households today buy broadband from cable companies that operate hybrid coaxial fiber networks (HFC) that us some version of DOCISIS technology to control the networks. The largest cable companies have upgraded most of their networks to DOCSIS 3.1 that allows for gigabit download speeds.

The biggest weakness in the cable networks is the upload data links. The DOCSIS standard limits the upload path to me no larger than 1/8th of the total bandwidth uses – but it’s not unusual for the cable companies to make this path even smaller and offer products like 100/10 Mbps where the upload is 1/11th of the total bandwidth provided to customers.

This is not a new concern for the cable companies and the engineering folks at Comcast and other big cable companies have been discussing ways to improve upload bandwidth for much of the last decade. They understood that the need for uploading would someday overwhelm the bandwidth path provided – they just didn’t expect to get there so explosively as been done in reaction to the COVID-19 crisis.

Every student and employee trying to work from home is carving out an uploaded VPN when they connect to a school or work server. Customers are also using significant upload bandwidth when they join a video call on Zoom or other platforms. While carriers report 30–40% overall increases in traffic due to COVID-19, they are not disclosing that a lot of that increase is demand for uploading.

Cable companies are now faced with solving the upload crisis. Practically every prognosticator in the country is predicting that we’re not going to return to pre-COVID behavior. There is likely to be a lot of people who will continue to work from home. While students will return to the classroom eventually, this grand experiment has shown that’s it’s feasible to involve students in the classroom remotely, and so school systems are likely to continue this practice for students with long-term illnesses or other reasons why they can’t always be in the classroom. Finally, we’ve taught a whole generation of people that video meetings can work, so there is going to be a whole lot more of that. The day of traveling to attend a few hour meeting might be over.

There is one other interesting fact to consider when looking at a cable company upload data path. Cable companies have generally devalued the upload path quality and have assigned the upload path to the low frequencies on the cable network spectrum. Historically upload data speeds were provisioned on the 5-42 MHz range of spectrum. This is the spectrum in a cable system that experiences the most interference from things like microwave ovens, vacuum cleaners and passing large trucks. Cable companies could get away with this because historically most people didn’t care if it took longer to upload a file or if packets had to be retransmitted due to interference. But people connecting to WANs and video conferences care about the upload quality as well as speed.

One solution, and something that some cable providers have already done is to do what is called a mid-split upgrade that extend the spectrum for uploading to the 5-85 MHz band. This still includes a patch of the worst spectrum inside the cable system, but is a significant boost in the amount of upload broadband available. Depending upon the settop boxes being used, this upgrade can require some new customer boxes.

Another idea is to do more traditional node splits, meaning to reduce the number of customers included in a neighborhood node. Traditionally, node splits were done to improve the performance of download speeds – this was the fastest way to relieve network congestion when a local neighborhood network bogged down unduly in the evening. It’s an interesting idea to consider splitting nodes to relive pressure on the upload data path.

After those two idea the upgrades get expensive. Migrating to switched digital video could free up a mountain of system bandwidth which would allow for a larger data path, including an enlarged upload path. The downside of this kind of upgrade is that it moves outside of the DOCSIS technology and starts to look more like providing Ethernet over fiber. This is not just a forklift upgrade it changes the basic way the network operates.

The final way to get more upload speed would be an upgrade to the upcoming DOCSIS 4.0 standard. Everything I read about this makes it sound expensive. But the new standard would allow for nearly symmetrical data services and would let cable network broadband compete head-on with fiber network. It will be interesting to see if the cable companies view the upload crisis as bad enough to warrant spending huge amounts of money to fix the problem.

10 GB Cable Broadband?

CableLabs recently put the finishing touches on the new DOCSIS 4.0 standard – the latest standard for delivering broadband across hybrid fiber-coaxial networks. The changes are intended to make cable networks more competitive with fiber networks – although it’s unlikely that we’ll see anybody use the full capacity of the new standard.

Following are the benefits promised by CableLabs in the DOCSIS 4.0 press release:

Speeds up to 10 Gbps. This capability isn’t quite what it seems. The current DOCSIS 3.1 standard already allows speeds up to 5 Gbps and no cable company is delivering superfast speeds because it’s not practical. The only way to achieve 5 – 10 Gbps broadband would be to remove most or all of the channels of TV signal from the system.

Cable systems could consider upgrading to speeds faster than 1 Gbps if they make another costly upgrade of migrating to IP video. Today a cable company sends all TV channels to every customer and blocks the channels that are not subscribed. An IP video system, which is used on DSL and fiber networks only sends a customer the channel they are watching. Such a migration would free tremendous bandwidth within the network but would mean replacing the cable headend and all settop boxes.

Faster Upload Speeds. DOCSIS 4.0 will be able to deliver as much as 6 Gbps upload speeds to go along with the 10 Gbps download speeds. This would allow a cable company to offer a symmetrical 1 Gbps bandwidth product – something that is not possible today and that puts cable networks at a marketing disadvantage compared to fiber. The current DOCSIS 3.1 technology provides for one-tenth of the download speed to be provided as upload – meaning a 100 Mbps download product typically has a 10 Mbps upload speed. Interestingly, many cable products today don’t even use the full 10% upload allotment and assign more bandwidth instead to download.

Improved Reliability. CableLabs is pushing higher reliability as part of the DOCSIS 4.0 upgrade. This will be done using new software they’ve labeled as Proactive Network Maintenance (PNM) that will use increased monitoring and machine learning to identify network problems earlier. However, this technology will become available with DOCSIS 3.1 as well and doesn’t require an upgrade.

Better Security. CableLabs is also touting increased security through a software suite labeled as Micronets. This will allow the cable operator to isolate different parts of the network into separate trust domains so that a problem in one part of the network doesn’t infect the rest of the network. That’s an interesting concept that would also benefit fiber networks.

Low Latency. The last touted benefit of DOCSIS 4.0 is lower latency. There is no way to improve the native latency across a coaxial cable network, so the improved latency comes as a result of a bit of smoke and mirrors. A given customer, such as a small cell site in a residential neighborhood could be given the best possible latency by giving it traffic a priority over all other traffic. This was exactly what net neutrality was supposed to protect against and residential customers in that neighborhood would likely be highly dismayed to know that connection problems are due to the cable company selling priority bandwidth to AT&T in the neighborhood.

Expanded Use of Spectrum. The extra bandwidth for DOCSIS 4.0 comes from the expansion of the frequencies used inside the coaxial cable. Coaxial networks function like a captive radio system inside of the coax and DOCSIS 4.0 expands the usable spectrum on the cable to 1.8 GHz – most cable networks today use 1 to 1.2 GHz of spectrum.

What’s not mentioned in the press releases is the cost. Expanding the spectrum likely means replacing amplifiers and possibly even power taps in the cable network. Using the new spectrum and standard means a swap-out of cable modems. It’s possible in some networks that moving to the increased spectrum will also mean replacing older or frail coaxial cable in the network.

So far, no major cable company has said that they are interested in this upgrade. However, cable companies are under increased pressure to improve upload speeds. They don’t necessarily need to move to symmetrical speeds. However, the weakness of upload bandwidth in cable networks have been highlighted as millions of people try to connect to work and school servers from home during the COVID-19 crisis – I’ve seen more demand for faster upload speeds in the last few weeks than I can ever remember in the past.

The Future of Coaxial Networks

My blog devotes a lot of time looking at fiber deployment, but since the majority of people in the US get broadband from cable companies using hybrid fiber/coaxial (HFC) technology, today’s blog looks at the next generation of changes planned for HFC.

DOCSIS 4.0. The current generation of HFC technology is DOCSIS 3.1 This technology uses 1.2 GHz of spectrum over coaxial cable. DOCSIS 3.1 has several competitive drawbacks compared to fiber. First, while the technology can deliver gigabit download speeds to customers, the dirty secret of the industry is that gigabit speeds can only be given to a limited number of customers. With current node sizes, cable companies can’t support very many large data users without sacrificing the performance of everybody in a node. This is why you don’t see cable companies pricing gigabit broadband at competitive prices or pushing it very hard.

The other big drawback is that upload speeds on DOCSIS 3.1 are set by specification to be no more than one-eighth of the total bandwidth on the system. Most cable companies don’t even allocate that much to upload speeds.

The primary upgrade with DOCSIS 4.0 will be to increase system bandwidth to 3 GHz. That supplies enough additional bandwidth to provide symmetrical gigabit service or else offer products that are faster than 1 Gbps download. It would also allow a cable company to support a lot more gigabit customers.

The big drawback to the upgrade is that many older coaxial cables won’t be able to handle that much bandwidth and will have to be replaced. Further, upgrading to 3 GHz is going to mean replacing or upgrading power taps, repeaters, and other field hardware in the coaxial network. CableLabs is talking about finalizing the DOCSIS 4.0 specification by the end of 2020. None of the big cable companies have said if and when they might embrace this upgrade. It seems likely that many of the bigger cable companies are in no hurry to make this upgrade.

Low Latency DOCSIS (LLD). Another drawback of HFC networks is that they don’t have the super-low latency needed to support applications like intense gaming or high-quality video chat. The solution is a new encoding scheme being called low latency DOCSIS (LLD).

The LLD solution doesn’t change the overall latency of the cable network but instead prioritizes low-latency applications. The result is to increase the latency for other applications like web-browsing and video streaming.

This can be done because most of the latency on an HFC network comes from the encoding schemes used to layer broadband on top of cable TV signals. The encoding schemes on coaxial cable networks are far more complex than fiber encoding. There are characteristics of copper wires that cause natural interference within a transmission path. A coaxial encoding scheme must account for attenuation (loss of signal over distance), noise (the interference that appears from external sources since copper acts as a natural antenna), and jitter (the fact that interference is not linear and comes and goes in bursts). Most of the latency on a coaxial network comes from the encoding schemes that deal with these conflicting characteristics. The LLD solution bypasses traditional encoding for the handful of applications that need low latency.

Virtual CMTS. One of the more recent improvements in coaxial technology was distributed access architecture (DAA). This technology allows for disaggregating the CMTS (the router used to provide customer broadband) from core routing functions, meaning that the CMTS no longer has to sit at the core of the network. The easiest analogy to understand DAA is to consider modern DSLAM routers. Telephone companies can install a DSLAM at the core of the network, but they can instead put the DSLAM at the entrance to a subdivision to get it closer to customers. DAA allowed cable companies to make this same change.

With virtual CMTS a cable network takes DAA a step further. In a virtual CMTS environment, the cable company might perform some of the CMTS functions in remote data centers in the cloud. There will still be a piece of electronics where the CMTS used to sit, but many of the computing functions can be done remotely.

A cloud-based CMTS offers some advantages to the cable operator:

  • Allows for customizing portions of a network. The data functions provided to a business district can be different from what is supplied to a nearby residential neighborhood. Customization can even be carried down to the customer level for large business customers.
  • Allows for the use of cheap off-the-shelf hardware, similar to what’s been done in the data centers used by the big data complies like Google and Facebook. CMTS hardware has always been expensive because it’s been made by only a few vendors.
  • Improves operations by saving on local resources like local power, floor/rack space, and cooling by moving heavy computing functions to data centers.

Summary. There is a lot of discussion within the cable industry asking how far cable companies want to push HFC technology. Every CEO of the major cable companies has said that their eventual future is fiber, and the above changes, which each bring HFC closer to fiber performance, are still not as good as fiber. Some Wall Street analysts have predicted that cable companies won’t embrace bandwidth upgrades for a while since they already have the marketing advantage of being able to claim gigabit speeds. The question is if the cable companies are willing to make the expensive investment to functionally come closer to fiber performance or if they are happy to just claim to be equivalent to fiber performance.