Expect a New Busy Hour

One of the many consequences of the coronavirus is that networks are going to see a shift in busy hour traffic. Busy hour traffic is just what is sounds like – it’s the time of the day when a network is busiest, and network engineers design networks to accommodate the expected peak amount of bandwidth usage.

Verizon reported on March 18 that in the week since people started moving to work from home that they’ve seen a 20% overall increase in broadband traffic. Verizon says that gaming traffic is up 75% as those stuck at home are turning to gaming for entertainment. They also report that VPN (virtual private network) traffic is up 34%. A lot of connections between homes and corporate and school WANs are using a VPN.

These are the kind of increases that can scare network engineers, because Verizon just saw a typical year’s growth in traffic happen in a week. Unfortunately, the announced Verizon traffic increases aren’t even the whole story since we’re just at the beginning of the response to the coronavirus. There are still companies figuring out how to give secure access to company servers and the work-from-home traffic is bound to grow in the next few weeks. I think we’ll see a big jump in video conference traffic on platforms like Zoom as more meeting move online as an alternative to live meetings.

For most of my clients, the busy hour has been in the evening when many homes watch video or play online games. The new paradigm has to be scaring network engineers. There is now likely going to be a lot of online video watching and gaming during the daytime in addition to the evening. The added traffic for those working from home is probably the most worrisome traffic since a VPN connection to a corporate WAN will tie up a dedicated path through the Internet backbone – bandwidth that isn’t shared with others. We’ve never worried about VPN traffic when it was a small percentage of total traffic – but it could become one of the biggest continual daytime uses of bandwidth. All of the work that used to occur between employees and the corporate server inside of the business is now going to traverse the Internet.

I’m sure network engineers everywhere are keeping an eye on the changing traffic, particularly to the amount of broadband used during the busy hour. There are a few ways that the busy hour impacts an ISP. First, they must buy enough bandwidth to the Internet to accommodate everybody. It’s typical to buy at least 15% to 20% more bandwidth than is expected for the busy hour. If the size of the busy hour shoots higher, network engineers are going to have to quickly buy a larger pipe to the Internet, or else customer performance will suffer.

Network engineers also keep a close eye on their network utilization. For example, most networks operate with some rule of thumb, such as it’s time to upgrade electronics when any part of the network hits some pre-determined threshold like 85% utilization. These rules of thumb have been developed over the years as warning signs to provide time to make upgrades.

The explosion of traffic due to the coronavirus, might shoot many networks past these warning signs and networks start experiencing chokepoints that weren’t anticipated just a few weeks earlier. Most networks have numerous possible chokepoints – and each is monitored. For example, there is usually a chokepoint going into neighborhoods. There are often chokepoints on fiber rings. There might be chokepoints on switch and router capacity at the network hub. There can be the chokepoint on the data pipe going to the world. If any one part of the network gets overly busy, then network performance can degrade quickly.

What is scariest for network engineers is that traffic from the reaction to the coronavirus is being layered on top of networks that already have been experiencing steady growth. Most of my clients have been seeing year-over-year traffic volumes increases of 20% to 30%. If Verizon’s experience in indicative of what we’ll all see, then networks will see a year’s typical growth happen in just weeks. We’ve never experienced anything like this, and I’m guessing there aren’t a lot of network engineers who are sleeping well this week.

Setting the FCC Definition of Broadband

In the recently released 2018 Broadband Progress Report the FCC reluctantly kept the official definition of broadband at 25/3 Mbps. I say reluctantly because three of the Commissioners were on record for either eliminating the standard altogether or else reverting back to the older definition of 10/1 Mbps.

I’m guessing the Commissioners gave in to a lot of public pressure to keep the 25/3 standard. Several Commissioners had also taken a public stance that they wanted to allow cellular data to count the same for a household as landline broadband – and that desire was a big factor in lowering the definition since cellphones rarely meet the 25/3 speed standard.

The deliberation on the topic this year raises the question if there is some way to create a rule that would better define the speed of needed broadband. It’s worth looking back to see how the Tom Wheeler FCC came up with the 25/3 definition. They created sample profiles of the way that households of various sizes are likely to want to use broadband. In doing so, they added together the bandwidth needed for various tasks such as watching a movie or supporting a cellphone.

But the FCC’s method was too simple and used the assumption that various simultaneous uses of broadband are additive. They added together the uses for a typical family of four which resulted in bandwidth needs greater than 20 Mbps download, and used that as the basis for setting the 25/3 standard. But that’s now home broadband works. There are several factors that affect the actual amount of bandwidth being used:

For example, doing simultaneous tasks on a broadband network increases the overhead on the home network. If you are watching a single Netflix stream, the amount of needed bandwidth is predictable and steady. But if three people in a home are each watching a different Netflix the amount of needed bandwidth is greater than adding together the three theoretical streams. When your ISP and your home router try to receive and untangle multiple simultaneous streams there are collisions of packets that get lost and which have to be retransmitted. This is described as adding ‘overhead’ to the transmission process. Depending on the nature of the data streams the amount of collision overhead can be significant.

Almost nobody directly wires the signal from their ISP directly too all of their devices. Instead we use WiFi to move data around to various devices in the home. A WiFi router has an overhead of its own that adds to the overall bandwidth requirement. As I’ve covered in other blogs, a WiFi network is not impacted only by the things you are trying to do in your home, but a WiFi network is slowed when it pauses to recognizes demands for connection from your neighbor’s WiFi network.

Any definition of home broadband needs should reflect these overheads. If a household actually tries to download 25 Mbps of usage from half a dozen sources at the same time on a 25 Mbps, the various overheads and collisions will nearly crash the system.

The FCC’s definition of broadband also needs to reflect the real world. For example, most of the unique programming created by Netflix and Amazon Prime are now available in 4K. I bought a large TV last year and we now watch 4K when it’s available. That means a stream of 15-20 Mbps download. That stream forced me to upgrade my home WiFi network to bring a router into the room with the TV.

The FCC’s speed definition finally needs to consider the busy hour of the day – the time when a household uses the most broadband. That’s the broadband speed that the home needs.

We know household bandwidth needs keep increasing. Ten years ago I was happy with a 5 Mbps broadband product. Today I have a 60 Mbps product that seems adequate, but I know from tests I did last year that I would be unhappy with a 25 Mbps connection.

The FCC needs a methodology that would somehow measure actual download speeds at a number of homes over time to understand what homes area really using for bandwidth. There are ways that this could be done. For example, the FCC could do something similar for broadband like what Nielsen does for cable TV. The FCC could engage one of the industry firms that monitor broadband usage such as Akamai to sample a large number of US homes. There could be sample voluntary homes that meet specific demographics that would allow monitoring of their bandwidth usage. The accumulated data from these sample homes would provide real-life bandwidth usage as a guide to setting the FCC’s definition of broadband. Rather than changing the official speed periodically, the FCC could change the definition as needed as dictated by the real-world data.

The FCC does some spot checking today of the broadband speeds as reported by the ISPs that feed the national broadband map. But that sampling is random and periodic and doesn’t provide the same kind of feedback that a formal ongoing measuring program would show. We have tools that could give the FCC the kind of feedback it needs. Of course, there are also political and other factors used in setting the official definition of broadband, and so perhaps the FCC doesn’t want real facts to get into the way.