The Upload Crisis

Carriers continue to report on the impact of COVID-19 on their networks. One of the more interesting statistics that caught my eye was when Comcast reported that upload traffic on their network was up 33% since March 1. Comcast joins the rest of big ISPs in saying that their networks are handling the increased traffic volumes.

By ‘handling’ the volumes they mean that their networks are not crashing and shutting down. But I think there is a whole lot more to these headlines than what they are telling the public.

I want to start with an anecdote. I was talking to a client who is working at home along with her husband and two teenagers. The two adults are trying to work from home and the two kids are supposed to be online keeping up with schoolwork. Each of them needs to create a VPN to connect to their office or school servers. They are also each supposed to be connecting to Zoom or other online services for various meetings, webinars, or classes.

These functions all rely on using the upload path to the Internet. The family found out early in the crisis that their broadband connection did not provide enough upload speed to create more than one VPN at a time or to join more than one video call. This has made their time working at home into a major hassle because they are being forced to schedule and take turns using the upload link. This is not working well for any of them since the family has to prioritize the most important connections while other family members miss out on expected calls or classes.

The family’s upload connection is a choke point in the network and is seriously limiting their ability to function during the stay-at-home crisis. But the story goes beyond that. We all recall times in the past when home Internet bogged down in the evenings when everybody in the neighborhood was using broadband to watch videos or play games. Such slowdowns occurred when the download data path into the neighborhood didn’t deliver enough bandwidth to satisfy everybody’s request for broadband. When that download path hit maximum usage, everybody in the neighborhood got a degraded broadband connection. When the download path got overloaded, the network responded by giving everybody a little less bandwidth than they were requesting – and that resulted in pixelating video or websites that lose a connection.

The same thing is now happening with the upload links, but the upload path is a lot more susceptible to overload.  For technologies like coaxial cable networks or telephone DSL the upload path leaving the neighborhood is a lot smaller than the download path into the area. As an example, the upload link on a coaxial network is set to be no more than 10% of the total bandwidth allowed for the neighborhood. It takes a lot more usage to overload the download path into the neighborhood since that path is so much larger. On the upload path, the homes are now competing for a much smaller data path.

Consider the difference in the way that homes use the download path compared to the new way we’re all using uploading. On the download side, networks get busy mostly due to streaming video. Services like Netflix stay ahead of demand by downloading content that will be viewed five minutes into the future. By doing so, the neighborhood download network can have cumulative delays of as much as five minutes before the video streams collapse and stop working. The very nature of streaming creates a buffer against failure – sort of a network insurance policy.

Homes are not using the upload links in the same way. Connecting to a school server, a work server, or a video chat service creates a virtual private network (VPN) connection. A VPN connection grabs and dedicates some minimum amount of bandwidth to the user even during times when the person might not be uploading anything. A VPN carves out a small dedicated path through the upload broadband connection provided by the ISP. There is no buffer like there is with downloading of streaming video – when the upload path gets full, there’s no room for anybody else to connect.

The nearest analogy to this situation harkens back to traditional landline telephone service. We all remember times, like after 911, when you couldn’t make a phone call because all of the circuits were busy. That’s what’s happening with the increased use of VPNs. Once the upload path from the neighborhood is full of VPNs, nobody else is going to be able to grab a VPN connection until somebody ‘hangs up’.

Residential customers have historically valued download speeds over upload speeds and ISPs have configured their networks accordingly. Many technologies allow an ISP to balance the upload and download traffic, and ISPs can help upload congestion by providing a little more bandwidth on the upload stream. Unfortunately for cable companies, the current DOCSIS standards don’t allow them to provide more than 10% of bandwidth on the upload side – so their ability to balance is limited.

As I keep hearing these stories from real users I am growing less and less impressed by the big ISPs saying that everything is well and that their networks are handling the increased load. I think there are millions of households struggling due to inadequate upload speeds. It’s true, as the big ISPs are reporting, that the networks are not crashing – but the networks are not providing the connections people want to make. No big ISP is going to admit this to their stockholders – but I bet a lot of those stockholders already understand this first-hand from having troubles trying to work from home.

Expect a New Busy Hour

One of the many consequences of the coronavirus is that networks are going to see a shift in busy hour traffic. Busy hour traffic is just what is sounds like – it’s the time of the day when a network is busiest, and network engineers design networks to accommodate the expected peak amount of bandwidth usage.

Verizon reported on March 18 that in the week since people started moving to work from home that they’ve seen a 20% overall increase in broadband traffic. Verizon says that gaming traffic is up 75% as those stuck at home are turning to gaming for entertainment. They also report that VPN (virtual private network) traffic is up 34%. A lot of connections between homes and corporate and school WANs are using a VPN.

These are the kind of increases that can scare network engineers, because Verizon just saw a typical year’s growth in traffic happen in a week. Unfortunately, the announced Verizon traffic increases aren’t even the whole story since we’re just at the beginning of the response to the coronavirus. There are still companies figuring out how to give secure access to company servers and the work-from-home traffic is bound to grow in the next few weeks. I think we’ll see a big jump in video conference traffic on platforms like Zoom as more meeting move online as an alternative to live meetings.

For most of my clients, the busy hour has been in the evening when many homes watch video or play online games. The new paradigm has to be scaring network engineers. There is now likely going to be a lot of online video watching and gaming during the daytime in addition to the evening. The added traffic for those working from home is probably the most worrisome traffic since a VPN connection to a corporate WAN will tie up a dedicated path through the Internet backbone – bandwidth that isn’t shared with others. We’ve never worried about VPN traffic when it was a small percentage of total traffic – but it could become one of the biggest continual daytime uses of bandwidth. All of the work that used to occur between employees and the corporate server inside of the business is now going to traverse the Internet.

I’m sure network engineers everywhere are keeping an eye on the changing traffic, particularly to the amount of broadband used during the busy hour. There are a few ways that the busy hour impacts an ISP. First, they must buy enough bandwidth to the Internet to accommodate everybody. It’s typical to buy at least 15% to 20% more bandwidth than is expected for the busy hour. If the size of the busy hour shoots higher, network engineers are going to have to quickly buy a larger pipe to the Internet, or else customer performance will suffer.

Network engineers also keep a close eye on their network utilization. For example, most networks operate with some rule of thumb, such as it’s time to upgrade electronics when any part of the network hits some pre-determined threshold like 85% utilization. These rules of thumb have been developed over the years as warning signs to provide time to make upgrades.

The explosion of traffic due to the coronavirus, might shoot many networks past these warning signs and networks start experiencing chokepoints that weren’t anticipated just a few weeks earlier. Most networks have numerous possible chokepoints – and each is monitored. For example, there is usually a chokepoint going into neighborhoods. There are often chokepoints on fiber rings. There might be chokepoints on switch and router capacity at the network hub. There can be the chokepoint on the data pipe going to the world. If any one part of the network gets overly busy, then network performance can degrade quickly.

What is scariest for network engineers is that traffic from the reaction to the coronavirus is being layered on top of networks that already have been experiencing steady growth. Most of my clients have been seeing year-over-year traffic volumes increases of 20% to 30%. If Verizon’s experience in indicative of what we’ll all see, then networks will see a year’s typical growth happen in just weeks. We’ve never experienced anything like this, and I’m guessing there aren’t a lot of network engineers who are sleeping well this week.

How Do VPNs Work?

After Congress clarified last month that ISPs have the right to monitor and use customer data I have read dozens of articles that recommend that people start using VPNs (Virtual Private Networks) to limit ISP access to their data. I’ve received several emails asking how VPNs work and will discuss the technology today.

Definition. A VPN is a virtualized extension of a private network across a public network, like the open Internet. What that means in plain English is that VPN technology tries to mimic the same kind of secure connection that you would have in an office environment where your computer is directly connected to a corporate server. In a hard-wired environment everything is secure between the server and the users and all data is safe from anybody that does not have access to the private network. If the private network is not connected to the outside world, then somebody would have to have a physical connection to the network in order to read data on the private network.

Aspects of a VPN Connection. There are several different aspects that are used to create the virtualized connection. A VPN connection today likely includes all of the following:

  • Authentication. A VPN connection always starts with authentication to verify the identity of the remote party that wants to make the VPN connection. This could use typical techniques such as passwords, biometrics or two-factor authentication.
  • Encryption. Most VPN connections then use encryption for the transmission of all data once the user has been authenticated. This is generally done by placing software on the user’s computer that scrambles the data and that can only be unscrambled at the VPN server using the same software. Encryption is not a foolproof technique and the Edward Snowden documents proved that the NSA knows how to read most kinds of encryption – but it’s still a highly effective technique to use for the general transmission of data.
  • IP Address Substitution. This is the technique that stops ISPs from seeing a customer’s Internet searches. When you use your ISP without a VPN, your ISP assigns you an IP address to identify you. This ISP-assigned IP address then can be used by anybody on the Internet to identify you and to track your location. Further, once connected your ISP makes all connections for you on the Internet using DNS (Domain Name Servers). For instance, if you want to visit this blog, your ISP is the one that finds PotsandPansbyCCG and makes the connection using the DNS system, which is basically a huge roadmap of the public Internet. Since they are doing the routing your ISP has complete knowledge of every website you visit (your browsing history).  But when you use a VPN, the VPN provider provides you with a new IP address, one that is not specifically identified as you. When you visit a website for the first time using the new VPN-provided IP address that website does not know your real location, but rather the location of the VPN provider. And since the VPN provider also does the DNS function for you (routes you to web pages) your ISP no longer knows your browsing history. Of course, this means that the VPN provider now knows your browsing history, so it’s vital to pick a VPN that guarantees not to use that information.

Different VPN Protocols and Techniques. This blog is too short to explore the various different software techniques used to make VPN connections. For example, early VPNs were created with the PPTP (Point-to-Point Tunneling Protocol). This early technique would encapsulate your data into larger packets but didn’t encrypt it. It’s still used today and is still more secure than a direct connection on the open Internet. There are other VPN techniques such as IPSec (IP Security), L2TP (Layer 2 Tunneling Protocol), SSL and TLS (Secure Socket Layer and Transport Layer Security), and SSH (Secure Shell). Each of these techniques handles authentication and encryption in different ways.

How Safe is a VPN? A VPN is a way to do things on the web in such a manner that your ISP no longer knows what you are doing. A VPN also establishes an encrypted and secure connection that makes it far harder for somebody to intercept your web traffic (such as when you make a connection through a hotel or coffee shop WiFi network). In general practice a VPN is extremely safe because somebody would need to expend a huge amount of effort to intercept and decrypt everything you are doing. Unless somebody like the NSA was watching you, it’s incredibly unlikely that anybody else would ever expend the effort to try to figure out what you are doing on the Internet.

But a VPN does not mean that everything you do on the Internet is now safe from monitoring by others. Any time you connect to a web service, that site will know everything you do while connected there. The giant web services like Google and Facebook derive most of their revenues by monitoring what you do while using one of their services and then use that information to create a profile about you.  Using a VPN does not stop this, because once you use the Google search engine or log onto Facebook they record your actions.

Users who want to be protective of their identities are starting to avoid these big public services. There are search engines other than Google that don’t track you. You can use a VPN to mask your real identify on social media sites. For example, there are millions of Twitter accounts that are not specifically linked back to the actual user. But a VPN or a fake identity can’t help you if you use a social media site like Facebook where you make connections to real-life friends. I recall an article a few years back from a data scientist who said that he only needed to know three facts about you to figure out online who you are. Companies like Facebook will quickly figure out your identity regardless of how you got to their site.

But a VPN will completely mask your web usage from your ISP. The VPN process bypasses the ISP and instead makes a direct, and encrypted connection to the VPN provider instead. A VPN can be used on any kind of data connection and you can use a VPN for home computers and also for cellphones. So if you don’t want Comcast or AT&T to monitor you and use and sell your browsing history to others, then a VPN service will cut your ISPs out of the loop.

Productizing Safety

padlockThe Internet is becoming a scarier place by the day to the average user. It seems like a week doesn’t go by when there isn’t news of some new and huge data breach or other nefarious use of the web. But as much as those big events might create a general industry sense of unease, these announcements also make people worried about their own individual Internet security.

The big ISPs like AT&T crow about recording and monetizing everything that their customers do on the web. And with a likely weakening or elimination of Title II regulation by the FCC this is likely to intensify. Every web site parks cookies on the computers of their visitors, and the bigger sites like Facebook and Google gather every fact fed to them and peddle it to the advertising machine. There are hackers that lock down PCs and hold them hostage until the owner pays a ransom. There are smart TVs that listen to us and IoT devices that track our movements inside our homes. There was news this week that smartphones with a certain Chinese chip have been sending every keystroke back to somebody in China.

All of this has to be making the average Internet user uneasy. And that makes me wonder if there is not a product of some sort that smaller ISPs can offer to customers that can make them feel safer on the web.

Savvy Internet users already take steps to protect themselves. They use ad blockers to reduce cookies. They use browsers like DuckDuckGo that don’t track them. They use encryption and visit sites using HTTPS. They scrub their machine regularly of cookies and extra and unidentified files. In the extreme some use a VPN to keep their ISP from spying on them.

Small ISPs are generally the good guys in the industry and don’t engage in the practices used by AT&T, Comcast and Verizon. I know some small ISPs that try to communicate to their customers about safety. But I think safety is now one of the biggest worries for people and I think small ISPs can do more.

Customers can really use the help. It’s easy to assume that customers ought to understand basic safety procedures, but the vast majority of them load some sort of virus protection on their PC the day they buy it and never think of safety again. They repeatedly do all of the bad things that lead to trouble. They open attachments on emails. They don’t update their software to have the latest security patches. They use social media and other sites without setting basic privacy filters.

I think there is an opportunity for small ISPs to be proactive in helping to make their customers feel safer, and in the process can create more loyal customers. I think there are two possible ways to undertake this. One is an intensive education campaign to inform customers about better web practices. I’m not talking about the occasional safety reminder, but instead a steady and concentrated effort to tell your customers ways to be safer on the web. Brand yourself as being a provider that is looking out for their safety. But don’t pay it lip service – do it in a proactive and concentrated way.

I also think there is a space for a ‘safety’ product line. For example, I have clients who run a local version of the Geek Squad and who repair and maintain people’s computers. It would not be hard to expand on that idea and to put together a ‘safety’ package to sell to customers.

Customers could have a service tech come to their home for a day each year and you could ‘fix’ all of their safety weaknesses. That might mean installing ad blockers and a spyware scrubber. It would mean updating their browsers and other software to the latest version. It could mean helping them to safely remove software they don’t use including the junkware that comes with new computers. It might include making sure they are using HTTPS everywhere. It also might mean selling a VPN for those who want the highest level of security.

I have clients who have been selling this kind of service to businesses for years, but I can’t think of anybody who does this in any meaningful way for residential customers. But since the web is getting less safe by the day there has to be an opportunity for small ISPs to distinguish themselves from larger competitors and to also provide a needed service – for pay of course.