The Upload Crisis

Carriers continue to report on the impact of COVID-19 on their networks. One of the more interesting statistics that caught my eye was when Comcast reported that upload traffic on their network was up 33% since March 1. Comcast joins the rest of big ISPs in saying that their networks are handling the increased traffic volumes.

By ‘handling’ the volumes they mean that their networks are not crashing and shutting down. But I think there is a whole lot more to these headlines than what they are telling the public.

I want to start with an anecdote. I was talking to a client who is working at home along with her husband and two teenagers. The two adults are trying to work from home and the two kids are supposed to be online keeping up with schoolwork. Each of them needs to create a VPN to connect to their office or school servers. They are also each supposed to be connecting to Zoom or other online services for various meetings, webinars, or classes.

These functions all rely on using the upload path to the Internet. The family found out early in the crisis that their broadband connection did not provide enough upload speed to create more than one VPN at a time or to join more than one video call. This has made their time working at home into a major hassle because they are being forced to schedule and take turns using the upload link. This is not working well for any of them since the family has to prioritize the most important connections while other family members miss out on expected calls or classes.

The family’s upload connection is a choke point in the network and is seriously limiting their ability to function during the stay-at-home crisis. But the story goes beyond that. We all recall times in the past when home Internet bogged down in the evenings when everybody in the neighborhood was using broadband to watch videos or play games. Such slowdowns occurred when the download data path into the neighborhood didn’t deliver enough bandwidth to satisfy everybody’s request for broadband. When that download path hit maximum usage, everybody in the neighborhood got a degraded broadband connection. When the download path got overloaded, the network responded by giving everybody a little less bandwidth than they were requesting – and that resulted in pixelating video or websites that lose a connection.

The same thing is now happening with the upload links, but the upload path is a lot more susceptible to overload.  For technologies like coaxial cable networks or telephone DSL the upload path leaving the neighborhood is a lot smaller than the download path into the area. As an example, the upload link on a coaxial network is set to be no more than 10% of the total bandwidth allowed for the neighborhood. It takes a lot more usage to overload the download path into the neighborhood since that path is so much larger. On the upload path, the homes are now competing for a much smaller data path.

Consider the difference in the way that homes use the download path compared to the new way we’re all using uploading. On the download side, networks get busy mostly due to streaming video. Services like Netflix stay ahead of demand by downloading content that will be viewed five minutes into the future. By doing so, the neighborhood download network can have cumulative delays of as much as five minutes before the video streams collapse and stop working. The very nature of streaming creates a buffer against failure – sort of a network insurance policy.

Homes are not using the upload links in the same way. Connecting to a school server, a work server, or a video chat service creates a virtual private network (VPN) connection. A VPN connection grabs and dedicates some minimum amount of bandwidth to the user even during times when the person might not be uploading anything. A VPN carves out a small dedicated path through the upload broadband connection provided by the ISP. There is no buffer like there is with downloading of streaming video – when the upload path gets full, there’s no room for anybody else to connect.

The nearest analogy to this situation harkens back to traditional landline telephone service. We all remember times, like after 911, when you couldn’t make a phone call because all of the circuits were busy. That’s what’s happening with the increased use of VPNs. Once the upload path from the neighborhood is full of VPNs, nobody else is going to be able to grab a VPN connection until somebody ‘hangs up’.

Residential customers have historically valued download speeds over upload speeds and ISPs have configured their networks accordingly. Many technologies allow an ISP to balance the upload and download traffic, and ISPs can help upload congestion by providing a little more bandwidth on the upload stream. Unfortunately for cable companies, the current DOCSIS standards don’t allow them to provide more than 10% of bandwidth on the upload side – so their ability to balance is limited.

As I keep hearing these stories from real users I am growing less and less impressed by the big ISPs saying that everything is well and that their networks are handling the increased load. I think there are millions of households struggling due to inadequate upload speeds. It’s true, as the big ISPs are reporting, that the networks are not crashing – but the networks are not providing the connections people want to make. No big ISP is going to admit this to their stockholders – but I bet a lot of those stockholders already understand this first-hand from having troubles trying to work from home.

Predictions for a Post-COVID-19 World

While it might still be too early to make predictions, there are dozens of articles on the web predicting how the COVID-19 pandemic might change our long-term behavior. Here are some of the more interesting predictions I’ve seen that involve broadband and telecom:

An Outcry for Better Home Broadband. Millions of people were sent home for work or school to homes that didn’t have good broadband. These folks have been telling the world for years that they don’t have good broadband. When this crisis is over these people are going to insist on being heard, and they are going to take out their anger on politicians who don’t help to find broadband solutions. This means Mayors and City Councils that are not pro-broadband. This means County Boards and Commissions that don’t offer matching grants to attract ISPs. This means any state politician who votes against significant state broadband grants or who votes against municipal participation in broadband. And this means federal Senators and Representatives that support the big cable companies and telcos over their constituents. Folks are not likely to be fooled any longer by false legislation that supposedly is pro-broadband but which is the exact opposite – because folks are going to be paying attention to any news concerning their home broadband.

Digital Meetings Are Here to Stay. We are all seeing how effective it can be to meet online. People are going to be a lot less willing to travel for a one or two-hour meeting. I know my days of doing that kind of traveling are over. This means airline business travel is likely never coming back to former levels, but it means a lot more. I was talking to somebody in local government the other day who told me that they spend over 10 hours of every workweek driving between meetings around a large county. He said he thinks the day or required live attendance at such meetings is likely over.

Demand for Faster Upload Speeds. The permanent uptick in more video meetings means there will be an increased demand for faster upload broadband speeds. The FCC still talks about 25/3 Mbps as acceptable broadband, but a home or office getting only 3 Mbps upload is not able to hold multiple simultaneous video calls. Homes and businesses are going to favor technologies willing to meet that upload speed demand.

Telemedicine has Arrived. I have been watching the glacial acceptance of telemedicine for fifteen years. The biggest hurdles have been the reluctance of doctors to try telemedicine and the willingness of insurance companies to pay for it. We’ve broken both of those barriers and telemedicine is here to stay. There are numerous routine doctor visits that don’t require an office visit. It’s never made sense to force patients who aren’t sick to march through a waiting room that has been filled all day with those with colds, the flu, or worse.

Expect Contactless Payments. I can remember being promised twenty years ago that we’d be able to pay for things by waving a cellphone. Nobody wants to hand a credit card to a clerk or even pass a credit card through a device that other people have used all day – so stores that install touchless payment systems are quickly going to become preferred. Expect an expansion of telephone, voice, and vision interface at checkout locations and a phase-out of credit card swiping. Also, expect an increased reluctance to take cash. There were already stores in New York City last year that made headlines by refusing to accept cash – expect a lot more of that.

More Telecommuting. Businesses have seen that people can be effective when working from home. Expect to see businesses more easily allowing for working from home at least part-time. This likely means a downturn in business real estate. For example, my neighbor is an architect who works at a small local branch of a larger firm. They’ve already seen the effectiveness of working from home and have already discussed not reopening the local office when the crisis is over. More telecommuting means more daytime use of neighborhood bandwidth and an increased expectation of residential broadband signal quality.

A Reboot for Corporate Security. We just spent a decade moving corporate data behind firewalls and restricting access to data from outside the business. Many businesses scrambled to find ways to allow employees to work from home, and in doing so undid many of their security protocols. Expect a major reboot as companies implement security solutions that support telecommuting.

The Demand for Upload Speeds

I was recently at a public meeting about broadband in Davis, California and got a good reminder of why upload speeds are as important to a community as download speeds. One of the people making public comments talked about how uploading was essential to his household and how the current broadband products on the market were not sufficient for his family.

This man needed good upload speeds for several reasons. First, he works as a photographer and takes pictures and shoots videos. He says that it takes hours to upload and send raw, uncompressed video to one of his customers and says the experience still feels like the dial-up days. His full-time job is working as a network security consultant for a company that specializes in big data. As such he needs to send and receive large files, and his home upload bandwidth is also inadequate for that – forcing him to go to an office for work that could otherwise be done from his home. Finally, his daughter creates YouTube content and has the same problem uploading content – which is particularly a problem when her content deals with time-sensitive current events and waiting four hours to get the content to YouTube kills the timeliness of her content.

This family is not unusual any more. A decade ago, a photographer led the community effort to get faster broadband in a city I was working with. But he was the only one asking for faster upload speeds and most homes didn’t care about it.

Today a lot of homes need faster upload speeds. This particular family had numerous reasons including working from home, sending large data files and posting original content to the web. But these aren’t the only uses for faster upload speeds. Gamers now need faster upload speeds. Anybody who wants to remotely check their home security cameras cares about upload speeds. And more and more people are migrating to 2-way video communications, which requires those at both ends to have decent uploading. We are just now seeing the early trials of virtual presence where communications will be by big-bandwidth virtual holograms at each end of the communications.

Davis is like many urban areas in that the broadband products available have slow upload speeds. Comcast is the cable incumbent, and while they recently introduced a gigabit download product, their upload speeds are still paltry. DSL is offered by AT&T which has even slower upload speeds.

Technologies differ in their ability to offer upload speeds. For instance, DSL is technically capable of sending the data at the same speeds for upload or download. But DSL providers have elected to stress the download speed, which is what most people value. So DSL products are set with small upload and a lot of download. It would be possible to give a customer the choice to vary the mix between upload and download speeds, but I’ve never heard of an ISP who tried to provide this as an option to customers.

Cable modems are a different story. Historically the small upload speeds were baked directly into the DOCSIS standard. When Cable Labs created DOCSIS they made upload speeds small in response to what cable companies asked from them. Until recently, cable companies have had no option to increase upload speeds beyond the DOCSIS constraints. But Cable Labs recently amended the new DOCSIS 3.1 standard to allow for much upload speeds of nearly a gigabit. The first release of the new DOCSIS 3.1 standard didn’t include this, but it’s now available.

However, a cable company has to make sacrifices in their network if they want to offer faster uploads. It takes about 24 empty channels (meaning no TV signal) on a cable system to provide gigabit download speeds. A cable company would need to vacate many more channels of programming to also offer faster uploads and I don’t think many of them will elect to do so. Programming is still king and cable owners need to balance the demand for more channels compared to demand for faster uploads.

Fiber has no real constraints on upload speeds up to the capability of the lasers. The common technologies being used for residential fiber all allow for gigabit upload speeds. Many fiber providers set speeds to symmetrical, but others have elected to limit upload speeds. The reason I’ve heard for that is to limit the attractiveness of their network for spammers and others who would steal the use of fast uploading. But even these networks offer upload speeds that are far faster than the cable company products.

As more households want to use uploading we are going to hear more demands for a faster upload option. But for now, if you want super-fast upload speeds you have to be lucky enough to live in a neighborhood with fiber-to-the-home.

Should an ISP Offer Fast Upload Speeds?

Speed_Street_SignOne question I am often asked is if clients should offer symmetrical data speeds for residential customers. I’ve noticed lately a number of fiber networks that are advertising symmetrical speeds, and so this option is gaining some market traction. This is not an easy decision to make and there are a lot of different factors to consider:

The Competition. Most fiber networks are competing against cable networks, and the HFC technology on those networks does not allow for very fast uploading. The number one complaint that cable companies get about upload speeds is from gamers who want fast low-latency upload paths. But they say that they get very few other complaints from residential customers about this issue.

So this leads me to ask if residential customers care as much about upload speeds as they do download speeds. I know that today that household use the bulk of their download capabilities to view video and there are very few households that have the desire to upload videos in the same manner or volume. One of the questions I ask clients is if they are just trying to prove that their network is faster. Because to promote something heavily that most customers don’t care about feels somewhat gimmicky.

Practical. At the residential level there are not many users who have enough legal content to justify a fast upload. There are a few legitimate uses of uploading, but not nearly as many as there are for downloading. Some of the normal uses for uploading include gaming, sending large files, sharing videos and pictures with friends and family, doing data backup and other related activities into the cloud. But these uses normally do not generate as much traffic as the download bandwidth that is used by most households to watch video. And so one must ask the practical question if offering symmetrical bandwidth is just a marketing ploy since customers are not expected to use the upload nearly as much as they download.

Cost. Another consideration is cost, or lack of cost. A lot of ISPs buy symmetrical data pipes on their connection to the Internet. To the extent that they download a lot more data than is uploaded, one can almost look at the excess headroom on the upload side as free. They are already paying for that bandwidth and often there is no incremental cost to an ISP for customers to upload more except at  the point where upload becomes greater than download.

Technical. One must ask if allowing symmetrical bandwidth will increase demand for uploading over time. We know that offering faster download speeds induces homes to watch more video, but it’s not clear if this is true in the upload direction. If uploading is stimulated over time then there are network issues to consider. It requires a more robust distribution network to support a network that has significant traffic in both directions. For example, most fiber networks are built in nodes of some sort and the fiber connection to those nodes needs to be larger to support two-way traffic than it would be if the traffic is almost entirely in the download direction.

Bad Behavior. One of the main arguments against offering fast upload speeds is that it can promote bad behavior or can draw attention from those with malicious intents. For example, fast upload speeds might promote more use of file sharing, and most of the content shared on file sharing sites is copyrighted and being illegally shared.

There has always been the concern that customers also might set up servers on fast connections that can upload things quickly. And one of the few things that requires a fast upward connection is porn. So I’ve always found it likely that having fast upload connections is going to attract people who want to operate porn servers.

But the real concern is that fast networks can become targets for those with malicious intent. Historically hackers took over computers to generate spam. That still happens today, but there are other more malicious reasons for hackers to take over computers. For instance, hackers who launch denial of service attacks do so by taking over many computers and directing them to send messages to a target simultaneously. Computers are also being hijacked to do things like mine bitcoins, which requires frequent communication outward.

One would think that a hacker would find a computer sitting on a network that allows 100 Mbps or 1 Gbps upload to be worth a whole lot more than a computer on a slower network. And so they might well be targeting customer on these networks.

What this all means to me is that if you offer fast upload connections that you ought to be prepared to monitor customer to know which ones upload a lot. If such customers are operating server businesses they might be directed to use business products. Or you can help them find and remove malware if their computers have been hacked. But I find the idea of allowing fast uploads without monitoring to be dangerous for the ISP and for customers.