Data Usage Remains Robust in 2Q20

OpenVault recently published its Broadband Insights Report for the second quarter of 2020. Since OpenVault’s software is used to track usage in major Internet POPs, the company has a unique perspective on broadband usage in the country.

The report says that the peak of data usage this year was in March when people first reacted to the pandemic. Data usage is down slightly compared with the first quarter, but still much higher than data usage a year ago, In the second quarter the average home used 380 gigabytes of data per month. This is down 6% compared to the average usage in March 2020 of 403 gigabytes. But the second quarter data usage is up 36% over the average usage of 280 gigabytes per household used in the second quarter of 2019. Before the pandemic, household broadband usage was growing at a rate just above 20% annually, so the 36% growth in a year demonstrates the huge impact on the pandemic on broadband.

Median data usage has increased even faster than average usage. The median usage measures the middle point where half of homes use less and have of homes use more broadband. The median usage in the second quarter of 2019 was 144 gigabytes and has grown 54% in a year to 223 gigabytes. This indicates that even households that previously would have been light data users are now using a lot more data during the pandemic. This likely can comes from both increase cord-cutting as well as from students and adults working from home.

OpenVault reports that usage for homes with unlimited broadband plans (no data caps) grew even faster and increased by 42% over 2019. The company surmises that the big increase is at least partially because the big ISPs are not enforcing data caps during the pandemic. However, part of this increase is also likely due to an increase of what OpenVault calls power users. These are homes that use more than 1 terabyte of data per month.

In the second quarter 8.7% of homes used at least 1 terabyte of data per month, more than double the 4.1% of terabyte homes a year earlier. This now includes 1% of all homes that are using more than 2 terabytes of data, triple since a year earlier in 2019.

One reason for the higher data usage might be explained by households subscribing to faster data plans. At the end of the second quarter, 4.9% of homes are now subscribed to gigabit data speeds, more than double the 2.1% of gigabit subscribed in the second quarter of 2019. Over 61% of homes in the country are now subscribed to broadband speeds greater than 100 Mbps. That includes 37.8% subscribed to plans between 100 Mbps and 200 Mbps, 13.5% subscribed to plans between 200 Mbps and 400 Mbps, 5% subscribed to speeds between 400 Mbps and 900 Mbps, and 4.9% subscribed to gigabit speeds. Less than 20% of homes nationwide are subscribed to plans slower than 40 Mbps.

There is one segment of broadband usage that continued to increase in the second quarter of 2020. Upload usage from homes is up 56% over a year earlier. Upload demand is directly related to the need to connect for homes to connect to school and work servers and to take part in Zoom and other video conferencing services. It’s likely before the pandemic that many homes had never much needed the upload link from home.

What is most intriguing about the continued increase in upload demand is that upload usage continued to grow even after school semesters were ending for the year. During the second quarter tens of millions of upload links to school servers would have gone quiet as school semesters ended, and yet upload demand continued to grow. It’s going to be interesting to see what the fall school semester does to broadband usage.

Home Broadband Usage Explodes

ISPs have all been reporting increased bandwidth usage due to employees and students being asked to work from home during the COVID-19 pandemic. Perhaps the best proof we’ve seen yet of the huge increase in home broadband usage comes from OpenVault, which has been tracking home broadband usage for several years.

The company reported that as of the end of March that the average US home used 402.5 gigabytes of usage, up 17% from the 344.0 gigabytes reported just 3 months earlier at the end of 2019, and up 47% from the 273.5 gigabytes measured a year earlier. OpenVault says that most of the growth was realized in the last two weeks of March as employees and students started working from home in earnest.

The OpenVault numbers represent total bandwidth used by a home, meaning the numbers are a combination of download and upload usage. OpenVault validated the widely reported phenomenon that the demand for upload bandwidth is increasing far more than the need for downloading.

Another interesting way to look at broadband usage is by considering the median usage – which is the speed at which half of homes use more and half less broadband. The median broadband usage is the US has always been lower than average usage because of the large number of rural homes that are stuck using slow broadband connections. A home with a 1 Mbps download speed cannot easily use the same amount of bandwidth as a home with a 100 Mbps connection. Median usage for the first quarter was at 233.6 gigabytes, up 60% from 146.0 gigabytes from a year earlier, and up 22% from the 190.7 gigabytes used at the end of 2019. The big news in the growth of median speeds is that even homes with slower broadband connections are burning through more broadband.

One of the most startling numbers to come from OpenVault is what they call power users – homes that are using more than 1 terabyte of data per month. At the end of March, 10% of all US homes were using a terabyte of data, an increase of 138% over the 4.2% of homes that used a terabyte of data just three months earlier at the end of 2019. Even more interesting, 1.2% of homes used 2 terabytes of data at the end up march, up 215% from the end of December. The big ISPs like Comcast are supposedly not billing for data caps during the pandemic – but they must be licking their chops at the flood of new revenues this is going to create if broadband usage doesn’t return to pre-COVID levels.

We saw the demand for faster broadband products also leap upward. At the end of March the percentage of homes subscribing to gigabit data products jumped to 3.75% of homes, up from 2.8% at the end of 2019 and up from 1.9% a year earlier. Amazingly, more than 1% of all homes in the US upgraded to a gigabit data plan in just the last three months – that’s something that’s been predicted for years. Those homes are not likely going to downgrade to slower speeds – so gigabit broadband is now becoming a significant segment of the market. OpenVault says that 12% of US homes now subscribe to speeds of 200 Mbps or faster.

The OpenVault data also validates what’s been reported widely by ISPs – that the patten of broadband usage is changing by time of day. In the recent past the peak period for broadband usage – the busy hour – was always in the evenings. In the first quarter the amount of usage in the evenings was flat and all of the increased usage came during the daytime as employees and students used broadband and video conferences to function.

OpenVault says that usage peaked in the third week of March. It will be interesting going forward to see the how home usage changes. OpenVault doesn’t have any better crystal ball than the rest of us, but they are predicting that broadband usage will never return to the historic patters. They predict that a lot of people will continue to work from home, meaning increased broadband demand during the day. They believe there will be continued pressure on the upload data paths. People who have learned to videoconference during the recent months are likely to continue that practice in the future. Companies and employees that realize they can be productive at home are likely to work more from home, even if only on a part-time basis.

Broadband Consumption Continues Explosive Growth

OpenVault Just released its Broadband Industry Report for 4Q 2019 that tracks the way that the US consumes data. The results of the reports are as eye-opening as OpenVault reports for the last few years. OpenVault has been collecting broadband usage for more than ten years.

As usual, the OpenVault statistics are a wake-up cry for the industry. The most important finding is that the average monthly data consumed by households grow by 27% from 2018 to 2019, and in the fourth quarter of 2019 the average home used 344 gigabytes of data, up from 275 gigabytes a year earlier. Note that consumption is a combination of download and upload usage – with most consumption being downloaded.

For the first time, the company compared homes with unlimited data plans to those that have plans with data caps. They reported that homes with no data caps used 353 gigabytes per month while homes with data caps used 337 gigabytes per month. That statistic would suggest that homes with data caps try to curb their usage to avoid overage charges.

Interestingly, median usage was significantly different than average usage. Median means the same as midpoint, and the median data usage was 191 gigabytes per month, meaning half of US homes used more than that and half used less. In looking at their numbers, I have to suppose that the median is a lot less than average due to the many homes using slow DSL that can’t consume a lot of broadband.

The report also looks at power users – homes that consume a lot of broadband. They report that nearly 1% of homes now use 2 terabytes per month and 7.7% use over 1 terabyte per month. A terabyte is 1,000 gigabytes. The percentage of homes using over 1 terabyte climbed from 4% a year earlier. This statistic is important because it shows a quickly increasing number of homes that will be hitting the 1 terabyte data caps of ISPs like Comcast, AT&T, Cox, and CenturyLink. I clearly remember Comcast saying just a few years ago that almost no homes had an issue with their data caps, but that no longer can be true.

Homes are starting to buy 1 gigabit broadband when it’s available and affordable. 2.8% of homes in the country now subscribe to gigabit speeds, up 86% from the 1.5% of homes that bought gigabit in 2018.

54% of homes now purchase broadband plans with speeds of 100 Mbps or faster. Another 23.6% of homes are subscribing to broadband between 50-75 Mbps. This means that nearly 78% of homes are subscribing to data plans of greater than 50 Mbps. The average subscribed speed grew significantly since 2018 from 103 Mbps to 128 Mbps. These subscriber statistics should shame the FCC for deciding to stick with the 25/3 Mbps definition of broadband. The agency is clearly setting a target speed for rural America that is far behind the reality of the marketplace.

OpenVault made one comparison to Europe and showed that we consume a lot more broadband here. While the US average consumption of broadband in 4Q 2019 was 344 gigabytes here, it was 196 gigabytes in Europe.

As OpenVault statistics have done in the past, they show network engineers that the demand for new broadband is not abating but is continuing to explode. An annual 27% increase in broadband consumption means that broadband demand is continuing to double every three years. If that growth rate is sustained, then our networks need to be prepared within a decade to carry 8.6 times more data than today. That’s enough to keep network engineers up at night.

The Terabyte Household

I was just in a meeting the other day with a bunch of ISPs were talking about household downloads. Several said that they were now seeing monthly data usage exceed a terabyte, and those with Comcast were lamenting that this is causing them a lot of money.

I wrote a lot about Comcast data caps a few years ago when the company experimented with really low data caps of 300 gigabytes per month. At that time a lot of households complained that they were exceeding those caps. Comcast was arguing at the time to end net neutrality and I think this persuaded them to back off of the low caps, which they set to 1 terabyte.

Here we are only a few years later and a lot of households are bumping up against and exceeding that data cap. Comcast absolutely knew this was coming and they just pushed the ability to monetize data caps a few years into the future. As an ISP the company knows better than most that the household demand for total downloaded data has been doubling every three years or so. That kind of growth will push a huge number of households over a terabyte within a decade – with many already hitting it now.

Comcast tries to justify data caps by arguing fairness – the same argument they made a few years ago. They say that those that use the Internet the most ought to pay the most. Even if you can buy that argument the penalty for exceeding the data caps are excessive. Comcast doesn’t charge a household for the first two months they exceed a terabyte. After that they have two plans. They will automatically bill $10 for every extra 50 Gigabytes over the data cap – with total excess charges capped at $200 per month. Customers who expect to exceed the data cap can also agree to pay $50 extra every month to get unlimited usage.

Comcast goes on to explain away the terabyte cap by describing what it takes to exceed the cap, as follows:

  • Stream between 600 and 700 hours of HD video
  • Play online games for more than 12,000 hours
  • Stream more than 15,000 hours of music
  • Upload or download more than 60,000 hi-res photos

This explanation is simplistic for a number of reasons. First, full Netflix HD broadcast at 1080p streams at over 7 Mbps and uses roughly 2.5 GB per hour, meaning a terabyte will cover about 400 hours of full HD video. If you have a good broadband connection the chances are that you are watching a lot of 4K video today – it’s something that Netflix and Amazon Prime offer automatically. It only takes only about 180 hours of 4K video in a month to hit the terabyte data cap – a number that is not hard to imagine in a cord-cutting home. The chart also misses obvious large uses like downloading games – with download sizes over 40 GB for one game becoming common.

The Comcast charts also fail to recognize the hidden ways that we all burn through bandwidth today. It’s not untypical for the average household to have a 30% to 40% overhead on Internet usage. That comes from the network having to transmit data multiple times to complete a download request. This overhead is caused for a number of reasons. First are inefficiencies inherent in the open Internet. There are always packets lost on transit that much be sent multiple times. There are also delays caused by the ISP network, particularly networks that are undersized in neighborhoods and that hit capacity during the busy hours. The biggest cause of delays for most of us is in-home WiFi networks that creates a lot of collisions from competing signals.

There are also a lot of background use of the Internet today that surprises people. We now routinely use web storage to back up files. All of the software on our machines upgrade automatically. Many now use applications like video cameras and home alarms that connect in the cloud and that ping back and forth all day. All sorts of other things go on in the background that are a mystery – I’ve noticed my house has significant broadband usage even when we aren’t home. I’ve estimated that this background communication probably eats about 150 gigabytes per month at my house.

When I consider those issues the Comcast terabyte data caps are stingy. A household with a lot of network noise and with a lot of background traffic might hit the data caps using only half of a terabyte of downloaded video or other services like those listed by Comcast. A home today might hit the cap with 200 hours of full HD streaming or 90 hours of 4K streaming.

The other amazing aspect of the terabyte data caps is the charge for using more than a terabyte in a month. As mentioned above, Comcast charges $10 for every extra 50 GB. I’ve done the math for dozens of ISPs and most of my clients spend between $2 and $4 per month on average for the bandwidth per broadband customer. That number includes not only residential users, but for most ISPs also includes some huge commercial broadband customers. The average price varies the most according to how far an ISP is away from the Internet, and that component of the cost is fixed and doesn’t increase due to higher data volumes by the ISP. After backing out this fixed transport cost, my math says that an extra 50 GB of broadband costs an ISP only a few pennies. For a large ISP like Comcast that cost is significantly lower since they peer with the big broadband companies like Netflix, Google and Amazon – and traffic exchanged in those arrangements have nearly zero incremental cost of extra bandwidth.

Finally, the Comcast website claims that less than 1% of their users exceed the terabyte data caps. Only they know the numbers, but I find that hard to believe. When you look at the amount of usage needed to hit that cap there has to be a lot of cord-cutter households already exceeding a terabyte.

The bottom line is that Comcast is extorting homes when they force them to spend $50 per month for unlimited data usage. That extra bandwidth costs them almost nothing. Unfortunately, there isn’t a damned thing any of us can do about this any since Comcast and the other big ISPs got their wish and broadband is no longer regulated by the FCC.

The Race to Zero


This is my 500th blog entry, which means I have written several books worth of words. When I started the blog I feared I might run out of ideas in a few months, but our industry has become so dynamic that I am regularly handed more topics than there are days in the week.  

The cloud industry is often characterized by what is being called the race to zero. This is the phenomenon of ever-dropping prices for data storage. The race towards cheaper prices has always been driven by Amazon through repeatedly reducing their cloud storage prices. Every time they reduce the price of their AWS storage services, the other big cloud companies like Microsoft and Google always go along.

There are numerous reasons for the price drops, all having to do with improved computer technology. Memory storage devices have dropped in price regularly, while at the same time a number of new storage technologies are being used. The large cloud companies have moved to more efficient large data centers to gain economy of scale. And lately the large companies are all designing their own servers to be faster and more energy efficient, since energy prices for cooling are one of the largest costs of running a data center.

I remember in the late 90s looking to back up my company LAN offsite for the peace of mind of having a backup of our company records. At that time our data consisted of Word, Excel, PowerPoint, and Outlook files for around twenty people, and I’m sure that wasn’t more than a a dozen gigabytes of total data. I got a quote for $2,000 per month, which consisted of setting up a shadow server that would mimic everything done on my server, backed up once per day. At the time I found that was too expensive, so we stayed with using daily cassette back-up.

Let’s compare that number to today. There are now numerous web services that give away a free terabyte of storage. I can get a free terabyte from Flickr to back up photos. I can get the same thing from Oosah that allows me to store any kind of media and not just pictures. I can get a huge amount of free storage from companies like Dropbox to store and transmit almost anything. And the Chinese company Tencent is offering up to 10 terabytes of storage for free.

It’s hard for somebody who doesn’t work in a data center to understand all of the cost components for storage. I’ve seen estimates on tech sites that say that storage costs for a gigabyte’s worth of data have dropped from $9,000 in 1993 to around 3 cents in 2013. Regardless of how accurate those specific numbers are, they demonstrate the huge decrease in storage cost over the last few decades.

But consumers and businesses don’t necessarily see all of these savings, because the industry has gotten smarter and now mostly charges for value-added services rather than the actual storage. Take the backup service Carbonite as an example. Their service will give you unlimited cloud storage for whatever data you have on your computer. Their software then activates each night and backs up whatever changes you made on your computer during the day. This is all done by software and there are no people involved in the process.

Carbonite charges $59.99 per year to back up any one computer. For $99.99 per year you can add in one external hard drive to any one computer. And for $149.99 per year you can back up videos (not included in the other packages) plus they will courier you a copy of your data if you have a crash.

The value of Carbonite is that their software automatically backs you up once a day (and we all know we forget to do that). But that is not a complicated process and there have been external hard drives available for years with the same feature. But Carbonite is selling the peace-of-mind of not losing your data by putting it in the cloud. It must be a very profitable business since the cost of the actual data storage is incredibly cheap. Consider how much extra profit they make when somebody pays them $40 extra to back up an external hard drive.

In the business world, the fees paid for the cloud are all about software and storage cost isn’t an issue other than for someone who wants to store massive amounts of data. One might think the companies in the cloud business are selling offsite storage, but their real revenue comes from selling value-added software that helps you operate your business and manage your data. The storage costs are almost an afterthought these days.

The race to zero is not even close to over. In one of my blogs last week I talked about how using magnetized graphene might increase the storage capacity of devices by a million-fold. That upgrade is still in the labs, but it demonstrates that progress to ever-cheaper storage is far from done. We’ve come a long way from the 720 kb that I used to squeeze onto a floppy diskl!