The Terabyte Household

I was just in a meeting the other day with a bunch of ISPs were talking about household downloads. Several said that they were now seeing monthly data usage exceed a terabyte, and those with Comcast were lamenting that this is causing them a lot of money.

I wrote a lot about Comcast data caps a few years ago when the company experimented with really low data caps of 300 gigabytes per month. At that time a lot of households complained that they were exceeding those caps. Comcast was arguing at the time to end net neutrality and I think this persuaded them to back off of the low caps, which they set to 1 terabyte.

Here we are only a few years later and a lot of households are bumping up against and exceeding that data cap. Comcast absolutely knew this was coming and they just pushed the ability to monetize data caps a few years into the future. As an ISP the company knows better than most that the household demand for total downloaded data has been doubling every three years or so. That kind of growth will push a huge number of households over a terabyte within a decade – with many already hitting it now.

Comcast tries to justify data caps by arguing fairness – the same argument they made a few years ago. They say that those that use the Internet the most ought to pay the most. Even if you can buy that argument the penalty for exceeding the data caps are excessive. Comcast doesn’t charge a household for the first two months they exceed a terabyte. After that they have two plans. They will automatically bill $10 for every extra 50 Gigabytes over the data cap – with total excess charges capped at $200 per month. Customers who expect to exceed the data cap can also agree to pay $50 extra every month to get unlimited usage.

Comcast goes on to explain away the terabyte cap by describing what it takes to exceed the cap, as follows:

  • Stream between 600 and 700 hours of HD video
  • Play online games for more than 12,000 hours
  • Stream more than 15,000 hours of music
  • Upload or download more than 60,000 hi-res photos

This explanation is simplistic for a number of reasons. First, full Netflix HD broadcast at 1080p streams at over 7 Mbps and uses roughly 2.5 GB per hour, meaning a terabyte will cover about 400 hours of full HD video. If you have a good broadband connection the chances are that you are watching a lot of 4K video today – it’s something that Netflix and Amazon Prime offer automatically. It only takes only about 180 hours of 4K video in a month to hit the terabyte data cap – a number that is not hard to imagine in a cord-cutting home. The chart also misses obvious large uses like downloading games – with download sizes over 40 GB for one game becoming common.

The Comcast charts also fail to recognize the hidden ways that we all burn through bandwidth today. It’s not untypical for the average household to have a 30% to 40% overhead on Internet usage. That comes from the network having to transmit data multiple times to complete a download request. This overhead is caused for a number of reasons. First are inefficiencies inherent in the open Internet. There are always packets lost on transit that much be sent multiple times. There are also delays caused by the ISP network, particularly networks that are undersized in neighborhoods and that hit capacity during the busy hours. The biggest cause of delays for most of us is in-home WiFi networks that creates a lot of collisions from competing signals.

There are also a lot of background use of the Internet today that surprises people. We now routinely use web storage to back up files. All of the software on our machines upgrade automatically. Many now use applications like video cameras and home alarms that connect in the cloud and that ping back and forth all day. All sorts of other things go on in the background that are a mystery – I’ve noticed my house has significant broadband usage even when we aren’t home. I’ve estimated that this background communication probably eats about 150 gigabytes per month at my house.

When I consider those issues the Comcast terabyte data caps are stingy. A household with a lot of network noise and with a lot of background traffic might hit the data caps using only half of a terabyte of downloaded video or other services like those listed by Comcast. A home today might hit the cap with 200 hours of full HD streaming or 90 hours of 4K streaming.

The other amazing aspect of the terabyte data caps is the charge for using more than a terabyte in a month. As mentioned above, Comcast charges $10 for every extra 50 GB. I’ve done the math for dozens of ISPs and most of my clients spend between $2 and $4 per month on average for the bandwidth per broadband customer. That number includes not only residential users, but for most ISPs also includes some huge commercial broadband customers. The average price varies the most according to how far an ISP is away from the Internet, and that component of the cost is fixed and doesn’t increase due to higher data volumes by the ISP. After backing out this fixed transport cost, my math says that an extra 50 GB of broadband costs an ISP only a few pennies. For a large ISP like Comcast that cost is significantly lower since they peer with the big broadband companies like Netflix, Google and Amazon – and traffic exchanged in those arrangements have nearly zero incremental cost of extra bandwidth.

Finally, the Comcast website claims that less than 1% of their users exceed the terabyte data caps. Only they know the numbers, but I find that hard to believe. When you look at the amount of usage needed to hit that cap there has to be a lot of cord-cutter households already exceeding a terabyte.

The bottom line is that Comcast is extorting homes when they force them to spend $50 per month for unlimited data usage. That extra bandwidth costs them almost nothing. Unfortunately, there isn’t a damned thing any of us can do about this any since Comcast and the other big ISPs got their wish and broadband is no longer regulated by the FCC.

The Race to Zero

Zero

This is my 500th blog entry, which means I have written several books worth of words. When I started the blog I feared I might run out of ideas in a few months, but our industry has become so dynamic that I am regularly handed more topics than there are days in the week.  

The cloud industry is often characterized by what is being called the race to zero. This is the phenomenon of ever-dropping prices for data storage. The race towards cheaper prices has always been driven by Amazon through repeatedly reducing their cloud storage prices. Every time they reduce the price of their AWS storage services, the other big cloud companies like Microsoft and Google always go along.

There are numerous reasons for the price drops, all having to do with improved computer technology. Memory storage devices have dropped in price regularly, while at the same time a number of new storage technologies are being used. The large cloud companies have moved to more efficient large data centers to gain economy of scale. And lately the large companies are all designing their own servers to be faster and more energy efficient, since energy prices for cooling are one of the largest costs of running a data center.

I remember in the late 90s looking to back up my company LAN offsite for the peace of mind of having a backup of our company records. At that time our data consisted of Word, Excel, PowerPoint, and Outlook files for around twenty people, and I’m sure that wasn’t more than a a dozen gigabytes of total data. I got a quote for $2,000 per month, which consisted of setting up a shadow server that would mimic everything done on my server, backed up once per day. At the time I found that was too expensive, so we stayed with using daily cassette back-up.

Let’s compare that number to today. There are now numerous web services that give away a free terabyte of storage. I can get a free terabyte from Flickr to back up photos. I can get the same thing from Oosah that allows me to store any kind of media and not just pictures. I can get a huge amount of free storage from companies like Dropbox to store and transmit almost anything. And the Chinese company Tencent is offering up to 10 terabytes of storage for free.

It’s hard for somebody who doesn’t work in a data center to understand all of the cost components for storage. I’ve seen estimates on tech sites that say that storage costs for a gigabyte’s worth of data have dropped from $9,000 in 1993 to around 3 cents in 2013. Regardless of how accurate those specific numbers are, they demonstrate the huge decrease in storage cost over the last few decades.

But consumers and businesses don’t necessarily see all of these savings, because the industry has gotten smarter and now mostly charges for value-added services rather than the actual storage. Take the backup service Carbonite as an example. Their service will give you unlimited cloud storage for whatever data you have on your computer. Their software then activates each night and backs up whatever changes you made on your computer during the day. This is all done by software and there are no people involved in the process.

Carbonite charges $59.99 per year to back up any one computer. For $99.99 per year you can add in one external hard drive to any one computer. And for $149.99 per year you can back up videos (not included in the other packages) plus they will courier you a copy of your data if you have a crash.

The value of Carbonite is that their software automatically backs you up once a day (and we all know we forget to do that). But that is not a complicated process and there have been external hard drives available for years with the same feature. But Carbonite is selling the peace-of-mind of not losing your data by putting it in the cloud. It must be a very profitable business since the cost of the actual data storage is incredibly cheap. Consider how much extra profit they make when somebody pays them $40 extra to back up an external hard drive.

In the business world, the fees paid for the cloud are all about software and storage cost isn’t an issue other than for someone who wants to store massive amounts of data. One might think the companies in the cloud business are selling offsite storage, but their real revenue comes from selling value-added software that helps you operate your business and manage your data. The storage costs are almost an afterthought these days.

The race to zero is not even close to over. In one of my blogs last week I talked about how using magnetized graphene might increase the storage capacity of devices by a million-fold. That upgrade is still in the labs, but it demonstrates that progress to ever-cheaper storage is far from done. We’ve come a long way from the 720 kb that I used to squeeze onto a floppy diskl!