Should You Have a Data Cap?

data recovery

data recovery (Photo credit: Sean MacEntee)

Over the last few years most of the cable companies and some telcos have implemented data caps on high-speed Internet access. They always claimed that caps were necessary to help protect their networks from congestion. They claimed that heavy users would clog the networks and make data speeds slow for everybody else. But as someone who sees hundreds of networks, this claim holds no technical validity, except in some isolated instances and in some parts of some networks.

Michael Powell, the head of the National Cable and Telecommunications Association admitted publicly last week that caps are not about congestion, but rather are about ‘pricing fairness”. In the telecom world there is a general rule of thumb that the most active 15% of your users will use 85% of any resource, be that minutes, data, etc. And it’s a pretty good rule of thumb. If cable companies had come along and lowered prices for the 85% who are not heavy users and then made up the difference on higher rates for the 15%, then his argument would resonate with the public. But nobody saw any rate reductions and it’s hard to see data caps as anything more than a way to make even more money from data service.

One has to just note that the US has some of the highest-priced Internet services in the world to poke holes in NCTA’s announcement. If you compare US rates to the Far East or Europe it is easy to see that our rates are way out of line on a cost per megabit of service available to customers.

To make it worse, cable companies are starting to raise data rates. And this follows a ten-year period where the underlying cost of raw data has gotten cheaper every year. When a cable company set a monthly rate of $40 or $50 a decade ago, during that decade the cost of buying wholesale access to the Internet has probably dropped by 90%. It’s my opinion that cable companies know that in another decade that they are going to mostly become ISPs since cable and telephone are both dying products. They are starting to creep the rates up now to hedge against the day when that is their only product.

But even assuming that our rates are too high and that profits are really high, should any ISP consider any sort of cap or limitation on how customers use their data. I think the answer is yes, and it is not for any of the reasons that the cable companies have given.

Using my metric, 15% of the users on a network create most of the data usage. But absent any rules on how the network can be used, a small number of them could be using most of the usage for that group. For example, customers who operate servers and operate ecommerce site or other very busy sites like a pornography server can use huge amounts of data on the network. Much of that data is sent in the upload direction and doesn’t cost as much for a carrier as downloaded data, but a few such sites actually can clog a part of the network if they are busy continuously. The way around this problem is a prohibition against using servers on a basic residential data product. But if you are going to have this kind of policy you also need to have some way to measure how much data each customer is using.

On the download side of the equation, there are always a few customers who abuse any system. There are internet hoarders just as there are hoarders of anything else, and so you might want to set a cap that discourages continuous downloading. Comcast has implemented data caps of around 300 Gb in a lot of markets lately. If a customer downloads movies at a very high quality rate, they can use around 2 Gb per hour. If they watch non-HD movies it’s about half of that. And so a 300 Gb data cap would limit people to watching 150 hours of HD programming or 300 hours of normal programming per month. That works out to a limit of 5 hours per day or HD programming or 10 hours per day of normal programming. That may seem like a lot, but if each person in the family is watching their own programming, that is a really small limit.

I have advised my clients to institute a fairer cap, but to still have one. For instance, a cap set at 1 Tb (1,000 Gb) allows for over three times the usage than the Comcast cap. Anybody going over a 1 Tb cap is likely a data hoarder because that requires somebody to be downloading video more or less continuously every day of the week. Every network has a different configuration and so this is not a hard and fast limit. But I suggest some limit on data, at a very high rate that will only affect a truly small handful of people. The Comcast rate is set to make any family who actually uses their bandwidth to pay more. My suggestion is to set a cap that stops bad abuse, while giving people what they have paid for.

Data Mining – It’s Not What Customers Think

I know that when the public hears that their ISP is engaging in data mining that they assume this means that the ISP is reading their emails and monitoring their website viewing. And ISPs do have the ability to do those things although I don’t know any who spy on their customers in that way.

I can certainly understand why data mining scares the average consumer. Supermarkets get you to sign up for their loyalty programs so that they know everything you buy from them. And I know I get a spooky feeling when I express an interest about some product in one place on the Internet and then see ads for that product pop up on Facebook or my Google search.

But data mining is a valuable tool and every ISP should be using it – just not in the same way that the supermarkets and Facebook do it. In fact, we probably need to come up with a better terminology for doing the things I am suggesting below.

There are a number of tools around that let you look at data about customer usage and these tools allow an ISP to do the following:

  • Spambots. There is a wide array of spambots and other malware on the web that can infect customers’ computers. The worst of these, from a network perspective are spambots, which take over your customer’s computers and use it to send out spam. Most ISPs monitor email usage from their own domain and can spot when one of their users has been taken over by a spambot. But most customers these days do not use the email names and domains assigned by their ISP. Instead they web email addresses such as gmail or even the older AOL. And some spambots create new email addresses that the customer doesn’t even know about. And so data mining can be used to look for customers with unusual upload traffic. No customer is going to be offended if you ask them if they are uploading traffic 24 hours per day if in the process you help eliminate Trojan horses and spambots from their computer.
118 - Another File Sharing Session

118 – Another File Sharing Session (Photo credit: erickespinosa)

  • Web servers. Most ISPs do not want a customer to be using a residential ISP account to run a commercial web server. A web server is a device that is being used to run a website or service that drives a large amount of download traffic. Such a website might be used for e-commerce for example. But far too often web servers are used to run porn sites. ISPs are not against web servers, but they do expect people who operate them to buy the proper business level service. A web server can be full 24-hours per day, and that is generally not the level of service that is intended for a shared residential product. Data mining can be used to identify web servers and the customer can be directed to a more appropriate (and appropriately priced) service.
  • Data Caps. Most ISPs have set some cap on the amount of usage that a customer can download in a month. And these caps do not have to be small. I have one client that has a 2 terabyte cap each month for residential downloads. But there is no sense in having a data cap if you can’t actually measure how much bandwidth each customer is using. Data mining tools are the way to measure customers’ usage.
  • File sharing. Most ISPs have terms of service that prohibit customers from sharing copyrighted materials with others. But realistically an ISP is not going to know what customers are sharing with each other unless you get a complaint from a copyright holder. But many ISPs still like to get a handle on file-sharing because such traffic can eat up a lot of system bandwidth. Data mining can help you identify customers who are probably involved in one of the common file sharing programs.  An awful lot of file sharing is done by teenagers. I have clients who send out friendly reminders to customers who they think are file sharing that say something like: “We notice by your internet usage that you are probably running a file sharing program. We would just like to remind you that it is illegal to share copyrighted material and that there have been cases where copyright owners have gotten significant settlements by suing people who were sharing their property.” Such notices cut down on a lot of file sharing traffic as parent pressure kids into doing the right thing.

So you should be data mining. But perhaps the things I have described could all better be classified as network management, a term that would not dismay your customers.