Metering Broadband

A lot of the controversy about Comcast data caps disappeared last year when they raised the monthly threshold for data caps from 300 gigabytes to 1 terabyte. But lately I’ve been seeing folks complaining about being charged for exceeding the 1 TB data cap – so Comcast is still enforcing their data caps rules.

In order to enforce a data cap an ISP has to somehow meter the usage. It appears that in a lot of cases ISPs do a lousy job of measuring usage. Not all ISPs have data caps. The biggest ISPs that have them include Comcast, AT&T, CenturyLink for DSL, Cox and Mediacom. But even these ISPs don’t enforce data caps everywhere, like Comcast not enforcing them where they compete directly against Verizon FiOS.

Many customer home routers can measure usage and there are reports of cases where Comcast data usage measurements are massively different than what is being seen at the home. For example, there are customers who have seen big spikes in data measurement from Comcast at a time when their routers were disconnected or when power was out to the home. There are many customers who claim the Comcast readings always greatly exceed what they are seeing at their home routers.

Data caps matter because customer that exceed the caps get charged a fee. Comcast charges $10 for each 50 GB of monthly over the cap. Mediacom has the same fees, but with much smaller data caps such as a 150 GB monthly cap on customers with a 60 Mbps product.

It’s not hard to imagine homes now exceeding the Comcast data cap limit. Before I left Comcast a year ago they said that my family of three was using 600 – 700 GB per month. Since I didn’t measure my own usage I have no idea if their numbers were inflated. If my measurements were accurate it’s not hard to imagine somebody with several kids at home exceeding the 1 TB. The ISPs claim that only a small percentage of customers hit the data cap limits – but in world where data usage keep growing exponentially each year there are more homes that will hit the limit as time goes by.

What I find interesting is that there is zero regulation of the ISP data ‘meters’. Every other kind of meter that is used as a way to bill customers are regulated. Utilities selling water, electric or natural gas must use meters that are certified to be accurate. Meters on gas pumps are checked regularly for accuracy.

But there is nobody monitoring the ISPs and the way they are measuring data usage. The FCC effectively washed their hands from regulating ISPs for anything broadband when they killed Title II regulation of broadband. Theoretically the Federal Trade Commission could tackle the issue, but they are not required to do so. They regulate interactions with customers in all industries and can select the cases they want to pursue.

There are a few obvious reasons why the readings from an ISP would differ from a home, even under ideal conditions. ISPs measure usage at their network hub while a customer measurement happens at the home. There are always packets lost in the network due to interference or noise on the network, particularly with older copper and coaxial networks. The ISP would be counting all data passing through the hub as usage although many of the packets never make it to customers. But when you read some of the horror stories where homes that don’t watch video see daily readings from Comcast of over 100 GB in usage you know that there is something wrong in the way that Comcast is measuring usage. It has to be a daunting task to measure the usage directed for thousands of users simultaneously and obviously Comcast has problems in their measurement algorithms.

I’ve written about data caps before. It’s obvious that the caps are just a way for ISPs to charge more money, and it’s a gigantic amount of extra revenue if Comcast can bill $10 per month extra to only a few percent of their 23 million customers. Anybody that understand the math behind the cost of broadband understands that a $10 extra charge for 50 GB of usage is almost 100% profit. It doesn’t cost the ISP anything close to $10 for the connections for the first terabyte let alone an incrementally small additional amount. And there certainly is no cost at all if the Comcast meters are billing for phantom usage.

I don’t know that there is any fix for this. However, it’s clear that every customer being charged for exceeding data caps will switch to a new ISP at the first opportunity. The big ISPs wonder why many of their customers loathe them, and this is just one more way for a big ISP to antagonize their customers. It’s why every ISP that builds a fiber network to compete against a big cable companies understand that they will almost automatically get 30% of the market due to customers who have come to hate their cable ISP.

2 thoughts on “Metering Broadband

  1. Until an ISP can measure with close to 100% accuracy (and this is noooowhere close), they should not be billing or harassing any customers for their supposed usage – AND they should refund anyone previously charged extra, plus some sort of additional compensation for the trouble. This is ridiculous. If only we actually had effective regulators.

  2. Doug- I think you should have mentioned that the large MSO’s counting bytes have their “meters” certified by a third-party called NetForecast. I’ve attached a link to Comcast’s report that I found below.

    As you probably know the CMTS (device which controls cable modems) creates a standardized record of data downloads and uploads in a format called IPDR. This is very similar to records generated by a phone switch for long-distance calls. IPDR records activity by modem MAC and generates a record at a specified interval – likely once an hour or quarter-hour. This records are pulled out of the CMTS and stored in big data – type database.

    If you do the math – one record every quarter-hour for 30 days x millions customers you can see this is truly a big data effort to store and report the information. Very specialized databases are required.

    I have worked directly with several escalated customers who claimed inaccurate counts. In all cases, we were able to identify the reason for the discrepancy. Often the customer states “I don’t watch a lot of movies and there is nothing that could have generated 100 gig of data”.

    Here are the questions I generally ask:
    – How are you counting the your data?
    – Did you recently buy or upgrade any devices– including mobile devices
    – Do you use backup software (uploading data counts, too)
    – Do you have children in the home?
    – Is your Wi-Fi secure? How so?
    – Did you change your modem recently?

    Surprisingly – some customers count their data with PC software — which happens to ignore other devices in the home. New devices often download patches and update. Backup software (such as on iPhone) run every night). More sophisticated backup software may copy full disks images.

    The question about children in the home is usually very fruitful. Kids have gaming devices that download files which are very large.

    Early on, a modem swap-out mid-month by the MSO might cause an issue if the data from a prior user was somehow carried over. Obviously this was fixed long ago, but I always check just in case.

    I’ve found that only with a highly sophisticated home router, can a consumer measure their bandwidth close to accurately. Perhaps it’s changing, but my experience showed that a router needed special WRT software, etc.

    Also don’t forget there is a small amount of modem-to-CMTS overhead traffic that can’t be captured by a home router. In some cases, the MSO may choose to ignore counting certain types of traffic. IP phone traffic is an example, when it happens. The MSO could also choose to not count other traffic streams such as certain IP Video programs. A good MSO will figure

    Thus, a home measuring device will never be exactly the same as what the MSO reports.

    While it is indeed possible to have a “horror” story, I think the chances are quite rare because pulling off the big data effort doesn’t leave a lot to chance. I think suggesting the MSO’s would knowingly allow “phantom usage” is just plain wrong and specious.

    Also remember, If the plant has such poor performance that packet loss is “costing” a material amount of data bytes, the customer will be surfing slower and using the Internet less. Netflix will run in a degraded mode –etc — the end result is less data use.

    If, as you suggest, Comcast or other big MSO’s are so greedy, they should run pristine plant so their customers will watch 4K streams, get the fastest Netflix resolution, surf more, and run the data cash register up.

    Billing for data is a short term economic necessity given the runaway usage growth that threatens to jam up neighborhoods. As company’s deploy fiber deeper, I think usage billing will morph into something that isn’t so controversial.

    Here is the Netforecast info for Comcast. Anyone curious should research how the certification process works and ask for a report from their ISP.

Leave a Reply