Measuring Mobile Broadband Speeds

I was using Google search on my cellphone a few days ago and I thought my connect time was sluggish. That prompted me to take a look at the download speeds on cellular networks, something I haven’t checked in a while.

There are two different companies that track and report on mobile data speeds, and the two companies report significantly different results. First is Ookla, which offers a speed test for all kinds of web connections. Their latest US speed test results represent cellphone users who took their speed test in the first half of this year. Ookla reports that US cellular download speeds have increased 19% over the last year and are now at an average of 22.69 Mbps. They report that the average upload speeds are 8.51 Mbps, an improvement of 4% over the last year. Ookla also found that rural mobile broadband speeds are 20.9% slower at urban speeds and are at an average of 17.93 Mbps.

The other company tracking mobile broadband speeds reports a different result. Akamai reports that the average cellular download speed for the whole US was 10.7 Mbps for the first quarter of 2017, less than half of the result shown by Ookla.

This is the kind of difference that can have you scratching your head. But the difference is significant since cellular companies widely brag about the higher Ookla numbers, and these are the numbers that end up being shown to regulators and policy makers.

So what are the differences between the two numbers? The Ookla numbers are the results of cellphone users who voluntarily take their speed test. The latest published numbers represent tests from 3 million cellular devices (smartphones and tablets) worldwide. The Akamai results are calculated in a totally different way. Akamai has monitoring equipment at a big percentage of the world’s internet POPs and they measure the actual achieved speeds of all web traffic that comes through these POPs. They measure the broadband being used on all of the actual connections they can see (which in the US is most of them).

So why would these results be so different and what are the actual mobile broadband speeds in the US? The Ookla results are from speed tests, which last less than a minute. So Ookla speed test measures the potential speed that a user could theoretically achieve on the web. It’s a test of the full bandwidth capability of the connection. But this is not necessarily the actual results for cellphone users for a few reasons:

  • Cellphone providers and many other ISPs often provide a burst of speeds for the first minute or two of a broadband connection. Since the vast majority of web events are short-term events this provides users with greater speeds than would be achieved if they measured the speed over a longer time interval. Even with a speed test you often can notice the speed tailing off by the end of the test – this is the ‘burst’ slowing down.
  • Many web experts have suspected that the big ISPs provide priority routing for somebody taking a speed test. This would not be hard to do since there are only a few commonly used speed test sites. If priority routing is real, then speed test results are cooked to be higher than would be achieved when connecting to other web sites.

The Akamai numbers also can’t be used without some interpretation. They are measuring achieved speeds, which means the actual connection speeds for mobile web connections. If somebody is watching a video on their cellphone, then Akamai would be measuring the speed of that connection, which is not the same as measuring the full potential speed for that same cellphone.

The two companies are measuring something totally different and the results are not comparable. But the good news is that both companies have been tracking the same things for years and so they both can see the changes in broadband speeds. They also both measure speeds around the world and are able to compare US speeds with others. But even that makes for an interesting comparison. Ookla says that US mobile speed test results are 44th in a world ranking. That implies that the mobile networks in other countries make faster connections. Akamai didn’t rank the countries, but the US is pretty far down the list. A lot of countries in Europe and Asia have faster actual connection speeds than the US, and even a few countries in Africa like Kenya and Egypt are faster than here. My conclusion from all of this is that ‘actual’ speeds are somewhere between the two numbers. But I doubt we’ll ever know. The Akamai numbers, though, represent what all cell users in aggregate are actually using, and perhaps that’s the best number.

But back to my own cellphone, which is what prompted me to investigate this. Using the Ookla speed test I showed a 13 Mbps download and 5 Mbps upload speed. There was also a troublesome 147 ms of latency, which is probably what is accounting for my slow web experience. But I also learned how subjective these speeds are. I walked around the neighborhood and got different results as I changed distances from cell towers. This was a reminder that cellular data speeds are locally specific and that the distance you are from a cell site is perhaps the most important factor in determining your speed. And that means that it’s impossible to have a meaningful talk about mobile data speeds since they vary widely within the serving area of every cell site in the world.

What’s the Real Speed of Bandwidth in the US?

Polk County SignRecently there were two very different reports published about the speed of broadband in the US. Comparing these two figures tells us some interesting things about the politics of broadband.

The first report was the most recent quarterly report from Akamai. They rated the US as 16th globally with average landline broadband download speeds at 12.6 Mbps. This number is up 10% from the previous year’s average speed of 11.5 Mbps. Akamai derives their speeds from looking at millions of actual downloads around the world.

At about the same time the FCC released its latest report on broadband and they said that the average download speed in the US for September 2014 is now 31 Mbps, almost triple the 10 Mbps reported in March of 2011. That’s a huge disparity with the FCC’s numbers at almost 2.5 times higher than what was measured by Akamai. How can the numbers be that far apart?

Akamai is measuring the actual download speed of millions of transactions on the Internet. This is how fast things are actually downloading and, because of the huge number of transactions that they look at, the number should be very accurate.

The FCC on the other hand is using a very odd measuring technique. First, they rely on speeds that come from 5,000 homes in the US that have volunteered to have their speeds measured. It’s pretty obvious in looking at the FCC results that these homes are not a random sample. I would suspect they aren’t measuring too many homes with dial-up, for example. If their sample is not random, then the results of the FCC tests cannot be multiplied out to accurately represent the whole country. That is one of the basic precepts of statistical sampling.

Another reason for the difference is that the FCC is basically looking at speed tests and not at actual downloads. Speed tests measure the highest speed that can be achieved to a specific test site created for that purpose. The FCC further says that their sample shows customers’ actual speeds are not much different than subscribed speeds. But my recent experience is that this disparity still widely exists. For example, in helping a western city look at broadband issues, we recently asked citizens there to take the Ookla speed test and report to us the results of the speed tests along with the speed they are paying for. The results were eye-opening with about 80% of customers seeing speed test results much slower than what they were subscribing to. In some cases the results were drastically different with households that subscribe to 100 Mbps services seeing actual speeds under 20 Mbps.

Another difference between the two sets of numbers is the little-understood fact that you can’t download anything faster than what the Internet is sending you. Somebody with a gigabit fiber connection from Google is going to see the same download speed from watching a Netflix HD movie as somebody on a 20 Mbps cable modem connection. Netflix doesn’t care what connection you have and sends the same stream to everybody, as does a lot of the Internet. Akamai is measuring this real speed and not the theoretical speed from speed tests.

The problem with the FCC’s method from a mathematical perspective the FCC’s method will skew the average towards the faster speeds in the sample. In a very simple example, if you sampled one gigabit customer and nine dial-up customers you would say that the average data speed was a little over 100 Mbps. While that would be true for those ten people, that is obviously not a number that has any practical relevance.

What is troubling is that the FCC makes policy decisions based upon the results of these tests. For instance, they routinely publish the percentage of homes that now meet their new 25 Mbps download standard to be considered as broadband. On the surface there is nothing wrong with what the FCC is doing, and for those 5,000 customers the reported speeds really is the average for that group. The problem is that those results show a different Internet than is really out there. For example, the FCC just also declared that 39% of the homes in the country can’t get real broadband at all. If the FCC is going to report an average speed for the whole country, those homes ought to be represented somehow in the the average.

There might be help on the way since the FCC is finally starting to require large ISPs to tell their customers the truth about their networks, including actual download speeds. There are new reporting formats circulating now at the FCC that the ISPs are going to have to provide to every customer.

There is one promising take away from the FCC numbers which is that the large ISPs are increasing customer speeds. 2016 promises to be a year when many households will see a boost in speeds. I know Comcast just recently upgraded my speed, with no announcement, and my 50 Mbps cable modem now seems to regularly be getting 88 Mbps download. Almost every big cable companies has announced initiatives to increase customer speeds this year, across the board.

But this difference in speed reports does remind us that you always have to dig a little deeper when you see a statistic like average download speeds. There are different ways of measuring speeds and the FCC way of measuring speeds is suspect in several ways. What they published is not an untrue number for the way they measured it – but the number is just not relevant when talking about the US broadband experience.