Netflix just came out with a new speed test at fast.com which is intended to measure the download speed of Internet connections to determine if they are good enough to stream Netflix. The test only measures the speeds between a user and the Netflix servers. This is different than most other speed tests on the web that also look at upload speeds and latency.

This raises the question of how good speed tests are in general. How accurate are they and what do they really tell a user? There are a number of different speed tests to be found on the web. Over the years I have used the ones at speedtest.net (Ookla), dslreports.com, speed.io, the BandWidthPlace and TestMySpeed.

Probably the first thing to understand about speed tests is that they are only testing the speed of a ping between the user and the test site routers and are not necessarily indicative of the speeds for other web activities like downloading files, making a VoIP phone call or streaming Netflix. Each of those activities involves a different type of streaming and the speed test might not accurately report what a user most wants to know.

Every speed test uses a different algorithm to measure speed. For example, the algorithm for speedtest.net operated by Ookla discards the fastest 10% and the slowest 30% of the results obtained. In doing so they might be masking exactly what drove someone to take the speed test, such as not being able to hold a connection to a VoIP call. Ookla also multithreads, meaning that they open multiple paths between a user and the test site and then average the results together. This could easily mask congestion problems a user might be having with the local network.

Another big problem with any speed test is that it measures the connection between a customer device and the speed test site. This means that the customer parts of the network like the home WiFi network are included in the results. A lot of ISPs I know now claim that poor in-home WiFi accounts for the majority of the speed issue problems reported by customers. So a slow speed test doesn’t always mean that the ISP has a slow connection.

The speed of an Internet connection for any prolonged task changes from second to second. Some of the speed tests like Netflix Ookla show these fluctuations during the test. There are numerous issues for changing speeds largely having to do with network congestion at various points in the network. If one of your neighbors makes a big download demand during your speed test you are likely to see a dip in bandwidth. And this same network contention can happen at any one of numerous different parts of the network.

The bottom line is that speed tests are not much more than an indicator of how your network is performing. If you test your speed regularly then a slow speed test result can be an indicator that something is wrong. But if you only check it once in a while, then any one speed test only tells you about the minute that you took the test and not a whole lot more. It’s not yet time to call your ISP after a single speed test.

There have been rumors around the industry that the big ISPs fudge on the common speed tests. It would be relatively easy for them to do this by giving priority routing to anybody using one of the speed test web sites. I have no idea if they do this, but it would help to explain those times when a speed test tells me I have a fast connection and low latency and yet can’t seem to get things to work.

I think the whole purpose of the Netflix speed test is to put pressure on ISPs that can’t deliver a Netflix-capable connection. I don’t know how much good that will do because such connections are likely going to be on old DSL and other technologies where the ISP already knows the speeds are slow.