Tag: packet loss

Categories
Technology

Packet Loss and Broadband Performance

In a recent article in FierceWireless, Joe Madden wrote an article looking at the various wireless technologies he has used at his home in rural central California. Over time he subscribed to a fixed wireless network using WiFi spectrum, cellular LTE broadband, Starlink, and a fixed wireless provider using CBRS spectrum. A lot of rural folks can describe a similar path where they have tried all of the broadband technologies available to them.

Since Joe is a wireless expert who works at Mobile Experts, he was able to analyze his broadband performance in ways that are not easily understood by the average subscriber. Joe came to an interesting conclusion – the difference in performance between various broadband technologies has less to do with speed than with the consistency of the broadband signal.

The average speed tests on the various products varied from 10/2 Mbps on fixed wireless using WiFi, to 117/13 Mbps on Starlink. But what Joe found was that there was a huge difference in consistency as measured by packet loss. Fixed wireless on WiFi had packet loss of 8.5%, while the packet loss on fixed wireless using CBRS spectrum dropped to 0.1%. The difference is stark and is due to the interference that affects using unlicensed spectrum compared to a cleaner signal on licensed spectrum.

But just measuring packet loss is not enough to describe the difference in the performance of the various broadband connections. Joe looked at the number of lost packets that were delivered over 250 milliseconds. That will require some explanation. Packet loss in general describes the percentage of data packets that are not delivered on time. In an Internet transmission, some packets are always lost somewhere in the routing to customers – although most packets are lost due to the local technology at the user end.

When a packet doesn’t show up as expected, the Internet routing protocols ask for that packet to be sent again. If the second packet gets to the user quickly enough, it’s the same, from a user perspective, as if that packet was delivered on time. Joe says that re-sent packets that don’t arrive until after 250 milliseconds are worthless because by then, the signal has been delivered to the user. The easiest way to visualize this is to look at the performance of Zoom calls for folks using rural technologies. Packets that don’t make it on time result in a gap in the video signal that manifests as fuzziness and unclear resolution on the video picture.

Packet loss is the primary culprit for poor Zoom calls. Not receiving all of the video packets on time is why somebody on a Zoom call looks fuzzy or pixelated. If the packet loss is high enough, the user is booted from the Zoom call.

The difference in the percentage of packets that are delivered late between the different technologies is eye-opening. In the fixed wireless using WiFi spectrum an astounding 65% of re-sent packets took longer than 250 ms. Cellular LTE broadband was almost as bad at 57%. Starlink was better at 14%, while fixed wireless using CBRS was lowest at 5%.

Joe is careful to point out that these figures only represent his home and not the technologies as deployed everywhere. But with that said, there are easily explainable technology reasons for the different levels of packet delay. General interference plays havoc with broadband networks using unlicensed spectrum. Starlink has delay just from the extra time for broadband signals to go to and from the satellite and the ground in both directions. The low packet losses on a CBRS network might be due to having very few other neighbors using the new service.

Joe’s comparison doesn’t include other major broadband technologies. I’ve seen some cable networks with high packet loss due to years of accumulated repairs and unresolved issues in the network. The winner of the packet loss comparison is fiber, which typically has an incredibly low packet loss and also a quick recovery rate for lost packets.

The bottom line from the article is that speed isn’t everything. It’s just one of the characteristics that define a good broadband connection, but we’ve unfortunately locked onto speed as the only important characteristic.

Categories
The Industry

The Reality of Rural Broadband

I recently saw the results of several rural surveys that probably tell the best story about the state of rural broadband. The two areas being studied are far apart geographically, but they are similar in many ways. The areas are both rural and are not near to a metropolitan area. The areas have some modest manufacturing and some modest amount of tourism, but neither in a big way. Both areas included some small towns, and a few of these towns have cable TV. And in both places, the customers in the rural area have poor broadband choices. These are not small isolated pockets of people, and the two surveys cover nearly 20,000 homes.

If you listen to FCC rhetoric it’s easy to think that rural broadband is improving – but in areas like these you can’t see it. These areas have both were supposed to get some upgrades from CAF II – but from what the locals tell me there have been zero improvements so far. The CAF program still has a few years to go, so perhaps there will be some modest improvement in rural DSL.

For now, the broadband situation in these areas is miserable. There are homes with DSL with speeds of a few Mbps at best, with some of the worst speeds hovering at dial-up speeds. One respondent to a survey reported that it took 8 hours to download a copy of Microsoft Office online.

The other broadband choices are also meager. Some people use satellite broadband but complain about the latency and about the small data caps. These areas both have a smattering of fixed wireless broadband – but this is not the modern fixed wireless you see today in the open plains states that delivers 25 Mbps or faster broadband. Both of the areas in the surveys are heavily wooded with hilly terrain, and fixed wireless customers report seeing speeds of 1-2 Mbps. There are a number of homes using their cell phones in lieu of home broadband – an expensive alternative if there are school kids or if any video is watched. There were customers who reported using public hotspots in nearby small towns. And there were a number of households, included many with school kids who have given up and who have no broadband – because nothing they’ve tried has worked.

As would be expected in rural areas, slow speeds are not the only problem. Even homes that report data speeds that should support streaming video complain that streaming doesn’t work. This indicates networks with problems and it’s likely the networks have high latency, are full of jitter, or are over-subscribed and have a lot of packet loss. People don’t really judge the quality of their broadband connection by the speed they get on a speed test, but instead by the ability to do normally expected activities on the Internet.

Many of these homes can’t do things that the rest of us take for granted. Many report the inability to stream video – even a single stream. This is perhaps the biggest fallacy in the way the FCC measures broadband, because they expect that a house getting a speed like 5 Mbps ought to be able to do most needed tasks. In real life the quality of many rural connections are so poor that they won’t stream video. Many people in these areas also complained that their Internet often froze and they had to constantly reboot – something that can kill large downloads or kill online sessions for school or work.

One of the biggest complaints in these areas was that their network only supported one device at a time, meaning that members of the family have to take turns using the Internet. I picture a family with a few school kids and can see how miserable that must be.

The surveys produced a long list of other ways that poor broadband was hurting households. Number one was the inability of people to work at home. Many people said they could work at home more often if they had broadband. A few respondents want to start home businesses but are unable to because of the poor broadband. Another common complaint was the inability for kids to do schoolwork, or for adults to pursue college degrees on line.

The problems many people reported were even more fundamental than these issues. For instance, there were households saying that they could not maintain a good enough connection to bank online or pay their bills online. There were respondents who say they can’t shop online. Many households complained that they couldn’t offload cellular data at home to WiFi, driving up their cellular bills. A number of homes would like to cut the cord to save money but can’t stream Netflix as an alternative to cable.

When you look the raw data behind these kinds of surveys you quickly see the real issues with lack of broadband. In today’s society, not having home broadband literally takes a home out of the mainstream of society. It’s one thing to look at the national statistics and be told that the number of homes without broadband is shrinking. But it’s an entirely different story when you see what that means for the millions of homes that still don’t have adequate broadband. My guess is that some of the areas covered by these surveys show as underserved on the FCC maps – when in fact, their broadband is so poor that they are clearly unserved, ignored and forgotten.

Categories
Regulation - What is it Good For? The Industry

How Much Speed Do We Really Need?

There is a lot of buzz floating around in the industry that the FCC might lower the official definition of broadband from 25 Mbps down and 3 Mbps up. Two of the current FCC commissioners including the chairman opposed setting that definition a few years back. Lowering the speeds would let the FCC off the hook for the requirement by law to make sure that the whole country can get broadband. If they lower the definition, then voila, millions more Americans would be declared to have adequate broadband.

So today I thought I’d take a look at the download speeds we really need at our homes. You may recall that back when the FCC set the 25/3 Mbps definition that they made a list of the broadband speed needed to do typical activities. And in doing so they tried to create profiles of some typical American households. That attempt was awkward, but it was a good starting point for examining household bandwidth needs. I’m updating their list a bit for things that people do today, which is already different than just a few years ago. Consider the following web activities:

  • Web Background 5 Mbps
  • Web Browsing 1 – 2 Mbps
  • Online Class 1 – 2 Mbps
  • Social Media 1 – 2 Mbps
  • Streaming Music 3 Mbps
  • Voice over IP 2 Mbps
  • SD Video stream 1 – 3 Mbps
  • HD Video Stream 4 – 6 Mbps
  • 4K Video Stream 15 – 20 Mbps
  • Gaming 1 – 3 Mbps
  • Skype / Video Conference 1 – 3 Mbps
  • Big File Downloader 50 Mbps

People don’t agree with all of these listed speeds because there are no standards for how the web works. For example, by using different compression schemes a video stream from Netflix is not identical to one from Amazon. And even from one source there is variation since an action move takes more bandwidth than something like a stand-up comedy routine.

It’s important to remember that broadband demand can come from any device in your house – desktop, laptop, smartphone, tablet, etc. It’s also important to note that these are speed requirements for a single user. If two people in the house are watching an separate video, then you have to double the above number.

What the FCC failed to consider back when they set the speed definition is that households need enough bandwidth to handle the busiest times of the day. What matters is the number of simultaneous activities a home can do at the same time on the web, with most families being busiest in the evenings. There might be somebody on social media, somebody watching an HD movie, while somebody else is doing homework while also using a smartphone to swap pictures.

There is another issue to consider when trying to do simultaneous tasks on the Internet – packet loss. The connection between the ISP and a customer gets more congested when it’s trying to process multiple data streams at the same time. Engineers describe this as packet collision – which sounds like some kind of bumper-car ride – but it’s an apt way to describe the phenomenon. Most home routers are not sophisticated enough to simultaneously handle too many multiple streams at once. Packets get misdirected or lost and the router requests the missing packets to be sent again from the originator. The busier the router, the more packet interference. This is also sometimes called ‘overhead’ in the industry and this overhead can easily grow to 15% or more of the total traffic on a busy connection, meaning it takes 15% more bandwidth to complete a task than if that task was the only thing occurring on the broadband connection.

There is another kind of interference that happens in homes that have a WiFi network. This is a different kind of interference that has to do with the way that WiFi works. When a WiFi network gets multiple requests for service, meaning that many devices in the home are asking for packets, the WiFi router gets overwhelmed easily and shuts down. It then reinitiates and sends packets to the first device that gets its attention. In a busy network environment the WiFi router will shut down and restart constantly as it tries to satisfy the many needed devices. This kind of interference was designed into the WiFi specification as a way to ensure that WiFi could satisfy the needs of multiple devices. This WiFi overhead can also easily add 15% or more to the network demand.

Anybody who lives in a home with active users understands how networks can get overwhelmed. How many of you have been frustrated trying to watch a movie when others in the house are using the Internet? Even big bandwidth can be overwhelmed. I have a friend who has a 100 Mbps fiber connection on Verizon FiOS. He went to watch a video and it wouldn’t stream. He found that his two teenage sons were each using half a dozen gaming streams at the same time and had basically exhausted his fast bandwidth pipe.

The FCC can tinker with the official definition of broadband since that is their prerogative. But what they can’t do is to define for any given home how much bandwidth they really need. The funny thing is that the big ISPs all understand this issue. The cable companies have unilaterally increased speeds across-the-board to urban customers several times in recent years and in most markets offer speeds considerably faster than the current FCC definition of broadband. These ISPs know that if they were only delivering 25 Mbps that they would be overwhelmed with customers complaining about the connection. Those complaints are the real proof of how much bandwidth many homes need. If the FCC lowers the definition of broadband then they have on blinders and are ignoring how homes really use broadband today. If they lower the speed definition it’s hard to see it as anything other than a political move.

Categories
Technology

Our Degrading Networks

Lately I’ve been hearing a lot of stories about rural broadband with a common theme. People say that their broadband has been okay for years and is now suddenly terrible. This seems to be happening more on DSL networks than with other technologies, but you hear this about rural cable networks as well.

There are several issues which contribute to the problem – more customers sharing a local network, increasing data usage for the average customer, and a data backbone feeding the neighborhood that is has grown too small for the current usage.

Broadband adoption rates have continued to grow as more and more households find it mandatory to use broadband. And so neighborhoods that once had 50% of homes using a local network will have grown to more than 70%. That alone can stress a local network.

Household broadband usage has also been increasing. A lot of the new usage is streaming video. This video doesn’t just come from Netflix but there is now video all over the web and social media. It’s hard to go to the web today and not encounter video. As more and more customers are using video at the same time they can quickly be asking for more aggregate data in a network than the network can supply. Where the demand has outstripped network capability there is a remedy available for most situations and increasing the size of the bandwidth pipe feeding a neighborhood will typically fix the problem.

Let’s look at an example. Consider a neighborhood that has 100 DSL customers and that is fed by a DS3 (45 Mbps). In the days before a lot of streaming video such a neighborhood probably felt like it had good broadband. The odds against more than a few customers trying to download something really large at exactly the same time meant that there was almost always enough bandwidth for everybody.

But today people want to watch streaming video. Netflix recommends that there be at least a 1.5 Mbps continuous stream available to watch a video. So up to about 30 households in this theoretical neighborhood could watch Netflix at the same time. That math is not quite that linear as I will explain below, but you can see how the math works. The problem is that it’s not hard to imagine that with 100 homes that there would be demand for more than 30 video streams at the same time, particularly when considering that some households want to watch more than one Netflix stream at the same time.

The problems in this theoretical neighborhood are made worse by what is called packet loss. Packet loss occurs when a network tries to download multiple signals at the same time. When that happens some packets are accepted, but some are just lost. Our current web protocols correct this problem by sending out a message from the receiving router asking for the retransmission of missing packets, and they are sent again. As networks get busy the amount of contention and packet loss increases and the percentage of the packets that are sent multiple times increases. And so as networks get busy they grow increasingly less efficient. Where this theoretical neighborhood network can theoretically accommodate 30 Netflix streams, in real life it might actually only handle 20 due to the extra traffic caused by resending lost packets.

This theoretical network has grown over time from being efficient to now being totally inadequate. Customers who were once happy with speeds are now unable to watch Netflix on an average evening. The network will still function great at 4:00 AM when nobody is trying to use it, but during the times when people want use it, it will fail more often than not. The only way to fix this theoretical neighborhood is increase the backbone from 45 Mbps to something much larger. And that requires capital – and we all know that the large telcos are not putting capital into copper neighborhoods.

Cellular companies have been dealing with these growth issues for a number of years now. Cellular networks are seeing annual growth between 60% and 120% per year, meaning that any improvement in the network is quickly eaten up by increased demand. But t’s a much bigger issue to keep upgrading all landline networks. While there are just over 200,000 cell towers in the US there must be several million local broadband backbone connections into neighborhoods. These range from tiny backbones with a few T1s feeding a few homes up to networks with a few hundred people sharing a larger backbone. Upgrading that many networks backbone connection means a huge capital outlay is needed to maintain acceptable levels of service.

Unfortunately my theoretical neighborhood is not really all that theoretical. The big increase in landline broadband demand is now starting to max out the bandwidth utilization in many neighborhoods. The FCC says that there are 34 million people in the country that don’t have adequate broadband today. But with the rate that neighborhood networks are degrading, that number of households with inadequate broadband is growing rapidly – and not get smaller as the FCC is hoping.

Exit mobile version