Tag: busy hour

Categories
The Industry

The Busy Hour and Data Caps

As states are getting ready to create their broadband plans for the NTIA’s $42.5 billion BEAD grants, we’re starting to see some interesting arguments being made by incumbents to influence state broadband plans. One of the aspects of the BEAD plan that hasn’t been discussed much yet is that the NTIA is stressing affordability. For example, the NOFO states several times that states must develop a middle-class rate plan. Everybody I know is scratching their head on that what that means, but to the big ISPs, this must sound like rate caps – something they have vigorously fought everywhere.

One of my readers from Maine says that one aspect of high rates – data caps – is a big topic of discussion in the state. ISPs are making the claim that they’ve made many times that data caps are needed to manage the network. The rest of the ISP argument is that heavy broadband users are creating extra costs and should pay for the extra usage.

I’ve written about this before, and the big ISP argument is pure bosh. Broadband costs are not related to the overall volume of broadband being delivered on a network. The cost is determined almost entirely by what network engineers call the busy hour. The business hour each day is when a network sees the busiest broadband usage. For networks with a lot of residential customers, the busy hour is usually during the evening when the most families are streaming video. However, I’ve seen a few networks where the busy hour was at some different time of the day. One example I know of is a network serving college students that has more usage after 10:00 PM and into the early hours of the morning, which the ISP accredits to gaming. Networks that serve business customers might have the busy hour at almost any time of the day, depending upon the mix of businesses using the network.

The busy hour is important because it determines how much bandwidth is needed for the various components of the network. For example, the data connection between the ISP core and a neighborhood must be large enough to handle the busy hour – otherwise, bandwidth and speeds get slower. Another important network segment that depends upon the busy hour is the connection to and from the ISP and the Internet – there must be enough bandwidth available to accommodate the expected busiest time each month. Most ISPs buy enough bandwidth to the Internet to have at least a 20% cushion of capacity greater than the expected busiest hour.

As long as an ISP is buying enough broadband to the Internet to satisfy the busy hour, it doesn’t matter how busy the network is the rest of the time – because all other times are less busy than the peak time. The ISP is paying for the peak bandwidth and doesn’t get any savings in bandwidth cost for times when customers don’t use the bandwidth.

The data cap for most ISPs is set at or close to 1 terabyte per month (1,000 gigabytes) of total combined upload and download usage for a customer. According to the most recent statistics from OpenVault, 15% of all homes are now using more than a terabyte of bandwidth per month. I see network performance statistics from my clients, and none of them believe that the busy hour is caused by a few big users – it happens at the time of the day when the highest number of homes use broadband at the same time. There is a limit on how much bandwidth a given home can realistically use if they are doing the same things that many homes do – like watching video or gaming. A house that uses a terabyte of data per month is likely no busier than a home that uses half that. The house with the higher monthly usage normally just uses the broadband for more hours per day to accumulate the terabyte of usage.

The only time that data usage costs more money for an ISP is when it has to increase the size of the data connection to the Internet. But interestingly, even that doesn’t always cost an ISP more money because as network usage has grown, the cost per gigabit of traffic to reach the Internet has dropped year after year. I have ISPs that are carrying four times the traffic of only a few years ago but are still paying no more for the connection – which is another argument about heavy users creating more costs for an ISP.

Data caps do not make any significant change to lower the busy hour, and any ISP that says they do is being untruthful. Data caps serve only one purpose – to make additional money for ISPs. There is no other explanation, and any ISP that argues that data caps are a network control issue is counting on the technology naivety of regulators and policy people. When Comcast first introduced data caps, only a fraction of one percent of homes used more than a terabyte per month. Comcast made the same argument then that the data caps were to rein in the usage of the few heavy users. The argument was no more valid then than it is now. ISPs have to be thrilled to see so many homes now exceeding the data caps – the overage fees for exceeding the data caps can be as much as $50 per household. Seeing 15% of households exceeding the data caps is like Christmas to big ISPs, like a big secret rate increase.

Categories
The Industry

Did Broadband Deregulation Save the Internet?

Something has been bothering me for several months, and that usually manifests in a blog at some point. During the COVID-19 crisis, the FCC and big ISPs have repeatedly said that the only reason our networks weathered the increased traffic during the pandemic was due to the FCC’s repeal of net neutrality and deregulation of the broadband industry. Nothing could be further from the truth.

The big increase in broadband traffic was largely a non-event for big ISPs. Networks only get under real stress during the busiest times of the day. It’s during these busy hours when network performance collapses due to networks being overloaded. There was a big increase in overall Internet traffic during the pandemic, but the busy hour was barely affected. The busy hour for the Internet as a whole is mid-evenings when the greatest number of homes are watching video at the same time. Every carrier that discussed the impact of COVID-19 said that the web traffic during the evening busy-hour didn’t change during the pandemic. What changed was a lot more usage during the daytime as students took school classes from home and employees worked from home. Daytime traffic increased, but it never grew to be greater than the evening traffic. As surprising as that might seem to the average person, ISP networks were never in any danger of crashing – they just got busier than normal during the middle of the day, but not so busy as to threaten any crashes of the Internet. The big ISPs are crowing about weathering the storm when their networks were not in any serious peril.

It’s ironic to see the big ISPs taking a victory lap about their performance during the pandemic because the pandemic shined a light on ISP failures.

  • First, the pandemic reminded America that there are tens of millions of rural homes that don’t have good broadband. For years the ISPs argued that they didn’t invest in rural America because they were unwilling to invest in an overregulated environment. The big ISPs all promised they would increase investment and hire more workers if they were deregulated. That was an obvious lie, since the big ISPs like Comcast and AT&T have cut investments since the net neutrality appeal, and collectively the big ISPs have laid off nearly 100,000 workers since then. The fact is that the big ISPs haven’t invested in rural broadband in decades and even 100% deregulation is not enough incentive for them to do so. The big ISPs wrote off rural America many years ago, so any statements they make to the contrary are purely rhetoric and lobbying.
  • The pandemic also highlighted the stingy and inadequate upload speeds that most big ISPs offer. This is the broadband crisis that arose during the pandemic that the big ISPs aren’t talking about. Many urban homes that thought they had good broadband were surprised when they had trouble moving the office and school to their homes. The problem was not with download speeds, but with the upload speeds needed to connect to school and work servers and to talk all day on video chat platforms – activities that rely on a solid and reliable upload speed. Homes have reacted by migrating to fiber when it is available. The number of households that subscribe to gigabit broadband doubled from December 2019 to the end of March 2020.

The big ISPs and the FCC have also made big political hay during the crisis about the Keep America Connected Pledge where ISPs promised to not disconnect homes for non-payment during the pandemic. I’m pretty sure the ISPs will soon go silent on that topic because soon the other shoe is going to drop as the ISPs expect homes to catch up on those ‘excused’ missed payments if they want to keep their home broadband. It’s likely that millions of homes that ran out of money due to losing their jobs will soon be labeled as deadbeats by the ISPs and will not be let back onto the broadband networks until they pay their outstanding balance, including late fees and other charges.

The shame of the Keep America Connected Pledge was that it had to be voluntary because the FCC destroyed its ability to regulate ISPs in any way. The FCC has no tools left in the regulatory quiver to deal with the pandemic after it killed Title II regulation of broadband.

I find it irksome to watch an industry that completely won the regulatory battle keep acting like it is under siege. The big ISP lobbyists won completely and got the FCC to neuter itself, and yet the big ISPs miss no opportunity to keep making the same false claims they used to win the regulation fight.

It’s fairly obvious that the big ISPs are already positioning themselves to fight off the time when the regulatory pendulum swings the other way. History has shown us that monopoly overreach always leads to a reaction from the public that demands stronger regulation. It’s in the nature of all monopolies to fight against regulation – but you’d think the ISP industry could come up with something new rather than to repeat the same lame arguments they’ve been making for the last decade about how overregulation is killing them.

Categories
The Industry

Putting COVID-19 Traffic Growth into Perspective

Nokia Deepfield is another company that works in the background on the web and that analyzes data traffic patters for the big ISPs. Their June 4 report on web traffic reports about the same thing we’re hearing from most large ISPs – that the volume of web traffic suddenly shot up since the onset of the pandemic.

Nokia Deepfield says that the increase in traffic has settled in at about a 25% increase over pre-COVID levels. The most important aspect of the increase has been that almost all of the increases have been during the daytime, including on weekends. Networks have not seen any surge (or decrease) in the evening busy hour traffic.

To people who don’t follow the industry, those increases likely sound astronomical. Any other businesses would find a sudden 25% increase in business to be an extraordinary event. Imagine the impact of a sudden and sustained 25% increase in customer demand at a coffee shop, a bank, or a drug store. A business would have to scramble to increase inventory and staff to keep up with the new demand.

But in the world of ISPs this kind of growth is a lot less astounding. Cisco has been reporting for years that residential web traffic has been growing by 21% annually and business broadband by 24%. The ISP industry just absorbed in a single month the growth that would have normally been expected for all of 2020 – but any ISP worth their salt was already braced for this kind of growth this year.

It’s probably hard for the average person to digest that fact that the ISP industry has been coping with this kind of sustained growth for decades. If an ISP makes an expensive investment to double network capacity they’ll see the newly-created capacity filled within three or four years. ISP network engineers face a never-ending task of staying a step ahead of constant and relentless broadband growth.

It’s also worth noting that the growth due to COVID was less dramatic than the industry press might make you believe. Networks are engineered to satisfy the demands at the busy hour – those times of the day when networks are the busiest. During the rest of the day much of the network sits idle since the data pipes aren’t as full. The business hour for residential neighborhoods has been the evenings when homes watch video. Almost all of the growth from COVID came during the workday as students and employees worked from home. For most ISPs, the busy hour is still the evenings, and so there has been far less than a 25% increase in busy-hour demand. Most network should have been able to absorb this burst in growth.

This is not to say that all networks handled COVID growth well. For example, it’s been clear that the big telcos haven’t been investing money in their DSL networks for many years. Performance in those networks has been degrading every year as broadband usage increases. Customers in neighborhood with any significant number of DSL customers have seen broadband speeds decrease year after year as their the demand for broadband has increased. Anybody who has been working at home on DSL during the pandemic saw the network performance in the daytime nearly die.

The Nokia Deepfield blog introduces a new fact that I’d not heard before. They report that distributed denial of service (DDoS) attacks are up 50% during the COVID crisis. At first blush this seems counterintuitive because a lot of businesses have been shut down during the pandemic. Nokia Deepfield says the increased DDoS traffic comes from gamers. Apparently gamers can pay $30 to launch a custom 5-minute DDoS attack against an opponent. Anybody that has seen their neighborhood broadband become useless for five minutes might have been the unintended victim of such an attack. If we had an FCC that regulated broadband they might be investigating this kind of destructive web practice – but this is something they will leave to somebody else.

The bottom line on traffic growth is that. overall, most networks should have been prepared to absorb the growth in traffic due to the pandemic. Most of the growth happened during non-busy hours, and so, while the networks saw a lot of growth in traffic volumes they didn’t see an equal growth in network stress. The bad news for network engineers is that a lot of the recent growth looks like it will stick around, and the overall volumes of web traffic will probably continue to grow at 20% annually on top of the COVID growth.

Categories
Improving Your Business Technology

Expect a New Busy Hour

One of the many consequences of the coronavirus is that networks are going to see a shift in busy hour traffic. Busy hour traffic is just what is sounds like – it’s the time of the day when a network is busiest, and network engineers design networks to accommodate the expected peak amount of bandwidth usage.

Verizon reported on March 18 that in the week since people started moving to work from home that they’ve seen a 20% overall increase in broadband traffic. Verizon says that gaming traffic is up 75% as those stuck at home are turning to gaming for entertainment. They also report that VPN (virtual private network) traffic is up 34%. A lot of connections between homes and corporate and school WANs are using a VPN.

These are the kind of increases that can scare network engineers, because Verizon just saw a typical year’s growth in traffic happen in a week. Unfortunately, the announced Verizon traffic increases aren’t even the whole story since we’re just at the beginning of the response to the coronavirus. There are still companies figuring out how to give secure access to company servers and the work-from-home traffic is bound to grow in the next few weeks. I think we’ll see a big jump in video conference traffic on platforms like Zoom as more meeting move online as an alternative to live meetings.

For most of my clients, the busy hour has been in the evening when many homes watch video or play online games. The new paradigm has to be scaring network engineers. There is now likely going to be a lot of online video watching and gaming during the daytime in addition to the evening. The added traffic for those working from home is probably the most worrisome traffic since a VPN connection to a corporate WAN will tie up a dedicated path through the Internet backbone – bandwidth that isn’t shared with others. We’ve never worried about VPN traffic when it was a small percentage of total traffic – but it could become one of the biggest continual daytime uses of bandwidth. All of the work that used to occur between employees and the corporate server inside of the business is now going to traverse the Internet.

I’m sure network engineers everywhere are keeping an eye on the changing traffic, particularly to the amount of broadband used during the busy hour. There are a few ways that the busy hour impacts an ISP. First, they must buy enough bandwidth to the Internet to accommodate everybody. It’s typical to buy at least 15% to 20% more bandwidth than is expected for the busy hour. If the size of the busy hour shoots higher, network engineers are going to have to quickly buy a larger pipe to the Internet, or else customer performance will suffer.

Network engineers also keep a close eye on their network utilization. For example, most networks operate with some rule of thumb, such as it’s time to upgrade electronics when any part of the network hits some pre-determined threshold like 85% utilization. These rules of thumb have been developed over the years as warning signs to provide time to make upgrades.

The explosion of traffic due to the coronavirus, might shoot many networks past these warning signs and networks start experiencing chokepoints that weren’t anticipated just a few weeks earlier. Most networks have numerous possible chokepoints – and each is monitored. For example, there is usually a chokepoint going into neighborhoods. There are often chokepoints on fiber rings. There might be chokepoints on switch and router capacity at the network hub. There can be the chokepoint on the data pipe going to the world. If any one part of the network gets overly busy, then network performance can degrade quickly.

What is scariest for network engineers is that traffic from the reaction to the coronavirus is being layered on top of networks that already have been experiencing steady growth. Most of my clients have been seeing year-over-year traffic volumes increases of 20% to 30%. If Verizon’s experience in indicative of what we’ll all see, then networks will see a year’s typical growth happen in just weeks. We’ve never experienced anything like this, and I’m guessing there aren’t a lot of network engineers who are sleeping well this week.

Categories
Regulation - What is it Good For?

Setting the FCC Definition of Broadband

In the recently released 2018 Broadband Progress Report the FCC reluctantly kept the official definition of broadband at 25/3 Mbps. I say reluctantly because three of the Commissioners were on record for either eliminating the standard altogether or else reverting back to the older definition of 10/1 Mbps.

I’m guessing the Commissioners gave in to a lot of public pressure to keep the 25/3 standard. Several Commissioners had also taken a public stance that they wanted to allow cellular data to count the same for a household as landline broadband – and that desire was a big factor in lowering the definition since cellphones rarely meet the 25/3 speed standard.

The deliberation on the topic this year raises the question if there is some way to create a rule that would better define the speed of needed broadband. It’s worth looking back to see how the Tom Wheeler FCC came up with the 25/3 definition. They created sample profiles of the way that households of various sizes are likely to want to use broadband. In doing so, they added together the bandwidth needed for various tasks such as watching a movie or supporting a cellphone.

But the FCC’s method was too simple and used the assumption that various simultaneous uses of broadband are additive. They added together the uses for a typical family of four which resulted in bandwidth needs greater than 20 Mbps download, and used that as the basis for setting the 25/3 standard. But that’s now home broadband works. There are several factors that affect the actual amount of bandwidth being used:

For example, doing simultaneous tasks on a broadband network increases the overhead on the home network. If you are watching a single Netflix stream, the amount of needed bandwidth is predictable and steady. But if three people in a home are each watching a different Netflix the amount of needed bandwidth is greater than adding together the three theoretical streams. When your ISP and your home router try to receive and untangle multiple simultaneous streams there are collisions of packets that get lost and which have to be retransmitted. This is described as adding ‘overhead’ to the transmission process. Depending on the nature of the data streams the amount of collision overhead can be significant.

Almost nobody directly wires the signal from their ISP directly too all of their devices. Instead we use WiFi to move data around to various devices in the home. A WiFi router has an overhead of its own that adds to the overall bandwidth requirement. As I’ve covered in other blogs, a WiFi network is not impacted only by the things you are trying to do in your home, but a WiFi network is slowed when it pauses to recognizes demands for connection from your neighbor’s WiFi network.

Any definition of home broadband needs should reflect these overheads. If a household actually tries to download 25 Mbps of usage from half a dozen sources at the same time on a 25 Mbps, the various overheads and collisions will nearly crash the system.

The FCC’s definition of broadband also needs to reflect the real world. For example, most of the unique programming created by Netflix and Amazon Prime are now available in 4K. I bought a large TV last year and we now watch 4K when it’s available. That means a stream of 15-20 Mbps download. That stream forced me to upgrade my home WiFi network to bring a router into the room with the TV.

The FCC’s speed definition finally needs to consider the busy hour of the day – the time when a household uses the most broadband. That’s the broadband speed that the home needs.

We know household bandwidth needs keep increasing. Ten years ago I was happy with a 5 Mbps broadband product. Today I have a 60 Mbps product that seems adequate, but I know from tests I did last year that I would be unhappy with a 25 Mbps connection.

The FCC needs a methodology that would somehow measure actual download speeds at a number of homes over time to understand what homes area really using for bandwidth. There are ways that this could be done. For example, the FCC could do something similar for broadband like what Nielsen does for cable TV. The FCC could engage one of the industry firms that monitor broadband usage such as Akamai to sample a large number of US homes. There could be sample voluntary homes that meet specific demographics that would allow monitoring of their bandwidth usage. The accumulated data from these sample homes would provide real-life bandwidth usage as a guide to setting the FCC’s definition of broadband. Rather than changing the official speed periodically, the FCC could change the definition as needed as dictated by the real-world data.

The FCC does some spot checking today of the broadband speeds as reported by the ISPs that feed the national broadband map. But that sampling is random and periodic and doesn’t provide the same kind of feedback that a formal ongoing measuring program would show. We have tools that could give the FCC the kind of feedback it needs. Of course, there are also political and other factors used in setting the official definition of broadband, and so perhaps the FCC doesn’t want real facts to get into the way.

Exit mobile version