Categories
Uncategorized

Gaming and Broadband Demand

Broadband usage has spiked across the US this year as students and employees suddenly found themselves working from home and needing broadband to connect to school and work servers. But there is another quickly growing demand for broadband coming from gaming.

We’ve had online gaming of some sort over the last decade, but gaming has not been data-intensive activity for ISPs. Until recently, the brains for gaming has been provided by special gaming computers or game boxes run locally by each gamer. These devices and the game software supplied the intensive video and sound experience and the Internet was only used to exchange game commands between gamers. Command files are not large and contain the same information that is exchanged between a game controller and a gaming computer. In the past, gamers would exchange the command files across the Internet, and local software would interpret and activate the commends being exchanged.

But the nature of online gaming is changing rapidly. Already, before the pandemic, game platforms had been migrating online. Game companies are now running the core software for games in a data center and not on local PCs or game consoles. The bandwidth path required between the data center core and a gamer is much larger than the command files that used to be exchanged since the data path now carries the full video and music signals as well as 2-way communications between gamers.

There is a big benefit of online gaming for gamers, assuming they have enough bandwidth to participate. Putting the gaming brains in a data center reduces the latency, meaning that game commands can be activated more quickly. Latency is signal delay, and the majority of the delay in any internet transmission happens inside the wires and electronics of the local ISP network. With online gaming, a signal between a gamer only has to cross the gamer’s local ISP network. Before online gaming, that signal had to pass through the local ISP network of both gamers.

There are advantages for gaming companies to move online. They can release a new title instantly to the whole country. Game companies don’t have to manufacture and distribute copies of games. Games can now be sold to gamers who can’t afford the expensive game boxes or computers. Gamers benefit because gaming can now be played on any device and a gamer isn’t forced into buying an expensive gaming computer and then only playing in that one location. Game companies can now sell a gaming experience that can be played from anywhere, not just sitting at a gamer’s computer.

A gaming stream is far more demanding on the network than a video stream from Netflix. Netflix feeds out the video signal in advance of what a viewer is watching, and the local TV or PC stores video content for the next few minutes of viewing. This was a brilliant move by video streamers because streaming ahead of where what viewers are watching largely eliminated the delays and pixelation of video streams that were common when Netflix was new. By streaming in advance of what a viewer is watching, Netflix has time to resend any missed packets so that the video viewing experience has ideal quality by the time a viewer catches up to the stream.

Gaming doesn’t have this same luxury because gaming is played in real time. The gamers at both ends of a game need to experience the game at the same time. This greatly changes the demand on the broadband network. Online gaming means a simultaneous stream being sent from a data center to both gamers, and it’s vital that both gamers receive the signal at the same time. Gaming requires a higher quality of download path than Netflix because there isn’t time to resend missed data packets. A gamer needs a quality downstream path to receive a quality video transmission in real-time.

Gaming adds a second big demand in that latency becomes critical. A player that receives signal just a little faster than an opponent has an advantage. A friend of mine has symmetrical gigabit Verizon FiOS fiber broadband at his home which is capable of delivering the best possible gaming data stream. Yet his son is driving his mother crazy by running category 6 cables between the gaming display and the FiOS modem. He sears that bypassing the home WiFi lowers the latency and gives him an edge over other gamers. From a gamer perspective, network latency is becoming possibly more important than download speed. A gamer on fiber has an automatic advantage over a gamer on a cable company network.

At the same time as the gaming experience has gotten more demanding for network operators the volume of gaming has exploded during the pandemic as people stuck at home have turned to gaming. All of the major game companies are reporting record earnings. The NPD Group that tracks the gaming industry reports that spending on gaming was up 30% in the second quarter of this year compared to 2019.

ISPs are already well aware of gamers who are the harshest critics of broadband network performance. Gamers understand that little network glitches, hiccups, and burps that other uses may not even notice can cost them a game, and so gamers closely monitor network performance. Most ISPs know their gamers who are the first to complain loudly about network problems.

Categories
Improving Your Business Technology

Expect a New Busy Hour

One of the many consequences of the coronavirus is that networks are going to see a shift in busy hour traffic. Busy hour traffic is just what is sounds like – it’s the time of the day when a network is busiest, and network engineers design networks to accommodate the expected peak amount of bandwidth usage.

Verizon reported on March 18 that in the week since people started moving to work from home that they’ve seen a 20% overall increase in broadband traffic. Verizon says that gaming traffic is up 75% as those stuck at home are turning to gaming for entertainment. They also report that VPN (virtual private network) traffic is up 34%. A lot of connections between homes and corporate and school WANs are using a VPN.

These are the kind of increases that can scare network engineers, because Verizon just saw a typical year’s growth in traffic happen in a week. Unfortunately, the announced Verizon traffic increases aren’t even the whole story since we’re just at the beginning of the response to the coronavirus. There are still companies figuring out how to give secure access to company servers and the work-from-home traffic is bound to grow in the next few weeks. I think we’ll see a big jump in video conference traffic on platforms like Zoom as more meeting move online as an alternative to live meetings.

For most of my clients, the busy hour has been in the evening when many homes watch video or play online games. The new paradigm has to be scaring network engineers. There is now likely going to be a lot of online video watching and gaming during the daytime in addition to the evening. The added traffic for those working from home is probably the most worrisome traffic since a VPN connection to a corporate WAN will tie up a dedicated path through the Internet backbone – bandwidth that isn’t shared with others. We’ve never worried about VPN traffic when it was a small percentage of total traffic – but it could become one of the biggest continual daytime uses of bandwidth. All of the work that used to occur between employees and the corporate server inside of the business is now going to traverse the Internet.

I’m sure network engineers everywhere are keeping an eye on the changing traffic, particularly to the amount of broadband used during the busy hour. There are a few ways that the busy hour impacts an ISP. First, they must buy enough bandwidth to the Internet to accommodate everybody. It’s typical to buy at least 15% to 20% more bandwidth than is expected for the busy hour. If the size of the busy hour shoots higher, network engineers are going to have to quickly buy a larger pipe to the Internet, or else customer performance will suffer.

Network engineers also keep a close eye on their network utilization. For example, most networks operate with some rule of thumb, such as it’s time to upgrade electronics when any part of the network hits some pre-determined threshold like 85% utilization. These rules of thumb have been developed over the years as warning signs to provide time to make upgrades.

The explosion of traffic due to the coronavirus, might shoot many networks past these warning signs and networks start experiencing chokepoints that weren’t anticipated just a few weeks earlier. Most networks have numerous possible chokepoints – and each is monitored. For example, there is usually a chokepoint going into neighborhoods. There are often chokepoints on fiber rings. There might be chokepoints on switch and router capacity at the network hub. There can be the chokepoint on the data pipe going to the world. If any one part of the network gets overly busy, then network performance can degrade quickly.

What is scariest for network engineers is that traffic from the reaction to the coronavirus is being layered on top of networks that already have been experiencing steady growth. Most of my clients have been seeing year-over-year traffic volumes increases of 20% to 30%. If Verizon’s experience in indicative of what we’ll all see, then networks will see a year’s typical growth happen in just weeks. We’ve never experienced anything like this, and I’m guessing there aren’t a lot of network engineers who are sleeping well this week.

Categories
What Customers Want

The Terabyte Household

I was just in a meeting the other day with a bunch of ISPs were talking about household downloads. Several said that they were now seeing monthly data usage exceed a terabyte, and those with Comcast were lamenting that this is causing them a lot of money.

I wrote a lot about Comcast data caps a few years ago when the company experimented with really low data caps of 300 gigabytes per month. At that time a lot of households complained that they were exceeding those caps. Comcast was arguing at the time to end net neutrality and I think this persuaded them to back off of the low caps, which they set to 1 terabyte.

Here we are only a few years later and a lot of households are bumping up against and exceeding that data cap. Comcast absolutely knew this was coming and they just pushed the ability to monetize data caps a few years into the future. As an ISP the company knows better than most that the household demand for total downloaded data has been doubling every three years or so. That kind of growth will push a huge number of households over a terabyte within a decade – with many already hitting it now.

Comcast tries to justify data caps by arguing fairness – the same argument they made a few years ago. They say that those that use the Internet the most ought to pay the most. Even if you can buy that argument the penalty for exceeding the data caps are excessive. Comcast doesn’t charge a household for the first two months they exceed a terabyte. After that they have two plans. They will automatically bill $10 for every extra 50 Gigabytes over the data cap – with total excess charges capped at $200 per month. Customers who expect to exceed the data cap can also agree to pay $50 extra every month to get unlimited usage.

Comcast goes on to explain away the terabyte cap by describing what it takes to exceed the cap, as follows:

  • Stream between 600 and 700 hours of HD video
  • Play online games for more than 12,000 hours
  • Stream more than 15,000 hours of music
  • Upload or download more than 60,000 hi-res photos

This explanation is simplistic for a number of reasons. First, full Netflix HD broadcast at 1080p streams at over 7 Mbps and uses roughly 2.5 GB per hour, meaning a terabyte will cover about 400 hours of full HD video. If you have a good broadband connection the chances are that you are watching a lot of 4K video today – it’s something that Netflix and Amazon Prime offer automatically. It only takes only about 180 hours of 4K video in a month to hit the terabyte data cap – a number that is not hard to imagine in a cord-cutting home. The chart also misses obvious large uses like downloading games – with download sizes over 40 GB for one game becoming common.

The Comcast charts also fail to recognize the hidden ways that we all burn through bandwidth today. It’s not untypical for the average household to have a 30% to 40% overhead on Internet usage. That comes from the network having to transmit data multiple times to complete a download request. This overhead is caused for a number of reasons. First are inefficiencies inherent in the open Internet. There are always packets lost on transit that much be sent multiple times. There are also delays caused by the ISP network, particularly networks that are undersized in neighborhoods and that hit capacity during the busy hours. The biggest cause of delays for most of us is in-home WiFi networks that creates a lot of collisions from competing signals.

There are also a lot of background use of the Internet today that surprises people. We now routinely use web storage to back up files. All of the software on our machines upgrade automatically. Many now use applications like video cameras and home alarms that connect in the cloud and that ping back and forth all day. All sorts of other things go on in the background that are a mystery – I’ve noticed my house has significant broadband usage even when we aren’t home. I’ve estimated that this background communication probably eats about 150 gigabytes per month at my house.

When I consider those issues the Comcast terabyte data caps are stingy. A household with a lot of network noise and with a lot of background traffic might hit the data caps using only half of a terabyte of downloaded video or other services like those listed by Comcast. A home today might hit the cap with 200 hours of full HD streaming or 90 hours of 4K streaming.

The other amazing aspect of the terabyte data caps is the charge for using more than a terabyte in a month. As mentioned above, Comcast charges $10 for every extra 50 GB. I’ve done the math for dozens of ISPs and most of my clients spend between $2 and $4 per month on average for the bandwidth per broadband customer. That number includes not only residential users, but for most ISPs also includes some huge commercial broadband customers. The average price varies the most according to how far an ISP is away from the Internet, and that component of the cost is fixed and doesn’t increase due to higher data volumes by the ISP. After backing out this fixed transport cost, my math says that an extra 50 GB of broadband costs an ISP only a few pennies. For a large ISP like Comcast that cost is significantly lower since they peer with the big broadband companies like Netflix, Google and Amazon – and traffic exchanged in those arrangements have nearly zero incremental cost of extra bandwidth.

Finally, the Comcast website claims that less than 1% of their users exceed the terabyte data caps. Only they know the numbers, but I find that hard to believe. When you look at the amount of usage needed to hit that cap there has to be a lot of cord-cutter households already exceeding a terabyte.

The bottom line is that Comcast is extorting homes when they force them to spend $50 per month for unlimited data usage. That extra bandwidth costs them almost nothing. Unfortunately, there isn’t a damned thing any of us can do about this any since Comcast and the other big ISPs got their wish and broadband is no longer regulated by the FCC.

Categories
The Industry

Verizon’s Case for 5G, Part 3

Ronan Dunne, an EVP and President of Verizon Wireless recently made Verizon’s case for aggressively pursuing 5G. In this blog I want to examine the two claims based upon improved latency – gaming and stock trading.

The 5G specification sets a goal of zero latency for the connection from the wireless device to the cellular tower. We’ll have to wait to see if that can be achieved, but obviously the many engineers that worked on the 5G specification think it’s possible. It makes sense from a physics perspective – a connection of a radio signal through air travels for all practical purposes at the speed of light (there is a miniscule amount of slowing from interaction with air molecules). This makes a signal through the air slightly faster than one through fiber since light slows down when passing through fiberglass by 0.83 milliseconds for every hundred miles of fiber optic cable traversed.

This means that a 5G signal will have a slight latency advantage over FTTP – for the first few connection from a customer. However, a 5G wireless signal almost immediately hits a fiber network at a tower or small cell site in a neighborhood, and from that point forward the 5G signal experiences the same latency as an all-fiber connection.

Most of the latency in a fiber network comes from devices that process the data – routers, switches and repeaters. Each such device in a network adds some delay to the signal – and that starts with the first device, be it a cellphone or a computer. In practical terms, when comparing 5G and FTTP the network with the fewest hops and fewest devices between a customer and the internet will have the lowest latency – a 5G network might or might not be faster than an FTTP network in the same neighborhood.

5G does have a latency advantage over non-fiber technologies, but it ought to be about the same advantage enjoyed by FTTP network. Most FTTP networks have latency in the 10-millisecond range (one hundredth of a second). Cable HFC networks have latency in the range of 25-30 ms; DSL latency ranges from 40-70 ms; satellite broadband connections from 100-500 ms.

Verizon’s claim for improving the gaming or stock trading connection also implies that the 5G network will have superior overall performance. That brings in another factor which we generally call jitter. Jitter is the overall interference in a network that is caused by congestion. Any network can have high or low jitter depending upon the amount of traffic the operator is trying to shove through it. A network that is oversubscribed with too many end users will have higher jitter and will slow down – this is true for all technologies. I’ve had clients with first generation BPON fiber networks that had huge amounts of jitter before they upgraded to new FTTP technology, so fiber (or 5G) alone doesn’t mean superior performance.

The bottom line is that a 5G network might or might not have an overall advantage compared to a fiber network in the same neighborhood. The 5G network might have a slight advantage on the first connection from the end user, but that also assumes that cellphones are more efficient than PCs. From that point forward, the network with the fewest hops to the Internet as well the network with the least amount of congestion will be faster – and that will be case by case, neighborhood by neighborhood when comparing 5G and FTTP.

Verizon is claiming that the improved latency will improve gaming and stock trading. That’s certainly true where 5G competes against a cable company network. But any trader that really cares about making a trade a millisecond faster is already going to be on a fiber connection, and probably one that sits close to a major internet POP. Such traders are engaging in computerized trading where a person is not intervening in the trade decision. For any stock trades that involve humans, a extra few thousandths of a second in executing a trade is irrelevant since the human decision process is far slower than that (for someone like me these decisions can be measured in weeks!).

Gaming is more interesting. I see Verizon’s advantage for gaming in making game devices mobile. If 5G broadband is affordable (not a given) then a 5G connection allows a game box to be used anywhere there is power. I think that will be a huge hit with the mostly-younger gaming community. And, since most homes buy broadband from the cable company, lower latency with 5G ought to be to a gamer using a cable network, assuming the 5G network has adequate upload speeds and low jitter. Gamers who want a fiber-like experience will likely pony up for a 5G gaming connection if it’s priced right.

Categories
Technology What Customers Want

The Demand for Upload Speeds

I was recently at a public meeting about broadband in Davis, California and got a good reminder of why upload speeds are as important to a community as download speeds. One of the people making public comments talked about how uploading was essential to his household and how the current broadband products on the market were not sufficient for his family.

This man needed good upload speeds for several reasons. First, he works as a photographer and takes pictures and shoots videos. He says that it takes hours to upload and send raw, uncompressed video to one of his customers and says the experience still feels like the dial-up days. His full-time job is working as a network security consultant for a company that specializes in big data. As such he needs to send and receive large files, and his home upload bandwidth is also inadequate for that – forcing him to go to an office for work that could otherwise be done from his home. Finally, his daughter creates YouTube content and has the same problem uploading content – which is particularly a problem when her content deals with time-sensitive current events and waiting four hours to get the content to YouTube kills the timeliness of her content.

This family is not unusual any more. A decade ago, a photographer led the community effort to get faster broadband in a city I was working with. But he was the only one asking for faster upload speeds and most homes didn’t care about it.

Today a lot of homes need faster upload speeds. This particular family had numerous reasons including working from home, sending large data files and posting original content to the web. But these aren’t the only uses for faster upload speeds. Gamers now need faster upload speeds. Anybody who wants to remotely check their home security cameras cares about upload speeds. And more and more people are migrating to 2-way video communications, which requires those at both ends to have decent uploading. We are just now seeing the early trials of virtual presence where communications will be by big-bandwidth virtual holograms at each end of the communications.

Davis is like many urban areas in that the broadband products available have slow upload speeds. Comcast is the cable incumbent, and while they recently introduced a gigabit download product, their upload speeds are still paltry. DSL is offered by AT&T which has even slower upload speeds.

Technologies differ in their ability to offer upload speeds. For instance, DSL is technically capable of sending the data at the same speeds for upload or download. But DSL providers have elected to stress the download speed, which is what most people value. So DSL products are set with small upload and a lot of download. It would be possible to give a customer the choice to vary the mix between upload and download speeds, but I’ve never heard of an ISP who tried to provide this as an option to customers.

Cable modems are a different story. Historically the small upload speeds were baked directly into the DOCSIS standard. When Cable Labs created DOCSIS they made upload speeds small in response to what cable companies asked from them. Until recently, cable companies have had no option to increase upload speeds beyond the DOCSIS constraints. But Cable Labs recently amended the new DOCSIS 3.1 standard to allow for much upload speeds of nearly a gigabit. The first release of the new DOCSIS 3.1 standard didn’t include this, but it’s now available.

However, a cable company has to make sacrifices in their network if they want to offer faster uploads. It takes about 24 empty channels (meaning no TV signal) on a cable system to provide gigabit download speeds. A cable company would need to vacate many more channels of programming to also offer faster uploads and I don’t think many of them will elect to do so. Programming is still king and cable owners need to balance the demand for more channels compared to demand for faster uploads.

Fiber has no real constraints on upload speeds up to the capability of the lasers. The common technologies being used for residential fiber all allow for gigabit upload speeds. Many fiber providers set speeds to symmetrical, but others have elected to limit upload speeds. The reason I’ve heard for that is to limit the attractiveness of their network for spammers and others who would steal the use of fast uploading. But even these networks offer upload speeds that are far faster than the cable company products.

As more households want to use uploading we are going to hear more demands for a faster upload option. But for now, if you want super-fast upload speeds you have to be lucky enough to live in a neighborhood with fiber-to-the-home.

Categories
The Industry

Broadband and Gaming

I recently saw an interesting statistic that showed that the most popular worldwide video content is gaming. The worldwide gaming video content industry has more than 665 million viewers which makes it bigger than HBO, Netflix, ESPN and Hulu combined. This is a segment of the video industry that I was only peripherally aware of, which I suspect is true for many of you as well.

The GVC (Gaming Video Content) industry is distinct from the on-line playing of games. The GVC content consists of watching others play games along with content that talks about gaming. The industry is estimated to generate $4.6 billion in revenues in 2017. One third of that will come from subscriptions to GVC content along with other direct consumer spending. The rest comes from advertising. There is a whole industry that has sprung up around the GVC content including big conventions and merchandise.

While you can’t characterize such a large group of people, the gamers and GVC viewers are often what you might think of as tech-savvy. In the US the average GVC viewer is around 30, has more education that average and makes a higher than average income of around $58,000. And while you might expect the viewers of GVC content to be largely male a surprising 46% of GVC viewers are female.

Around the world there are numerous video platforms that have been created for gaming content. In the US and Europe the biggest content provider is Twitch. This is a platform that was originally known as Justin.tv. The platform was created in 2007 by Justin Kan and Emmett Shear. The platform allowed users to post live video streams that could be watched by anybody else on the platform. The platform was often used to show pirated live sports feeds, but over time the majority of the content centered around gaming.

Justin.tv was a large content generator and in 2013 – before Netflix really took off – the service said it had 45 million unique viewers and was the fourth largest source of peak Internet traffic in the US. When the biggest competitor to Justin.tv shut down the platform had a near monopoly on gaming content.

The company was renamed to Twitch Interactive and was acquired at the end of 2014 by Amazon. Amazon beefed up the underlying delivery platform, which increased the quality of the streams. Since then Twitch has grown significantly. Amazon reports that the service has over 100 million unique viewers per month, nearly 10 million per day. The average number of simultaneous viewers at any given time on the platform is about 622,000.

Amazon has grown the service by opening up the platform to ‘partners’ much as it has done with OTT content. Twitch now has over 17,000 partners – those that stream unique content. 35% of the content is viewed on cellphones, with the rest on landline broadband connections.

Twitch viewers are loyal. Over half watch the service more than 20 hours per week – and for many of them this is their primary source of video content. The average Twitch user watches the service for 1 hour 46 minutes per day.

While the Twitch platform is free (and I recommend taking a few minutes to watch the above link), many of the channel partners charge monthly subscriptions.

I find it interesting that Twitch is not counted in the universe of OTT providers. But Twitch viewers and statistics are separate from, and not counted with viewers of Amazon Prime. Perhaps this is not considered as OTT content since a lot of the content is viewer-generated. But this is still largely true for YouTube, which is now counted among the OTT providers. Many of the channels on Twitch are now professionally produced and certainly are hard to distinguish from other OTT content.

The GVC industry is worth noting because they are a big source of video content on our broadband networks. The video watched on the web doesn’t just come from sources like Netflix and more and more of it is coming from platforms like Twitch that carry a mountain of viewer-generated content. This is just one more example of how the major programmers are going to be in trouble as the generations turn. Younger viewers are not watching traditional programming to nearly the degree of older generations.

Categories
Technology

A Network Without Wires

There is an emerging trend in the industry to try to create home networks without wires. ISPs and cable companies are all putting a lot of faith into WiFi as an alternative for wires running to computers and settop boxes.

It’s an interesting trend but one that is not without peril. The problem is that WiFi, at least like the big ISPs deliver it, is not always the best solution. The big cable companies like Comcast tend to provide customers with a cable modem with a decent quality WiFi router built in. This router is placed wherever the cable enters the home, which might not be the ideal location.

A single strong WiFi router can be a great device in a home with a simple network and uncomplicated demands. A family with two TVs, one computer, and a few smartphones is probably going to do fine with a strong WiFi router as long as the house isn’t too large for the signal to get where it’s needed.

But we are quickly changing to a society where many homes have complex data needs scattered throughout the house. People are likely to be demanding video streams from all over the home, and often many at the same time. There are bound to be a few computers and it’s not unlikely that somebody in the house works at home at least part of the time. Demands for big bandwidth for things like gaming and the new virtual reality sets that are just now hitting the market are increasing. And we are on the verge of seeing 4K video streams at 15 Mbps. On top of all this will be a variety of smart IoT devices that are going to want occasional attention from the network.

When a home gets crowded with devices it’s very easy to overwhelm a WiFi router. The new routers are pretty adept at setting up multiple data paths. But with too many streams the router will lose efficiency as it constantly tries to monitor and change the bandwidth for each stream it is managing. When this happens a home network can bog down, dropping the efficiency of the router precipitously.

There are a few solutions to this problem. First, you can run wires directly to a few of the bigger data eaters in a house and remove them from the WiFi network. Just make sure in doing so that you also disable having them search for a WiFi signal. But people don’t really want more wires in their home, and ISPs definitely do not like this idea.

The other solution is to add additional WiFi hotspots in the home. The simplest example of this are WiFi repeaters that simply amplify the signal from the base WiFi hotspot. However, repeaters don’t improve the contention issue, they simply bring a stronger signal closer to some of the devices that need them.

A more complex solution is to set up a network of interconnected WiFi hotspots. This consists of separate WiFi routers that all feed through one base router, a configuration that is familiar to any network engineer but alien to most home owners. The main problem with this solution is obvious to anybody who has ever operated a network with multiple routers – getting them to work together efficiently. Setting up a multiple-router network can be challenging to those unfamiliar with networks. And if configured poorly this kind of network can operate worse than one big hotspot.

But these kinds of interconnected WiFi networks are the cutting edge of home networking. I was recently talking to an engineer from a mid-size cable company and he admitted that as many as 20% of their customers already need this kind of solution. It’s a bit ironic that the demand for WiFi is mushrooming so soon after the ISPs went to the one-router solution. The percentage of homes that need a better solution is growing rapidly as homes jam more devices onto WiFi.

So there is an opportunity here for any ISP. Customers need better networks in their homes and there is a revenue opportunity in helping them to set these up. The downside, at least for now, is that this is labor intensive and there may be a lot of maintenance to keep these networks running right. But there are a number of vendors looking into solutions and one would hope that home WiFi networks will soon become plug and play.

Exit mobile version