Tag Archives: oversubscription

Urban WISPs

When Tarana released the new G2 generation of radios, one of the claims the company made was that the radios are powerful enough for ISPs to bring point-to-multipoint broadband to metropolitan areas.

The new specifications support the premise. The new G2 radio can support up to 512 customers in a sector (2,048 customers in total). Each sector can accept up to 6.4 Gbps of backhaul bandwidth. If all customers are provided with a gigabit product, the oversubscription ratio would be 20:1. That’s about double the ratio for current fiber technology, but many ISPs would find this to be acceptable. If a WISP were to install a Tarana radio on an existing tall urban tower, it should easily be able to see a lot more than 2,000 homes and businesses.

Selling fixed wireless in an urban area is an interesting marketing plan because there are a lot of locations that won’t be reachable from a single radio site. Rural areas have dead zones created for wireless technologies by hills and other impediments. In a city, the dead zones are more pronounced but localized. There is dead zone created behind every tall building. A home next to a taller neighbor might not be able to see a tower directly. This can be solved to some extent by using multiple towers, but tower real estate in many cities is hard to come by. Many towers are already loaded to capacity or are reserving any remaining capacity for the giant cell companies.

But a WISP could never serve everybody in an urban environment. There are nowhere near enough tall towers in a typical city or suburb to serve more than a relatively small percentage of households. A WISP marketing plan will shoot to fill all of the slots on a radio. However, there is a proceeding at the FCC that is considering overriding local restrictions on tower placement, which could be a huge boon to WISPs and cellular FWA providers.

This is an intriguing idea that would bring yet another competitive option to cities. There is already a price war of sorts for gigabit broadband in cities. For example, in highly competitive markets, customers can buy a gigabit from AT&T for $65 per month, with even lower introductory prices for new customers. Cable companies are falling all over themselves with low rates to keep customers from churning. It seems like a WISP that doesn’t have to support a wired network ought to be able to be price-competitive.

Wireless technology also brings a new option for redundancy for businesses. A growing percentage of businesses are willing to pay for two diversely routed ISPs to make sure they can stay connected to broadband. A wireless backup provides a lot more safety as a backup connection since local events like a downed pole might knock out both fiber and the cable company.

It’s an interesting option that brings a real wireless option for many urban homes. The FCC broadband map shows that I have eight choices of broadband at my house, but most of them are wireless carriers that can’t actually serve me with acceptable broadband. I’ve never understood why WISPs claim coverage across whole cities. When I look at detailed Ookla speed tests, most cities show barely any actual use from the WISPs. There are exceptions, but the current FCC maps are highly exaggerated for cities. But the FCC has no incentive to clean up the maps when every home in a city has one or two legitimate options for gigabit broadband.

The Tarana announcement has to worry the big cable companies that are already struggling to hang on to every customer. Losing even just a few thousand customers in every urban market to WISPs presents a new competitive worry.

Speed Isn’t Everything

The marketing are of the broadband industry spends a lot of time convincing folks that the most important part of a broadband product is download speed. This makes sense if fiber of cable are competing in a market against slower technologies. But it seems like most advertising about speed is to convince existing customers to upgrade to faster speeds. While download speed is performance, the industry doesn’t spend much time talking about the other important attributes of broadband.

Upload Speed. Households that make multiple simultaneous upload connections like video calls, gaming, or connecting to a work or school server quickly come to understand the importance of upload speeds if they don’t have enough. This was the primary problem that millions of households subscribed to cable companies encountered during the pandemic when they suddenly were using a lot of upload. Many homes still struggle with this today, and too many people upgrade to faster download speeds, hoping to solve the problem. ISPs using technologies other than fiber rarely mention upload speed.

Oversubscription. Home broadband connections are served by technologies that share bandwidth across multiple customers. Your ISP is very unlikely to tell you the number of people sharing your node or the amount of bandwidth feeding your node. The FCC’s broadband labels require ISPs to disclose their network practices, but nobody tells you statistics like this that would help you compare the ISPs competing for your business. The cable industry ran afoul of this issue fifteen years ago when large numbers of homes began streaming video, and many ran into it again during the pandemic. It still happens today any time a neighborhood has more demand than the bandwidth being supplied.

Latency. The simple description of latency is the delay in getting the packets to your home for something sent over the Internet. Latency increases any time that packets have to be resent and pile up. If enough packets get backlogged, latency can make it difficult or impossible to maintain a real-time connection. Latency issues are behind a lot of the problems that people have with Zoom or Teams calls – yet most folks assume the problem is not having fast enough speed.

Prioritization. A new problem for some broadband customers is prioritization. Customers buying FWA cellular wireless are told upfront that their usage might be slowed if there is too much cellular demand at a tower. Cellular carriers clearly (and rightfully) give priority to cell phones users over home broadband. Starlink customers who buy mobile broadband are given the same warning. Starlink will prioritize normal customers in an area over campers and hikers. Most ISPs say they don’t prioritize, but as AI is introduced into networks it will be a lot easier for them to do so. Over the last few months I’ve seen that several big ISPs are considering selling a priority (and more expensive) connection to gamers at the expense of everybody else.

Your Home Network. Everybody wants to blame the ISP when they have problems. However, a large percentage of broadband problems come from WiFi inside the home. People keep outdated and obsolete WiFi routers that are undersized for their bandwidth. Customers try to reach an entire home from a single WiFi device. Even when customers use WiFi extenders and mesh networks to reach more of the home, they often deploy the devices poorly. If you are having any broadband problems, give yourself a present and buy a new WiFi router.

Reliability. If operated properly, fiber networks tend to be the most reliable. But there are exceptions, and it all boils down to the quality of your local ISP as it does to the technology. It’s hard to say that any factor is more important than reliability if your ISP regularly has network outages when you want to use broadband.

It’s the ISP, Not Just the Technology

Davis Strauss recently wrote an article for Broadband Breakfast that reminded me that good technology does not always mean a good ISP. There are great and not so great ISPs using every technology on the market. Mr. Strauss lists a few of the ways that an ISP can cut costs when building and operating a fiber network that are ultimately detrimental to customers. Following are a few examples.

Redundant Backhaul. Many of the BEAD grants will be built in areas where the existing broadband networks fail regularly due to cuts in the single fiber backhaul feeding the area. I hear stories all of the time of folks who lose broadband regularly for a few days at a time and sometimes much longer. Building fiber last-mile will not solve the backhaul issue if the new network relies on the same unreliable backhaul.

Oversubscription. It’s possible to overload a local fiber network just like any other network if an ISP loads more customers into a network node that can be supported by the bandwidth supplying the node. There are multiple places where a fiber network can get overstressed, including the path between the core and neighborhoods and the path from the ISP to the Internet.

Lack of Spares. Fiber circuit cards and other key components in the network fail just like any other electronics. A good ISP will have spare cards within easy reach to be able quickly restore a network in the case of a component failure. An ISP that cuts corners by not stocking spares can have multi-day outages while a replacement is located.

Poor Network Records. This may not sound like an important issue  but it’s vital for good customer service and network maintenance. Fiber wires are tiny and are not easy for a field technician to identify if there are not great records that match a given fiber to a specific customer. There is an upfront effort and cost required to organize records, and an ISP that skimps on record keeping will be forever disorganized and take longer to perform routine repairs and maintenance.

Not Enough Technicians. Possibly the most important issue in maintaining a good network is to have enough technicians to support the network. The big telcos have historically understaffed field technicians which has resulted in customers waiting days or weeks just to have a technician respond to a problem. ISPs can save a lot of money by running a too-lean staff to the detriment of customers.

Inadequate Monitoring. ISPs that invest in good network monitoring can head off a huge percentage of customer problems by reacting to network issues before customers even realize there is a problem. A huge percentage of network problems can be remedied remotely by a skilled technician if the ISP is monitoring the performance of all segments of a network.

These are just a few examples of ways that ISPs can cut corners. It is these behind-the-scenes decisions on how to operate that differentiate good and poor ISPs. Mr. Strauss doesn’t come right out and say it, but his article implies that there will be ISPs chasing the giant BEAD funding that will be in the business of maximizing profits early to be able to flip the business. An ISP with this mentality is not going to spend money on redundant backhaul, record-keeping, spares, or network monitoring. An ISP with this mentality will hope that a new fiber network can eke by without the extra spending. They might even be right about this for a few years, but eventually, taking short cuts always comes back to cost more than doing things the right way.

We already know that some ISPs cut corners, because we’ve seen them for the last several decades. The big telcos will declare loudly that DSL network perform badly because of the aging of the networks. There is some truth in that, but there are other ISPs still operating DSL networks that perform far better. The rural copper networks of big telcos perform so poorly because the big telcos cut every cost possible. They eliminated technicians, didn’t maintain spare inventories, and invested nothing in additional backhaul.

I honestly don’t know how a state broadband office is going to distinguish between an ISP that will do things right and one that will cut corners – that’s not the sort of thing that can be captured in a grant application since every ISP will say it plans to do a great job and will offer superlative customer service.

A Strategy for Upgrading GPON

I’ve been asked a lot during 2018 if fiber overbuilders ought to be considering the next generation of PON technology that might replace GPON. They hear about the newer technologies from vendors and the press. For example, Verizon announced a few months ago that they would begin introducing Calix NGPON2 into their fiber network next year. The company did a test using the technology recently in Tampa and achieved 8 Gbps speeds. AT&T has been evaluating the other alternate technology, XGS-PON, and may be introducing it into their network in 2019.

Before anybody invests a lot of money in a GPON network it’s a good idea to always ask if there are better alternatives – as should be done for every technology deployed in the network.

One thing to consider is how Verizon plans on using NGPON2. They view this as the least expensive way to deliver bandwidth to a 5G network that consists of multiple small cells mounted on poles. They like PON technology because it accommodates multiple end-points using a single last-mile fiber, meaning a less fiber-rich network than with other 10-gigabit technologies. Verizon also recently began the huge task of consolidating their numerous networks and PON gives them a way to consolidate multi-gigabit connections of all sorts onto a single platform.

Very few of my clients operate networks that have a huge number of 10-gigabit local end points. Anybody that does should consider Verizon’s decision because NGPON2 is an interesting and elegant solution for handling multiple large customer nodes while also reducing the quantity of lit fibers in the network.

Most clients I work with operate PON networks to serve a mix of residential and business customers. The first question I always ask them is if a new technology will solve an existing problem in their network. Is there anything that a new technology can do that GPON can’t do? Are my clients seeing congestion in neighborhood nodes that are overwhelming their GPON network?

Occasionally I’ve been told that they want to provide faster connections to a handful of customers for which the PON network is not sufficient – they might want to offer dedicated gigabit or larger connections to large businesses, cell sites or schools. We’ve always recommended that clients design networks with the capability of large Ethernet connections external to the PON network. There are numerous affordable technologies for delivering a 10-gigabit pipe directly to a customer with active Ethernet. It seems like overkill to consider upgrading the electronics to all customers to satisfy the need of a few large customers rather than overlaying a second technology into the network. We’ve always recommended that networks have some extra fiber pairs in every neighborhood exactly for this purpose.

I’ve not yet heard an ISP tell me that they are overloading a residential PON network due to customer data volumes. This is not surprising. GPON was introduced just over a decade ago, and at that time the big ISPs offered speeds in the range of 25 Mbps to customers. GPON delivers 2.4 gigabits to up to 32 homes and can easily support residential gigabit service. At the time of introduction GPON was at least a forty-times increase in customer capacity compared to DSL and cable modems – a gigantic leap forward in capability. It takes a long time for consumer household usage to grow to fill that much new capacity. The next biggest leap forward we’ve seen was the leap from dial-up to 1 Mbps DSL – a 17-times increase in capacity.

Even if somebody starts reaching capacity on a GPON there are some inexpensive upgrades that are far less expensive than upgrading to a new technology. A GPON network won’t reach capacity evenly and would see it in some neighborhood nodes first. The capacity in a neighborhood GPON node can easily be doubled by cutting the size of the node in half by splitting it to two PONs. I have one client that did the math and said that as long as they can buy GPON equipment they would upgrade by splitting a few times – from 32 to 16 homes and from 16 homes to 8 homes, and maybe even from 8 to 4 customers before they’d consider tearing out GPON for something new. Each such split doubles capacity and splitting nodes three times would be an 8-fold increase in capacity. If we continue on the path of seeing household bandwidth demand double every three years, then splitting nods twice would easily add more than another decade to the life of a PON network. In doing that math it’s important to understand that splitting a node actually more than doubles capacity because it also decreases the oversubscription factor for each customer on the node.

AT CCG we’ve always prided ourselves on being technology neutral and vendor neutral. We think network providers should use the technology that most affordably fits the needs of their end users. We rarely see a residential fiber network where GPON is not the clear winner from a cost and performance perspective. We have clients using numerous active Ethernet technologies that are aimed at serving large businesses or for long-haul transport. But we are always open-minded and would easily recommend NGPON2 or XGS-PON if it is the best solution. We just have not yet seen a network where the new technology is the clear winner.

The WISP Dilemma

For the last decade I have been working with many rural communities seeking better broadband. For the most part these are places that the large telcos have neglected and never provided with any functional DSL. Rural America has largely rejected the current versions of satellite broadband because of the low data caps and because the latency won’t support streaming video or other real-time activities. I’ve found that lack of broadband is at or near the top of the list of concerns in communities without it.

But a significant percentage of rural communities have access today to WISPs (wireless ISPs) that use unlicensed frequency and point-to-multipoint radios to bring a broadband connection to customers. The performance of WISPs varies widely. There are places where WISPs are delivering solid and reliable connections that average between 20 – 40 Mbps download. But unfortunately there are many other WISPs that are delivering slow broadband in the 1 – 3 Mbps range.

The WISPs that have fast data speeds share two characteristics. They have a fiber connection directly to each wireless transmitter, meaning that there are no bandwidth constraints. And they don’t oversubscribe customers. Anybody who was on a cable modem five or ten years ago understands oversubscription. When there are too many people on a network node at the same time the performance degrades for everybody. A well-designed broadband network of any technology works best when there are not more customers than the technology can optimally serve.

But a lot of rural WISPs are operating in places where there is no easy or affordable access to a fiber backbone. That leaves them with no alternative but to use wireless backhaul. This means using point-to-point microwave radios to get bandwidth to and from a tower.

Wireless backhaul is not in itself a negative issue. If an ISP can use microwave to deliver enough bandwidth to a wireless node to satisfy the demand there, then they’ll have a robust product and happy customers. But the problems start happening when networks include multiple ‘hops’ between wireless towers. I often see WISP networks where the bandwidth goes from tower to tower to tower. In that kind of configuration all of the towers and all of the customers on those towers are sharing whatever bandwidth is sent to the first tower in the chain.

Adding hops to a wireless network also adds latency and each hop means it takes longer for the traffic to get to and from customers at the outer edges of one of these wireless chains. Latency, or time lag, in signal is an important factor in being able to perform real-time functions like data streaming, voice over IP, gaming, or functions like maintaining connections to an on-line class or a distant corporate WAN.

Depending upon the brand of the radios and the quality of the internet backbone connection, a wireless transmitter that is connected directly to fiber can have a latency similar to that of a cable or DSL network. But when chaining multiple towers together the latency can rise significantly, and real-time applications start to suffer at latencies of 100 milliseconds or greater.

WISPs also face other issues. One is the age of the wireless equipment. There is no part of our industry that has made bigger strides over the past ten years than the manufacturing of subscriber microwave radios. The newest radios have significantly better operating characteristics than radios made just a few years ago. WISPs are for the most part relatively small companies and have a hard time justifying upgrading equipment until it has reached its useful life. And unfortunately there is not much opportunity for small incremental upgrades of equipment. The changes in the technologies have been significant enough that that upgrading a node often means replacing the transmitters on towers as well as subscriber radios.

The final dilemma faced by WISPs is that they often are trying to serve customers that are in locations that are not ideally situated to receive a wireless signal. The unlicensed frequencies require good line-of-sight and also suffer degraded signals from foliage, rain and other impediments and it’s hard to serve customer reliably who are surrounded by trees or who live in places that are somehow blocked by the terrain.

All of the various issues mean that reviews of WISPs vary as widely as you can imagine. I was served by a WISP for nearly a decade and since I lived a few hundred feet from the tower and had a clear line-of-sight I was always happy with the performance I received. I’ve talked to a few people recently who have WISP speeds as fast as 50 Mbps. But I have also talked to a lot of rural people who have WISP connections that are slow and have high latency that provides a miserable broadband experience.

It’s going to be interesting to see what happens to some of these WISPs as rural telcos deploy CAF II money and provide a faster broadband alternative that will supposedly deliver at least 10 Mbps download. WISPs who can beat those speeds will likely continue to thrive while the ones delivering only a few Mbps will have to find a way to upgrade or will lose most of their customers.

Is There a Web Video Crisis? – Part I

The InternetThe whole net neutrality issue has been driven by the fact that companies like Comcast and Verizon want to charge large content providers like NetFlix to offset some of their network cost to carry their videos. Comcast implies that without such payments that NetFlix content will have trouble making it to customers. By demanding such payments Comcast is saying that their network is having trouble carrying video, meaning that there is a video crisis on the web or on the Comcast network.

But is there? Certainly video is king and constitutes the majority of traffic on the web today. And the amount of video traffic is growing rapidly as more customers watch video on the web. But everybody has known for years that this is coming and Comcast can’t be surprised that it is being asked to deliver video to people.

Let’s look at this issue first from the edge backwards. Let’s say that on average in a metro area that Comcast has sold a 20 Mbps download to each of its customers. Some buy slower or faster speeds than that, but every one of Comcast’s products is fast enough to carry streaming video. Like all carriers Comcast does something called oversubscription, meaning that they sell more access to customers than their network can supply. But in doing so they are banking on the fact that everybody won’t watch video at the same time. And they are right, it never happens. I have a lot of clients with broadband networks and I can’t think of one of them who has been overwhelmed in recent years by demands from customers on the edge. Those edge networks ought to be robust enough to deliver the speeds that are sold to customers. That is the primary thing customers are paying for.

So Comcast’s issues must be somewhere else in the network, because their connections to customers ought to robust enough to deliver video to a lot of people at the same time. One place that could be a problem is the Internet backbone. This is the connection between Comcast and the Internet. I have no idea how Comcast manages this, but I know how hundreds of smaller carriers do it. They generally buy enough capacity so that they are rarely use more than some base amount like 60% of the backbone. By keeping a comfortable overhead on the Internet pipe they are ready for those rare days when usage bursts much higher. And if they do get too busy they usually have the ability to burst above their proscribed bandwidth limits to satisfy customer demands. This costs them more, but the capacity is available to them without them having to ask for it.

So one would not think that the issue for Comcast is their connection to the Internet. They ought to be sizing this according to the capacity that they are selling in aggregate to all of the end users. The price of backbone has been dropping steadily for years and the price that their customers pay them for bandwidth should be sufficient for them to make the backbone robust enough.

That only leaves one other part of the network, which is what we refer to as distribution. This is the fiber connections that go between a headend or a hub out to neighborhoods. Certainly these connections have gotten larger over time and I would assume that like all carriers that Comcast has had to increase capacity in the distribution plant. Where a neighborhood might have once been perfectly fine sharing a gigabit of data they might now need ten gigabits. That kind of upgrade means upgrading to a larger laser on the fiber connection between the Comcast headend and neighborhood nodes.

Again, I would think that the prices that customers pay ought to cover the cost of the distribution network, just as it ought to cover the edge network and the backbone network. Comcast has been unilaterally increasing speeds to customers over time. They come along and periodically increase speeds for customers, say from 10 to 15 Mbps. One would assume that they would only increase speeds if they have the capacity to actually deliver those new higher speeds.

From the perspective of looking at the base components of the network – the edge, the backbone and the distribution, I can’t see where Comcast should be having a problem. The prices that customers pay ought to be more than sufficient to make sure that those three components are robust enough to deliver what Comcast is selling to customers. If it’s not, then Comcast has oversold their capacity, which sounds like their issue and not NetFlix.

In the next article in this series I will look at other issues such as caching as possible reasons for why Comcast needs extra payments from NetFlix. Because it doesn’t appear to me that NetFlix ought to be responsible for the way Comcast builds their own networks. One would think that their networks are built to generally deliver the bandwidth to customers they have paid for regardless of where on the web that bandwidth is coming from.