Tag Archives: DSL

Broadband Interference

Jon Brodkin of ArsTechnica published an amusing story about how the DSL went out of service in a 400-resident village in Wales each morning at 7:00 am. It turns out that one of the residents turned on an ancient television that interfered with the DSL signal to the extent that the network collapsed. The ISP finally figured this out by looking around the village in the morning with a spectrum analyzer until they found the source of the interference.

It’s easy to think that the story points out another weakness of old DSL technology, but interference can be a problem for a lot of other technologies.

This same problem is common on cable company hybrid-fiber coaxial networks. The easiest way to understand this is to think back to the old days when we all watched analog TV. Anybody who watched programming on channels 2 through 5 remembers times when the channels got fuzzy or even became unwatchable. It turns out that there are a lot of different devices that interfere with the frequencies used for these channels including things like microwave ovens, certain motors like power tools and lawnmowers, and other devices like blenders. It was a common household occurrence for one of these channels to go fuzzy when somebody in the house, or even in a neighboring home used one of these devices.

This same interference carries forward into cable TV networks. Cable companies originally used the same frequencies for TV channels inside the coaxial wires that were used over the air and the low TV channels sat between the 5 MHz and 42 MHz frequency. It turns out that long stretches of coaxial wires on poles act as a great antenna, so cable systems pick up the same kinds of interference that happens in homes. It was pretty routine for channels 2 and 3, in particular, to be fuzzy in an analog cable network.

You’d think that this interference might have gone away when cable companies converted TV signals to digital. The TV transmissions for channels 2 through 5 got crystal clear because cable companies relocated the digital version of these channels to better frequency. When broadband was added to cable systems the cable companies continue to use the low frequencies. CableLabs elected to use these frequencies for the upload portion of broadband. There is still plenty of interference in cable networks today – probably even more than years ago as coaxial networks have aged and have more points for interference to seep into the wires. Until the pandemic, we didn’t care much about upload bandwidth, but it turns out that one of the major reasons that cable companies struggle to deliver reliable upload speeds is that they are using the noisiest spectrum for the upload function.

The DSL in the village suffered from the same issue since the telephone copper wires also act as a big outdoor antenna. In this village, the frequency emanating from the old TV exactly matched the frequencies used for DSL.

Another common kind of interference is seen in fixed wireless networks in a situation where there are multiple ISPs using the same frequencies in a given rural footprint. I know of counties where there are as many as five or six different wireless ISPs, and most use the same frequencies since most WISPs rely on a handful of channels in the traditional WiFi bandwidth at 2.4 MHz and 5 MHz. I’ve heard of situations where WiFi is so crowded that the performance of all WISPs suffer.

WiFi also suffers from local interference in the home. The WiFi standard says that all devices have an equal chance of using the frequencies. This means that a home WiFi router will cycle through all the signals from all devices trying to make a WiFi connection. When a WiFi router connects with an authorized device inside the home it allows for a burst of data, but then the router disconnects that signal and tries the next signal – cycling through all of the possible sources of WiFi.

This is the same issue that is seen by people using WiFi in a high-rise apartment building or a hotel where many users are trying to connect to WiFi at the same time. Luckily this problem ought to improve. The FCC has authorized the use of 6 GHz spectrum for home broadband which opens up numerous new channels. Interference will only occur between devices trying to share a channel, but that will be far fewer cases of interference than today.

The technology that has no such interference is fiber. Nothing interferes with the light signal between a fiber hub and a customer. However, once customers connect the broadband signal to their home WiFi network, the same interference issues arise. I looked recently and can see over twenty other home WiFi networks from my office – a setup ripe for interference. Before making too much fun of the folks in the Welsh village, there is a good chance that you are subject to significant interference in your home broadband today.

AT&T Stops DSL Sales

USA Today reported last week that AT&T stopped selling new DSL to customers on October 1. This is an event that will transform the broadband landscape in a negative way across the country. There are a number of immediate consequences of this action by the company.

Probably the most dramatic impact will be that many rural customers will no longer have an option for landline broadband. While rural DSL broadband is slow, a DSL connection at speeds between 1 Mbps and 6 Mbps beats the alternatives – which is satellite broadband or cellular hotspots. Since there are a lot of rural homes where those two technologies don’t work, this means some homes will suddenly have no broadband option. Expect to soon see stories of folks who buy rural homes and then find they have no option to buy broadband.

In cities where AT&T DSL is the only alternative to a cable company broadband service, this move bestows total monopoly power to the cable company. Our firm does broadband surveys and we still find markets where AT&T DSL represents as much as a 30% market share. Many homes buy DSL because it costs less, and that option just got taken off the table in AT&T markets. And just like in rural markets, every city has customers who’s only choice is DSL. For various reasons, there are streets in most cities where the cable companies never constructed network. Any customer moving into one of these broadband deserts will find themselves with no broadband alternative.

According to an article just published by Ars Technica, only 28% of AT&T broadband customers have access to AT&T fiber – anybody living in the neighborhoods without fiber will no longer be able to buy broadband from AT&T. That has to equate to tens of millions of households that just lost a broadband option. The FCC proudly measure the number of homes with multiple broadband options, and I’ll be curious to see if they recognize this sea change in the market.

This change will stop the practice of customers who hop back and forth between DSL and cable company broadband to save money. I just talked to a customer the other day that has bounced between DSL and cable company broadband for almost twenty years. Both the cable company and the telco offer introductory prices each time for swapping, and this customer has gone back and forth between the ISPs regularly every few years. In neighborhoods where AT&T is the telco DSL provider, this might mean the end of introductory special prices from the cable company – they now have zero incentive to compete for customers.

I would have to think that Verizon will eye this announcement closely. They have openly said that they want to do away from copper network technology. This might be all of the push needed for Verizon to follow suit. This announcement might be citied in telco history as the beggining of the end of copper wires. AT&T says they won’t be tossing folks off DSL service, but will no longer connect new customers to the DSL technology. Over time this is going to mean fewer and fewer customers on copper, and I suspect AT&T already has a date in mind when they walk away from the technology completely.

Ironically, AT&T just recently announced that they were going to claim a seventh year of CAF II support in 2021 and will collect over $427 million in subsidies next year to supposedly support rural DSL. Hopefully, the FCC will view this announcement as grounds for stopping such payments. It would be absolutely insane to give millions to AT&T to support a technology that the company will no longer sell or install.

This timing of the announcement is also curious at a time when the pandemic is still raging. This means a home that needs to buy broadband to support students or adults working from home will no longer have that option if the only wired connection is AT&T DSL.

This announcement also creates an interesting dilemma for the FCC. Will the FCC pretend that the huge AT&T DSL footprint still exists? It’s impossible to pretend that areas have a broadband option when the only provider of landline service refused to connect new customers. I’m sure the FCC will act as if this announcement never happened – because recognizing it means now counting millions of homes as having no broadband option.

This day has been inevitably coming for decades. Regulators have long pretended that they could demand that the big telcos keep supporting an obsolete technology. AT&T and Verizon have been telling regulators for years that they are going to walk away from copper, and now one of the big telcos is doing so. It’s just a matter of time until AT&T begins decommissioning DSLAMs and starts tearing down copper wires for the salvage value – and I can’t see any way that regulators can stop them.

Who Has the Fastest Broadband?

Ookla recently released a report for the second quarter that summarizes its findings on speed tests conducted throughout the US. The report was generated using the results from 85.1 million speed tests taken during the quarter at the speed test site operated by Ookla. This kind of summary is always interesting, but I’m not sure how useful the results are.

The report looks at both wireless and landline speeds. Ookla says that AT&T was the fastest of the four major wireless carriers in the first quarter, with a ‘speed score’ of 41.23, with Verizon the slowest with a speed score of 30.77. The speed score is a unique metric from Ookla that weights 90% of the download speed and 10% of the upload speed. The reported speeds also toss out the slowest and fastest speeds and concentrate on the median speed.

T-Mobile had the best average latency at 31 milliseconds with Sprint the slowest at 39 milliseconds. The most interesting wireless statistic in the report is called the ‘consistency score’. This is the measure of the percentage of the traffic from each wireless carrier that was at least 5 Mbps download and 1 Mbps upload. AT&T had the highest consistency score at 79.7% with Sprint at the bottom with 66.1%. This score implies that between 20% and 35% of cellular data connections were are at speeds under 5/1 Mbps.

The landline speed results used the same criteria for summarizing the results of the many speed tests. For example, Ookla used the ‘speed score’ that uses 90% of the download speed and 10% of the upload speed – and the results also throw out the slowest and fastest speeds. Verizon had the highest speed score at 117.1, with Comcast and Cox being the only two other ISPs with speed scores over 100. Charter achieved a speed score of 95, AT&T at 82.8, and CenturyLink at 36.1. The AT&T and CenturyLink scores are lower due to customers still using DSL.

Verizon had the best latency at 9 milliseconds, which is a good indication that a large percentage of their customers are using Verizon FiOS on fiber. AT&T and Sprint had the highest latency of the big ISPs at 18 and 22 milliseconds, indicating that the two companies still have a lot of customers on DSL.

The consistency score is more of a headscratcher for the landline ISPs. For example. Spectrum and Comcast had the highest consistency ratings at over 84%, meaning that only 16% of the speed tests on these companies didn’t meet the 25/3 Mbps landline target speed. However, other than perhaps a few grandfathered customers that are still being sold slow products, these companies don’t sell products that should fail that test.

This raises the question of what speed test results mean since there are factors that likely influence the results. For example, I would guess that a lot of customers take a speed test when they are experiencing a problem. I know that’s what prompts me to take speed tests. The other issue that might make Comcast or Charter test at slower than 100 Mbps download is customer WiFi connections. It’s hard to know how many people get slow readings due to poor WiFi. I again understand this issue first-hand. I have a 3-story narrow and long house. The broadband enters on the first floor at the front of the house and my office is at the top of the rear of the house, with some thick hundred-year-old walls in between. Even with an array of WiFi repeaters, the speed in my office varies between 35 and 45 Mbps download – about one-third of the speed delivered at the router. How can Ookla understand the context of a given speed test result? Maybe it doesn’t matter since all of the ISPs have customers with WiFi issues and maybe it averages out. I would think situations like mine are what drive the consistency score. These kinds of questions make it hard to make meaningful sense out of the Ookla results in the report.

Ookla also uses the median broadband speeds to rank the 100 cities with the fastest broadband and also ranks the states. As would be expected, the states in the northeast with a lot of Verizon Fios like New Jersey, Massachusetts, and Rhode Island top the list as having the fastest average broadband speeds. More interesting to me is the bottom of the list. Ookla says that the states with the slowest median broadband are Wyoming, Montana, Idaho, and Alaska. Several other entities that rank state broadband usually put West Virginia and New Mexico at the bottom, followed by Idaho and Arkansas. Those other rankings include an assessment that there are many homes in some states with little or no broadband options at home, while a ranking using speed tests only counts home with broadband.

Overall, this is an interesting way to look at broadband. States with median download speeds under 50 Mbps (6 states) certainly have a different broadband environment than states with the median broadband speeds over 90 Mbps (11 states). But there are places in the highest-ranked states with no broadband options and places in the states with the poorest broadband that are served by fiber.

Will Congress Fund Rural Broadband?

Members of Congress seem to be competing to sponsor bills that will fund rural broadband. There are so many competing bills that it’s getting hard to keep track of them all. Hopefully, some effort will be made to consolidate the bills together into one coherent broadband funding bill.

The latest bill is the Accessible, Affordable Internet for All Act, introduced in the House of Representatives. This is part of a plan to provide $1.5 trillion of infrastructure funding that would include $100 billion for rural broadband. $80 billion of the funding would be used to directly construct rural broadband. It’s worth looking at the details of this bill since it’s similar to some of the other ideas floating around Congress.

The bill focuses on affordability. In addition to building broadband it would:

  • Require ISPs to offer an affordable service plan to every consumer
  • Provide a $50 monthly discount on internet plans for low-income households and $75 for those on tribal lands.
  • Gives a preference to networks that will offer open access to give more choice to consumers.
  • Direct the FCC to collect data on broadband prices and to make that data widely available to other Federal agencies, researchers, and public interest groups
  • Direct the Office of Internet Connectivity and Growth to conduct a biennial study to measure the extent to which cost remains a barrier to broadband adoption.
  • Provide over $1 billion to establish two new grant programs: the State Digital Equity Capacity Program, an annual grant program for states to create and implement comprehensive digital equity plans to help close gaps in broadband adoption and digital skills, and the Digital Equity Competitive Grant Program which will promote digital inclusion projects undertaken by individual organizations and local communities
  • Provide $5 billion for the rapid deployment of home internet service or mobile hotspots for students with a home Internet connection.

This bill also guarantees the right of local governments, public-private partnerships, and cooperatives to deliver broadband service – which would seemingly override the barriers in place today in 21 states that block municipal broadband and the remaining states that don’t allow electric cooperatives to be ISPs.

This and the other bills have some downsides. The biggest downside is the use of a reverse auction.  There are two big problems with reverse auctions that the FCC doesn’t seem to want to acknowledge. First, a reverse auction requires the FCC to predetermine the areas that are eligible for grants – and that means relying on their lousy data. Just this month I was working with three different rural counties where the FCC records show the entire county has good broadband because of over-reporting of speeds by a wireless ISP. In one county, a WISP claimed countywide availability of 300 Mbps broadband. In another county a WISP claimed countywide coverage of 100 Mbps symmetrical broadband coverage, when their closest transmitter was a county and several mountain ranges away. Until these kinds of mapping issues are fixed, any FCC auctions are going to leave out a lot of areas that should be eligible for grants. The people living in these areas should not suffer due to poor FCC data collection.

Second, there are not enough shovel ready projects ready to chase $80 billion in grant funding. If there is no decent ISP ready to build in a predetermined area, the funding is likely to revert to a satellite provider, like happened when Viasat was one of the largest winners in the CAF II reverse auction. The FCC also recently opened the door to allowing rural DSL into the upcoming RDOF grant – a likely giveaway to the big incumbent telcos.

This particular bill has a lot of focus on affordability, and I am a huge fan of getting broadband to everybody. But policymakers have to know that this comes at a cost. If a grant recipient is going to offer affordable prices and even lower prices for low-income households then the amount of grant funding for a given project has to be higher than what we saw with RDOF. There also has to be some kind of permanent funding in place if ISPs are to provide discounts of $50 to $75 for low-income households – that’s not sustainable out of an ISP revenue stream.

The idea of creating huge numbers of rural open-access networks is also an interesting one. The big problem with this concept is that there are many places in the country where there a few, or even no local ISPs. Is it an open-access network if only one, or even no ISPs show up to compete on a rural network?

Another problem with awarding this much money all at once is that there are not enough good construction companies to build this many broadband rural networks in a hurry. In today’s environment that kind of construction spending would superheat the market and would drive up the cost of construction labor by 30-50%. It would be just as hard to find good engineers and good construction managers in an overheated market – $80 billion is a lot of construction projects.

Don’t take my negative comments to mean I am against massive funding for rural broadband. But if we do it poorly a lot of the money might as well just be poured into a ditch. This much money used wisely could solve a giant portion of the rural broadband problem. But done poorly and many rural communities with poor broadband probably won’t get a solution. Congress has the right idea, but I hope that they don’t dictate how to disperse the money without talking first to rural industry experts, or this will be another federal program with huge amounts of wasted and poorly spent money.

The FCC Muddles the RDOF Grants

Last week the FCC ‘clarified’ the RDOF rules in a way that left most of the industry feeling less sure about how the auction will work.  The FCC is now supposedly taking a technologically neutral position on the auction. That means that the FCC has reopened the door for low-earth orbit satellites. Strangely, Chairman Ajit Pai said that the rules would even allow DSL or fixed wireless providers to participate in the gigabit speed tier.

Technologically neutral may sound like a fair idea, but in this case it’s absurd. The idea that DSL or fixed wireless could deliver gigabit speeds is so far outside the realm of physics as to be laughable. It’s more likely that these changes are aimed at allowing the providers of satellite, DSL, and fixed wireless providers to enter the auction at speeds faster than they can deliver.

For example, by saying that DSL can enter the auction at a gigabit, it might go more unnoticed if telcos enter the auction at the 100./10 Mbps tier. There is zero chance for rural DSL to reach those speeds – the CAF II awards six years ago didn’t result in a lot of rural DSL that is delivering even 10/1 Mbps. It’s worth remember that the RDOF funding is going to some of the most remote Census blocks in the country where homes are likely many miles from a DSL hub and also not concentrated in pockets – two factors that account for why rural DSL often has speeds that are not a lot faster than dial-up.

Any decision to allow low orbit satellites into the auction has to be political. There are members of Congress now pushing for satellite broadband. In my State of North Carolina there is even a bill in the Senate (SB 1228) that would provide $2.5 million to satellite broadband as a preferred solution for rural broadband.

The politics behind low orbit satellite broadband is crazy because there is not yet any such technology that can deliver broadband to people. Elon Musk’s satellite company currently has 362 satellites in orbit. That may sound impressive, but a functional array of satellites is going to require thousands of satellites – the company’s filed plan with the FCC calls for 4,000 satellites as the first phase deployment.

I’ve seen a lot of speculation in the financial and space press that Starlink will have a lot of challenge in raising the money needed to finish the constellation of satellites. A lot of the companies that were going to invest are now reluctant due to COVID-19. The other current competitor to Starlink is OneWeb, which went bankrupt a few months ago and may never come out of receivership. Jeff Bezos has been rumored to be launching a satellite business but still has not launched a single satellite.

The danger of letting these various technologies into the RDOF process is that a lot of rural households might again get screwed by the FCC and not get broadband after a giant FCC grant. That’s what happened with CAF II where over $9 billion was handed to the big telcos and was effectively washed down the drain in terms of any lasting benefits to rural broadband.

It’s not hard to envision Elon Musk and Starlink winning a lot of money in the CAF II auction and then failing to complete the business plan. The company has an automatic advantage over any company they are bidding against since Starlink can bid lower than any other bidder and still be ahead of the game. It’s not an implausible scenario to foresee Starlink winning every contested Census block.

Allowing DSL and fixed wireless providers to overstate their technical capacity will be just as damaging. Does anybody think that if Frontier wins money in this auction that they will do much more than pocket it straight to the bottom line? Rural America is badly harmed if a carriers wins and the RDOF money and doesn’t deliver the technology that was promised – particularly if that grant winner unfairly beat out somebody that would have delivered a faster technology. One has to only look back at the awards made to Viasat in the CAF II reverse auction to see how absurd it is when inferior technologies are allowed in the auction.

Probably the worst thing about the RDOF rules is that somebody who doesn’t deliver doesn’t have to give back all of the grant money. Even should no customer ever be served or if no customer ever receives the promised speeds, the grant winner gets to keep a substantial percentage of the grant funding.

As usual, this FCC is hiding their real intentions under the technology neutral stance. This auction doesn’t need the FCC to be ‘technology neutral’, and technologies that don’t exist yet today like LEO satellites or technologies that can’t deliver the speed tiers should not be allowed into the auction. I’m already cringing at the vision of a lot of grant winners that have no business getting a government subsidy at a time when COVID-19 has magnified the need for better rural broadband.

The Coming Year of Confusion

July 2020 Calendar

I’ve had a number of people ask me about how I think COVID-19 will impact the ISP industry over the next six months. It’s an interesting question to consider because there are both positive and negative trends that ISPs need to be concerned about. The chances are that these various trends will affect markets and ISPs differently – making it that much harder for an individual ISP to understand what they are going to see over the next six months. Following are some of the trends I think ISPs will need to deal with:

People Want Faster Broadband.  Many households came to the realization that their home broadband is inadequate when parents and students tried to work from home simultaneously. OpenVault reported that the number of households subscribing to gigabit service nearly doubled in the first quarter of this year. Clients are reporting an increased demand from first-time customers as well as customers wanting to upgrade to faster speeds.

Downturn in Small Businesses. Everything seems to indicate that a lot of small and medium businesses are not going to survive the pandemic. There have already been a number of businesses like restaurants and small retail stores that have gone under. The anchor stores at malls are failing right and left. There seems to be an expectation that travel-related businesses are going to take a long time to come back. Everything I read says that there is a coming crisis in the fall for business landlords when the finally digest that business tenants are either disappearing or will want to negotiate cheaper rent. That’s likely to have a secondary ripple effect as strip malls and other business landlords start declaring bankruptcy. Over time, new businesses will grow to fill many of the voids, but there has been a huge shift to shopping online that will likely not retreat to pre-COVID levels.

People Will Continue to Work from Home. Every day I read about businesses that say that working from home, at least part time, will become the new normal in many industries. The latest was a survey of law firms that said that a lot of lawyers are not going to return to the office full time when the pandemic is over. This is good news for ISPs that provide residential broadband, because people working from home are going to demand speeds and latency that will support their work. OpenVault just reported that as of the end of the first quarter of 2020 that the percentage of homes subscribing to gigabit broadband doubled over the last year and is now at 3.75% of all homes and growing rapidly. This is not such great news for ISPs that serve law offices.

The Big Unknown is the Impact of Unemployment. As businesses fail or downsize a lot of people are not going to be returning to their original job. ISPs are already reporting that people are ditching telecom products like landlines. The cord cutting in the last month of the first quarter of this year was record-setting. The big unknown will be the number of households that can no longer afford to buy landline broadband. Obviously, unemployment isn’t going to stay at the current 40 million people, but it’s not quickly going to return to pre-COVID levels. A secondary impact of a degraded economy will be a surge in bad debt as customers hang onto to home broadband as long as they can. We’re likely to see a big impact when the Keep America Connected pledge ends. If ISPs present a bill for multiple back months of billing we ought to see a lot of customers forced to default and cancel broadband.

The Pandemic is the Dagger That Will Finally Kill DSL. Homes that have an option of using DSL or something faster like cable broadband or fiber are going to be bailing on DSL in big numbers. Many people in towns have stuck with DSL because it is priced cheaper than cable broadband. However, for a lot of homes, the most important factor in broadband has become speed and performance.

The Rural Broadband Gap Will Keep Getting Headlines. COVID-19 made it clear to elected officials at all levels of government that the rural broadband gap is badly hurting the economy. Even if schools return to normal, businesses in rural areas are not going to have the same flexibility to send employees home, and unemployed people in rural area are not going to easily be able to accept at-home jobs. That’s going to keep a sizable slice of the economy from fully participating in any recovery. Almost everybody I talk to is hopeful that this might translate to increased grant money for rural broadband – but that’s no guarantee.

We’re Going to Have Unexpected Shortages in the Supply Chain. The best way to describe the supply chain right now is spotty. Manufacturers of telecom electronics are going to suddenly find they can’t buy one or two components, and manufacturing will come to unexpected halts. Anybody building a broadband network needs to expect delays, and if history is a good teacher, the delays will last longer than expected. This is going to play havoc with anybody that has financed a new network and needs to install customers to meet debt payments.

Banks Are Going to Tighten Lending. It’s inevitable that as banks digest bad loans from failing businesses that they are going to get more cautious about making new loans. Even if interest rates don’t rise, banks will do what they always do under stress and get more conservative. Some local banks are likely to get into real trouble and will fail if their portfolio was heavily invested  into businesses that are failing.

This all makes for an interesting short-term future. There will be more people yelling for faster broadband at the same time there will be more customers unable to afford broadband. There will be grants awarded for rural markets at a time when banks might not provide the matching funds. All in all, it’s going to be a mess for most ISPs who will see both good and bad things affecting them at the same time.

 

 

 

 

Verizon’s Network Performance

Verizon has been posting a weekly report of how COVID-19 has been impacting their network. The weekly blogs are rather short on facts and it’s clear that the intent of this weekly report is to put investors at ease that the company’s networks are coping with the burst of traffic that has come as a result of the pandemic. With that said, the facts that are discussed are interesting.

Verizon lead off the weekly entry for 5/21 saying that voice and text traffic are starting to return to pre-COVID levels. On the most recent Monday Verizon saw 776 million voice calls, down from 860 million calls at the peak of COVID-19. That falls under the category of interesting fact, but heavier telephone call volumes are not the cause of undue stress on the Verizon network. Telephone calls use tiny amounts of broadband – 64 kbps. Thirty telephone calls will fit into the same-size data path as one Netflix stream. Additionally, once voice calls reach a Verizon hub, telephone calls are routed using a separate public switched telephone network PSTN to transport calls across the country. Text messages use much less data than a telephone call and are barely noticed on telco networks.

The bigger news is that some other traffic is staying at elevated levels. Verizon reported for 5/21 that gaming is still up 82% over pre-pandemic levels and VPN connections used to connect to school and work servers are up 72%. The use of collaborative tools like Zoom and Go-to-Meeting are up ten times over pre-COVID levels (1,000%).

One of the more interesting statistics is that network mobility (people driving or walking and switching between cell towers) has increased in recent weeks and that one-third of states now have higher levels of mobility than pre-COVID. At first that’s a little hard to believe until you realize that in pre-COVID time students and employees were largely stationary at the school or office much of the day – any roaming by stay-at-home people is an increase.

Reading back through the weekly statistics shows that most web activities are at higher levels than pre-COVID. Fir example, in the 4/22 report the volumes of downloading, gaming, video usage, VPNs, and overall web traffic were higher than normal, with the only decrease being the volumes used for social media.

What none of these reports talk about is the stress put on the Verizon networks. It’s easy in reading these reports to forget that Verizon wears many hats and operates many networks. They are still a regulated telco in the northeast and still have a lot of telephone customers. That also means they still operate a sizable DSL network. The company, through Verizon FiOS is still the largest fiber-to-the-home provider. The company also owns and extensive enterprise and long-haul fiber network. Verizon also operates one of the largest cellular networks in the world.

When Verizon says all is well, they can’t mean that for each of these networks. The web is full right now of complaints from DSL customers (Verizon’s and other big telcos) complaining how inadequate DSL is for working at home. The Verizon DSL network was already overstressed in evenings and has to be near the point of collapse due to the big increases in VPNs and collaboration connections. Any Verizon DSL customer reading this Verizon blog that says everything is fine is probably spitting fire.

By contrast, Verizon’s FiOS networks are likely handing the pandemic traffic with ease. Verizon FTTH products have offered symmetrical data for years, with the upload data path was lightly utilized. The big uptick in VPN connections and collaboration connections ought to be handled well in that network. Any glitches might come from older FiOS neighborhoods where the backhaul oaths out of neighborhoods are too small.

What Verizon or AT&T haven’t talked about is the different impact on their various networks. For example, what’s the overall change in data usage on their cellular networks compared to other networks? The big telcos have been moot on this kind of detail, because admitting that some of the networks are handing the pandemic well might lead to an admission that other parts of the company are not doing so well. Instead we get the very generic story of how everything is fine with the company and their networks.

These companies probably do not have any obligations to report about their various networks in detail. Verizon DSL customers don’t need company pronouncements to know that their broadband experience has nearly collapsed since the pandemic. FiOS customers are likely happy that their broadband has weathered the storm. One of these days I’ll hopefully have a beer with some Verizon engineer who can tell me what really happened – both good and bad – behind the scenes.

Frontier’s Lack of Fiber

The primary reason that Frontier cites for going into bankruptcy is the lack of fiber. They are finally acknowledging that customers are bailing on them due to the poor broadband speeds on their copper networks. This is being presented as if this is a sudden revelation – as if the company woke up one day and realized that it’s selling services that nobody wants to buy. I must admit this gave me a chuckle and there are some giant flaws with this argument.

Rural customers don’t hate DSL – they hate DSL that doesn’t work. If Frontier had implemented the CAF II upgrades as had been promised, then rural customers would all be using the 10/1 Mbps or faster rural DSL that would have been created as a result of those upgrades. Instead, customers have gotten disgusted by overpriced DSL that is so slow that they can’t stream video or connect to a school or work server. We’ve been doing speed tests all over the country and it’s rare to find rural DSL in many markets that delivers even 5 Mbps download – much of it is far slower than that, some barely faster than dial-up. If Frontier had provided 10/1 Mbps DSL to millions of homes, those households would gratefully be buying that broadband during the COVID-19 crisis.

Frontier blames its woes on lack of fiber with no mention of their reputation for unconscionably bad customer service. I’ve talked to customers who talk about routine network outages that lasts for many days. Customers complain about losing broadband and having to wait weeks to get it repaired – or worse, are told that the electronics needed to replace a bad DSL modem are out of stock. This is a company that has trimmed, then trimmed again its maintenance staff to the bone. Talk to any rural Frontier technician and they’ll tell you that they don’t have the time or resources available to address routine customer problems.

Frontier complains about lack of fiber, but as recently as 2015 they purchased another huge pile or dilapidated Verizon copper networks as part of a $10.5 billion acquisition. While that acquisition came with some FiOS fiber networks, the company also doubled down on buying non-functional copper networks. The speculation in the industry was that Frontier continued to buy lousy properties because it created opportunities for huge management bonuses – the company never had any plans to make the purchased copper networks any better.

And that’s the real issue with Frontier’s claim – they have no fiber because they’ve made almost no effort to migrate to fiber. The company burned all of its cash on trying to service the debt for overpriced acquisitions rather than rolling cash back into its networks.

It’s interesting to compare Frontier to the many smaller independent telephone companies. The FCC brags about places like the Dakotas that have a huge amount of rural fiber to homes. But that rural fiber didn’t happen all at once. It happened over decades. Most rural telcos went through two rounds of investment where they invested to improve rural DSL. In doing so they built fiber to go deeper into the rural areas, the first build brought fiber within maybe ten miles of homes, the second got fiber to within 3 miles of most homes. When the rural telcos decided to take fiber the rest of the way, it was reasonably achievable because they already had fiber deep into rural neighborhoods.

Frontier has done very little of that kind of incremental improvements over the years. They found it more enticing to keep borrowing to buy new rural properties rather than roll cash back into the existing networks. It doesn’t even look like they did all of that much new fiber as part of the CAF II upgrades. I’m sure Frontier would refute that statement and say they are fully compliant with CAF II, but if they had built fiber deep into the network then rural DSL would have gotten better – and for the most part, it hasn’t.

I can’t how the bankruptcy will benefit frontier’s customers. The company will likely get to walk away from a lot of the debt that was provided for the last few acquisitions – and it’s hard to feel bad for lenders who thought it was a good idea in 2015 to lend to buy copper networks. But bankruptcy won’t fix any of the fundamental problems with the Frontier networks. Customers are going to continue to bail on inferior and nonfunctional broadband products. The upcoming RDOF auction is going to give a lot of money to ISPs that are going to overbuild Frontier copper with something better (even though Frontier made a last-minute filing at the FCC to block grant funding by claiming they had magically upgraded 16,000 rural census blocks).

Is Frontier going to somehow start investing in rural fiber? My best guess is that they won’t even after bankruptcy. If they can raise any money for new capital spending they’ll likely try to salvage some of the county seats and other markets where there is a mass of customers. However, in many of those markets they’ve already lost the battle to the cable companies.

Frontier is right in that they are failing from lack of fiber. But that statement doesn’t tell the full story. They are failing because the company decided decades ago to not invest capital into their own networks – and now they are paying the price.

Enough is Enough

CenturyLink recently petitioned the FCC to allow them to be late in implementing the CAF II upgrades where the FCC doled out $11 billion to upgrade rural broadband speeds to 10/1 Mbps. The ostensible reason for the delay is the COVID-19 pandemic, but CenturyLink was already behind and notified the FCC earlier this year that they hadn’t completed their 2019 CAF II installation in 23 out of 33 states.

I say enough is enough. It’s time for the FCC to demand a reckoning of CAF II and begin handing out draconian penalties to the telcos that didn’t meet their obligations. I’m positive that if this was assessed fairly that the FCC will find that the vast majority of big telco customers have never gotten an upgrade to 10/1 Mbps.

Let’s start by looking at CenturyLink’s request. There is no reasonable explanation they can offer for not meeting their obligations in 2019. That was the fourth of a five-year buildout obligation, and the company has known for years what’s needed to be done – and they had the federal money in their pocket to make the upgrades. The claim for this year is also largely bogus. I have a lot of clients that are being cautious now about entering customer premises, but I don’t know any carrier that has stopped doing work outside of customer homes. I can’t think of any practical reason that COVID-19 would cause a delay for CenturyLink. Even if they upgrade somebody’s DSL, they could mail them a new modem – telcos have been having customers self-install DSL modems for twenty years.

It’s time to stop the pretense that CenturyLink or the other big telcos have been busy upgrading rural DSL. I don’t know anybody who thinks that’s happened. I have anecdotal evidence that it hasn’t, My company has been helping rural counties with broadband feasibility studies for many years. In the last four years, we’ve been asking rural customers to take speed tests – and I’ve never seen even one rural DSL connection that transmits at a speed of 10/1 Mbps. I’ve haven’t seen many that have tested above 5 Mbps. I’ve seen a whole lot that tested at less than 3, 2 or even 1 Mbps. Many of these tests have been in areas that are supposed to have CAF II upgrades.

I’ve also never talked to any County officials who have heard from the telcos that their county got rural broadband upgrades. One would think the telcos would brag locally when they were finished with upgrades as a pitch to get new customers. After all, customers that have only had slow DSL or satellite service should be flocking to 10/1 DSL. I’ve also not seen a marketing campaign talking about faster speeds due to CAF II. I’ve been searching the web for years to find testimonials from customers talking about their free upgrade to 10/1 Mbps, but I’ve never found anybody who has ever said that. This is not to say there have been zero upgrades in the CAF II areas, but I see no evidence of widespread upgrades.

The reality is that CenturyLink got new leadership a few years ago who immediately announced that the company was going to stop making ‘infrastructure return’ investments. We have Frontier that miraculously recently found 16,000 Census blocks that now have speeds of at least 25/3 Mbps when I’m still looking for proof that they upgraded places to 10/1 Mbps. Go interview folks in West Virginia if you think they’ve made any CAF II upgrades.

The FCC has a choice now. They can wimp out and grant the delay that CenturyLink is requesting, or the agency can come down on the side of rural broadband. There is no middle ground when it comes to CAF II. This FCC didn’t make the original CAF II decision – but they are the ones that are supposed to make sure the upgrades are done, and they are supposed to be penalizing telcos that failed to make the upgrades.

The response to CenturyLink’s request should be a giant penalty for missing the 2019 deadlines and a reminder that the company is still on the hook for 2020 unless they want more fines.

The FCC also needs to aggressively start testing in the areas that have supposedly gotten CAF II upgrades. This doesn’t have to be a big expensive testing program. We know exactly where CAF II should have been implemented – the FCC has made it easy by overlaying the CAF II footprint over Google maps. The FCC could ask County administrators across the US to solicit a speed test at CAF II locations – the Counties would be glad to oblige. If the FCC wanted to know the truth about CAF II they could get massive feedback within a few weeks about the abject failure of the CAF II program.

The ultimate penalty ought to be the return of CAF II money to the Universal Service Fund for areas that aren’t upgraded to 10/1 Mbps. Then the money could finally be given to somebody that will upgrade to real broadband. The CAF II program was ill-conceived, but the big telcos should have used that money to bring rural speeds up to 10/1 Mbps. Had they done so, we’d have millions of more homes that wouldn’t be struggling so hard during COVID-19. This FCC has a chance to do their job and set things right.

The Dirty Secret of Coaxial Broadband

The US has clearly pinned our hopes for providing modern broadband on the big cable companies. At the end of 2019, the big cable companies had almost 68 million customers compared to 33 million for the big telcos. Any discussion of broadband in urban markets is mostly a discussion of big cable company broadband. Cable companies will continue to grow market dominance as urban DSL customers continue to migrate to cable modem. In 2019 the big cable companies added 3.1 million customers while the telcos lost over 600,000 customers.

The big cable companies have all advertised to their customers that they had upgraded to the latest technology in DOCSIS 3.1 and can now provide gigabit broadband – for an expensive price in most markets set well over $100 per month.

It’s easy to think of urban cable systems as up-to-date and high tech and ready and able to deliver fast broadband speeds. While this is true in some cities and in some neighborhoods, the dirty secret of the cable industry is that their networks are not all up to snuff. Everybody is aware of the aging problems that have plagued the telephone copper network – but it’s rare to hear somebody talking about the aging of the cable company copper networks.

Most of the cable networks were built in the 1970s, with some even a little older. Just like with telephone copper networks the coaxial networks are getting old and a network built around 1970 is now fifty years old.

Cable coaxial networks suffer more from deterioration than do telephone copper networks. The copper wires in a coaxial system are much larger and the wires hanging on poles act like a giant antenna that can receive a range of different frequencies. Any physical opening into the wire through a splice point or from aging creates a new ingress point for external frequencies – and that equates to noise on the coaxial network. Increased noise translates directly to decreased performance of the network. The capacity of the older coaxial networks is significantly lower than when the networks were first constructed.

Another issue with coaxial networks is that the type of coaxial cable used has changed over time and some of the coax used in the early networks can’t handle the capacity needed today. Some older coax has been replaced in urban networks, but not all. Coaxial networks in smaller towns still can contain a lot of older-generation coaxial cables.

These issues mean that coaxial networks don’t always perform as well as is touted by the cable companies. I can use the network in my city of Asheville NC as an example. Charter announced nationally that when it upgraded to DOCSIS 3.1 that it had a goal of raising broadband speeds everywhere to 200 Mbps. My speed at the modem is 135 Mbps. I’m not complaining about my speed and I’m glad they increased my speed, but there must be issues in the local network that stopped Charter from achieving its 200 Mbps goal.

We undertake surveys and citywide speed tests across the country and we often see that the performance of coaxial networks varies by neighborhood. We’ve seen neighborhoods where there are more outages, more variance in download speeds, and overall slower speeds than the rest of the city. These problems are almost certainly due to differences within a city of the quality of the coaxial network.

Cable companies could bring older neighborhoods up to snuff, but such upgrades are expensive. It might mean replacing a lot of drops and any runs of older coaxial cable. It might mean replacing or re-spacing amplifiers. It often means replacing all of the power taps (the devices that connect homes to the distribution cables). The upgrading effort is labor-intensive, and that means costly.

I think this means that many cities will never see another unilateral increase in broadband speeds unless the cable companies first make big investments. The cable companies have increased speeds every few years since 2000 to keep ahead of the telcos and to make customers happier with their service. I fear that since cable companies are becoming de facto monopolies in most cities that they have lost the incentive to get faster if that means spending money. The coaxial networks and speeds that we have in place today might be what we still have a decade from now, only with coaxial networks that are another ten years older.