Big Internet Outages

Last year I wrote about big disruptive outages on the T-Mobile and the CenturyLink networks. Those outages demonstrate how a single circuit failure on a transport route or a single software error in a data center can spread quickly and cause big outages. I join a lot of the industry in blaming the spread of these outages on the concentration and centralization of networks where the nationwide routing of big networks is now controlled by only a handful of technicians in a few locations.

In early October, we saw the granddaddy of all network outages when Facebook, WhatsApp, and Instagram all crashed for much of a day. This was a colossal crash because the Facebook apps have billions of users worldwide. It’s easy to think of Facebook as just a social media company, but the app of suites is far more than that. Much of the third world uses WhatsApp instead of text messaging to communicate. Small businesses all over the world communicate with customers through Facebook and WhatsApp. A Facebook crash also affected many other apps. Anybody who automatically logs into other apps using the Facebook login credentials was also locked out since Facebook couldn’t verify their credentials.

Facebook blamed the outage on what it called routine software maintenance. I had to laugh the second I saw that announcement and the word ‘routine’. Facebook would have been well advised to have hired a few grizzled telecom technicians when it set up its data centers. We learned in the telecom industry many decades ago that there is no such thing as a routine software upgrade.

The telecom industry has long been at the mercy of telecom vendors that rush hardware and software into the real world without fully testing it. An ISP comes to expect to have issues in glitches when it is taking part in a technology beta test. But during the heyday of the telecom industry throughout the 80s, and 90s, practically every system small telcos operated was in beta test mode. Technology was changing quickly, and vendors rushed new and approved features onto the market without first testing them in real-life networks. The telcos and their end-user customers were the guinea pigs for vendor testing.

I feel bad for the Facebook technician who introduced the software problem that crashed the network. But I can’t blame him for making a mistake – I blame Facebook for not having basic protocols in place that would have made it impossible for the technician to crash the network.

I bet that Facebook has world-class physical security in its data centers. I’m sure the company has redundant fiber transport, layers of physical security to keep out intruders, and fire suppression systems to limit the damage if something goes wrong. But Facebook didn’t learn the basic Telecom 101 lesson that any general manager of a small telco or cable company could have told them. The biggest danger to your network is not from physical damage – that happens only rarely. The biggest danger is from software upgrades.

We learned in the telecom industry to never trust vendor software upgrades. Instead, we implemented protocols where we created a test lab to test each software upgrade on a tiny piece of the network before inflicting a faulty upgrade on the whole customer base. (The even better lesson most of us learned was to let the telcos with the smartest technicians in the state tackle the upgrade first before the rest of us considered it).

Shame on Facebook for having a network where a technician can implement a software change directly without first testing it and verifying it a dozen times. It was inevitable that a process without a prudent upgrade and testing process would eventually result in the big crash we saw. It’s not too late for Facebook – there are still a few telco old-timers around who could teach them to do this right.

The Fixation on Speed

I was recently asked an interesting question. Is a 100/20 Mbps broadband connection really better than a 50/50 Mbps one? The question was referring to the new ReConnect grant rules that say that companies can seek grants to overbuild existing networks that are not performing at 100/20 Mbps. I have several different reactions to that question.

My first reaction is to ask an additional question. Do you think the RUS would provide a grant to overbuild a 50/50 Mbps WISP with 100/20 Mbps technology? In this example, the WISP doesn’t meet the RUS download speed threshold but is faster than the upload threshold. The answer is not clear to me. The definition of speed used for the grant has a download and upload component – what if an existing product only fails one of the two tests? I’ve never been a fan of using speeds as the definition of what is eligible for grants, and this kind of dilemma is one of many reasons what picking an arbitrary speed is likely to create controversies.

I next asked myself about how a household would feel about a 50/50 Mbps versus a 100/20 Mbps broadband product. I know some households who struggled with upload speeds during the pandemic even though they had 15 – 20 Mbps upload speed provided by a cable company. A home that struggled with a cable company upload connection likely had multiple students and adults trying to use upload at the same time. But the reason such a household struggled was more complex than just the speed. The upload path in cable company networks uses the worst spectrum inside the cable transmission path. The upload path often has a lot of noise and jitter – so even though a connection might show 20 Mbps on a speed test, the quality of the transmission can be highly compromised. If this theoretical home was fully informed it would choose a product with a faster upload link and should favor the 50/50 Mbps product. But there are plenty of homes with gamers and other heavy bandwidth users who care more about the download speed.

We also can’t forget about latency. A 50/50 Mbps connection on fiber might ‘feel’ faster to a customer than a 100/20 Mbps on a fixed wireless network with higher latency. The human eye is amazing at perceiving a slight difference in latency – fiber connections have an immediacy for the eye that is perceptible. I’ve always thought it would be interesting to set up a lab test where you could give people feeds with different speeds and latencies to measure the perceived differences from a customer perspective. I’ve always guessed that people care about latency a lot more than we think they do – but few customers understand this. I know that when customers first get fiber, they believe that it’s faster, even if they converted from a 100 Mbps download cable connection to a 100 Mbps fiber connection.

But back to the grant question – I don’t want to see federal grants used to build 100/20 Mbps broadband. If we are going to spend once-in-a-lifetime federal grant money we should be building networks that will be adequate a decade from now. If we don’t build for the future, then in ten years, we start the cycle all over again of talking about how to improve rural networks. There are plenty of arguments to be made why 100/20 Mbps is a good broadband connection today. But I defy anybody to say that it will be adequate a decade from now. A decade is a long time in the broadband world. A decade ago, the cable companies offered 30/3 Mbps speeds, and most people were happy with it. A decade from now a 100/20 Mbps connection will feel as inadequate as a 30/3 Mbps connection today.

Choosing between 100/20 Mbps and 50/50 Mbps today is an interesting thought exercise. There are some homes that should prefer each choice, so what matters most is the buyer of the broadband. I also think the answer is about more than just speed, and we must consider speed, jitter, and latency to fully compare two broadband options. But perhaps the most important thing to consider is that if a household buys a 100/20 Mbps connection today that they will likely be unhappy with that same product in a decade. It doesn’t take a crystal ball to understand this.

Being an ISP

Over time, this blog has talked about everything broadband, but I don’t think I’ve ever talked about being an ISP. In the simplest terms, an ISP is somebody that connects to a home or business and routes broadband traffic to and from the Internet. ISPs do a lot more these days. For example, they protect customers against hacking and bad behavior on the web.

We all know the big ISPs like Comcast, Charter, AT&T, and Verizon since these four ISPs serve over 75% of all broadband customers in the country. All of the other ISPs you hear about collectively serve the other one-fourth of the U.S. market.

The heyday of the ISP industry, in terms of the total number of ISPs, was probably in the late 1990s when anybody could be a dial-up ISP by buying a modem bank, some telephone lines, and a connection to the Internet. It seems like every small town and even many neighborhoods had one or more ISPs who competed with the few big nationwide players like AOL and CompuServe.

ISPs come in every shape and size. The ones we most think about as ISPs own networks that reach people’s homes, either through wires or wirelessly. Satellite companies like Viasat and Starlink are ISPs. But there are other kinds of ISPs. For example, some ISPs lease fiber connections from a city or somebody else that owns a network. There are still some ISPs delivering broadband over leased telco copper wires. A lot of people don’t think of cellular carriers as ISPs, but most people today have smartphones and connect to the Internet using apps. In many of the surveys we conduct, we see that as many as 10% of households only connect to the Internet over a cellular connection.

ISPs are somewhat regulated, but it gets complicated. The FCC under Ajit Pai largely deregulated broadband by wiping out the FCC’s Title II authority to regulate ISPs except for a handful of regulations specifically required by Congress. In doing so, Chairman Pai constantly referred to his deregulation as light-touch regulation, but the FCC eliminated 90% of the ways that the agency might theoretically be able to regulate ISPs. Consequently, the current FCC has very little regulatory authority over ISPs.

This doesn’t mean that ISPs are fully unregulated. ISPs are supposed to comply with a few regulations. For example, they are supposed to register with the FBI and describe the steps needed if the FBI wants to surveil a customer on an ISP network. An ISP has to officially register with the FCC if it wants to participate in receiving any funding from the Universal Service Fund. Many states expect ISPs to register as carriers – mostly, so the state knows who they are.

The FCC requires ISPs to use the Form 477 process to report the location of customers by Census Block, along with a description of the technology being used and the speeds delivered. But broadband regulation is taken so lightly that a lot of ISPs ignore this completely. For example, in almost every county I’ve ever worked in, there is a least one ISP that doesn’t report customers to the FCC. There doesn’t seem to be any penalty for not reporting or at least any that I’ve ever seen. Some of the ISPs that skirt regulation are sizable and sell fiber connections to large businesses in multiple markets.

ISPs are also theoretically regulated by the Federal Trade Commission. But that is truly light regulation because the FTC can’t easily establish rules or policies that affect all ISPs. Instead, the agency occasionally punishes a specific ISP for bad behavior, mostly centered on mistreating customers in some manner.

There are a lot of entities that don’t even realize they are ISPs. Governments often build fiber networks to connect various government buildings into a local network. But when cities then connect all of the government locations to the Internet, they have become an ISP. Cities also often branch out and provide a fiber connection to a few large businesses in a community – often without realizing this makes them an ISP like any other.

The big ISP industry believes that broadband regulation will be coming back when the FCC finally gets a fifth Commissioner. Companies with monopoly powers in all industries would love to be unregulated, and so far, the only two groups of companies that have largely been able to pull this off are ISPs and the giant web content companies. The need for some regulatory oversight is obvious. For example, the FCC is currently investigating the response efforts of big ISPs after a major storm. But without explicit regulatory authority, I’m not sure the agency has any authority to compel ISPs to do more to be ready for disaster recovery.

Being Serious about Local Resiliency

The FCC recently voted unanimously to adopt a Notice of Proposed Rulemaking to investigate the disaster resiliency plans of major telecom providers. The FCC noted that it has been taking too long after recent natural disasters to restore cellular and broadband services and wants to know if there are any steps the agency can take to improve recovery times after network outages.

As part of the investigation, the FCC wants to hear more about carriers’ existing recovery plans, and hopefully, the FCC will compare the actual behavior after recent disasters with the formal recovery plans. The FCC also wants to examine policies such as requiring more backup power.

One of the things I’ve noticed through the years in the industry is that small ISPs plan for disasters better than large ones. When I visit a small telco that sits in a known flood plain, I expect to find huts built on stilts to survive floods. I expect to find huts and buildings in hurricane zones built to withstand hurricane winds. I expect to find a robust system of backup generators, including many portable ones that can be deployed quickly when needed. If I go down the road a few miles to an area served by a big telco, I don’t expect to find any of these things.

Disaster recovery preparedness is not cheap. It costs more to build huts on stilts or to build a fortress for a hut to withstand storms – but in the long run, the prudent solution is also the lowest cost one. It costs far more to try to restore service after a big disaster. Big carriers will always say that resiliency is important, but you’ll have to dig hard to find examples of where they spent the extra money up-front to build things right.

Small companies are also more prepared in other ways. For example, small telcos, and electric coops both have formed pacts for aiding each other after disasters. When a bad storm hits a small company there is generally a swarm of technicians from around the country who converge to begin making repairs within hours after the end of the storm. Big companies bring in outside help as well, but not with the immediacy and vigor that I’ve witnessed many times with smaller companies.

Disaster recovery also means a lot of more subtle ways to be ready for trouble. Something as simple as having sufficient spare circuit cards, or a yard full of spare fiber and poles, or plenty of fuel for vehicles can make a huge difference in restoring service quickly.

Resiliency means a lot more than disaster recovery. Resiliency means planning ahead so that disasters don’t knock out service. Being prepared might mean a lot of little things like placing metal poles at key network intersections. It means designing fiber rings with automatic rollover that don’t lose service from a single fiber cut. It means burying fiber in places where damage can be expected, even if it costs more. Resiliency means adhering to a tree-trimming program.

If the FCC investigates the issue in the normal way, it will hear from a string of witnesses from big carriers that all swear their companies take resiliency seriously and who vow that it is a priority. Each big carrier will have a four-volume disaster recovery manual that looks impressive. But what the big carriers don’t have is a track record of recovering from disasters quickly – which is why the FCC created this docket.

I have a suggestion for the FCC if they want to do this right. Bring in frontline workers from smaller telcos, cellular companies, and electric cooperatives and have them examine the disaster and resiliency plans from the big carriers. Let these frontline workers visit the huts and pole yards of the big carriers. I’m sure they would find dozens of holes in the big company plans.

Unfortunately, I don’t think the FCC can force big companies to do the right thing. the FCC might be able to require a few things like requiring more backup generators. But the FCC can’t pass rules that are detailed enough to make sure that the big carriers do all of the little subtle things right. That only comes from companies that care for their customers as much as they care for the bottom line.

Broadband Adoption Grants

The recently enacted Infrastructure Investment and Jobs Act (IIJA) created two new grant programs to address digital equity and inclusion. This section of the IIJA recognizes that providing broadband access alone will not close the digital divide. There are millions of homes that lack computers and the digital skills needed to use broadband. The grant programs take two different approaches to try to close the digital divide.

The State Digital Equity Capacity Grant Program will give money to States to then distribute through grants. The stated goal of this grant program is to promote the achievement of digital equity, support digital inclusion activities, and build capacity for efforts by States relating to the adoption of broadband. I haven’t heard an acronym for this grant program – it’s likely that each state will come up with a name for the state program.

The Act allocates $1.5 billion to the States for this program – that’s $300 million per year from 2022 through 2026. Before getting any funding, each state must submit a plan to the NTIA on how it plans to use the funding. States will have to name the entity that will operate the program, and interestingly, it doesn’t have to be a branch of government. States could assign the role to a non-profit or other entity.

The amount of funding that will go to each state is formulaic. 50% will be awarded based upon the population of each state according to the 2020 Census. 25% will be awarded based upon the number of homes that have household incomes that are less than 150% of the poverty level, as defined by the U.S. Census. The final 25% will come from the comparative lack of broadband adoption as measured by the FCC 477 process, the American Community Survey conducted by the U.S. Census, and the NTIA Internet Use Survey.

The second new grant program is called the Digital Equity Competitive Grant Program. These are grants that will be administered by the NTIA and awarded directly to grant recipients. The budget for this grant program is $1.25 billion, with $250 million per year to be awarded in 2022 through 2026.

These grants can be awarded to a wide range of entities, including government entities, Indian Tribes, non-profit foundations and corporations, community anchor institutions, education agencies, entities that engage in workforce development, or a partnership between any of the above entities.

This will be a competitive grant program, with the rules to be developed by the NTIA. While the broadband infrastructure grants in the Act include a long list of proscribed rules, Congress is largely letting it up the NTIA to determine how to structure this grant program.

That’s going to make for some interesting choices for entities involved in digital inclusion. They can go after funding through the state or compete for nationwide grants. I doubt that anybody can make that decision until we see the specific grant rules coming out of each program.

I’ve been hearing about digital inclusion at every conference I’ve attended for the last fifteen years. For many years we talked about this as finding ways to solve the digital divide. We’ve known for all these years that there are homes that don’t have broadband because they can’t afford a broadband connection. We’ve known that homes can’t afford computers or other devices. And we’ve known for a long time that a lot of people don’t have the digital skills needed to use broadband.

There have been efforts over the years to address the issues, mostly done at the local level and mostly through non-profits. This is the first time that real funding is being aimed at solving these issues. It’s going to be interesting to see what comes out of this funding. I’m sure there will be some dazzlingly successful programs as well as some that will fizzle – but these grants will provide the grand experiment to find out what works the best. I like that these grants make new awards each year for five years – and I hope Congress pays attention because some of the best programs that get this funding will deserve to be funded when these grants are over.

We are only going to best thrive as a nation when everybody comes along for the ride, and this is the first set of grants that will take a serious shot at bringing broadband to those who are not benefitting from broadband technology.

New Middle-Mile Grants

One of the new programs established by the Infrastructure Investment and Jobs Act (IIJA) is $1 billion in grants to build middle-mile fiber. The grant program will be administered directly by the NTIA. The grant program defines middle-mile as any broadband infrastructure that does not directly connect to an end-user location. Projects that might be considered for the grants include building fiber, leasing of dark fiber, submarine cable, undersea cables, transport to data centers, carrier-neutral internet exchanges, and wireless microwave backhaul.

The amount of funding is disappointingly small. For comparison, the California legislature created a $3.25 billion middle-mile fund just within the state. Rural America is woefully underserved by middle-mile infrastructure. Rural communities can all attest to the pain of losing broadband, cellular coverage, and public safety systems anytime the backbone fiber into a rural area goes out of service.

The stated purpose of the grants is to reduce the cost of connecting unserved and underserved areas to the Internet and to promote resiliency – which is the phrase currently being used as a surrogate for redundancy. Resiliency means bringing a second transport route into an area so that a single fiber or microwave failure won’t strand a community without broadband.

The federal grants will provide up to 70% of the cost of constructing middle-mile connectivity. Priority will be given to projects that:

  • Leverage existing rights-of-way to minimalize regulatory and permitting challenges.
  • Enable the connection of unserved anchor institutions.
  • Facilitate the creation of carrier-neutral interconnection facilities (places where multiple carriers can meet and exchange traffic).
  • Improve the redundancy of existing middle-mile infrastructure.

Grant applicants must demonstrate financial, technical, and operational capability to be eligible. Grant applicants must also satisfy at least two of the following conditions (and more than two would be better):

  • Has a fiscally sustainable middle-mile strategy.
  • Will offer non-discriminatory access to other carriers and entities.
  • Projects which identify specific last-mile networks that will benefit.
  • Projects with identified investments or support that will accelerate the construction and completion of a project.
  • Projects that will benefit national security interests.

The NTIA is directed to prioritize grant applications that:

  • Connect middle-mile infrastructure to last-mile networks that plan to serve unserved areas.
  • Connect non-contiguous trust lands.
  • Offer wholesale broadband service to other carriers and entities.
  • Can complete the buildout in a timely manner.

It’s likely to be until late 2022 until the grant program is taking applications. While not gigantic compared to other parts of the infrastructure bill, this grant would still translate into 20,000 miles of middle-mile fiber at $50,000 per mile. These grants are going to most easily be awarded to solid financial recipients that have assembled a consortium of entities that will pledge to use the new middle-mile routes. I strongly suggest that regional groups start talking now to be ready when these grants when announced. At only $1 billion, it seems likely that there will only be one grant cycle.

BEAD – The $42.5 Billion Infrastructure Grants

The new acronym used in the title of this blog refers to the official name of the new $42.5 billion grant program just approved by Congress last Friday – The Broadband  Access, Equity, and Deployment program. Another new acronym is IIJA, for Infrastructure Investment and Jobs Act – the name of the bill just passed by Congress. Today’s blog will talk about a few high-level rules governing the BEAD grants. I’ll cover other issues of the IIJA in upcoming blogs – things like middle-mile grants and broadband adoption. Since the following provisions are in the legislation they will be in the grant – but there are always tweaks made for final grant rules that will emphasize some points and downplay others.

  • You don’t need to rush to be ready to file for BEAD grants. This funding is going to flow between the NTIA and the States before going to specific grant projects. The Act gives the NTIA 180 days to come up with a plan for inviting states to apply for the funding. After the NTIA approves state plans, the states will have to develop and announce grant programs. I find it highly unlikely that there will be any grant applications due to states until the end of 2022, more likely in early 2023. States will get at least $100 million each, with the rest distributed based upon the number of unserved households in each state. This is a good time to remind those who think that the lousy FCC maps don’t matter that the States with the worst FCC maps are going to lose funding.
  • Cross your fingers that your State is competent because there are several crucial steps that states must adhere to before funding is provided.
  • As expected, grants must adhere to two key definitions of broadband. Unserved are places with broadband speeds under 25/3 Mbps. Underserved are areas with speeds between 25/3 and 100/20 Mbps. Grants must first go to unserved areas before being used for underserved areas. Funding for anchor institutions is only to be considered after serving underserved areas.
  • Grant projects must provide speeds of at least 100/20 Mbps, but faster broadband speeds must be given priority. States must give priority to grants that are deployed in counties with persistent poverty. Projects that are shovel-ready will be given priority. Projects that pledge to pay Davis-Bacon wages will get priority.
  • States will likely not award all of the grants immediately, and the Act asks states to provide a 5-year plan for the use of the funds.
  • Grants don’t have to all go for broadband to unserved and underserved areas. States can use the money for data collection, broadband mapping, and planning. Funding can be used to bring low-cost broadband or WiFi to qualifying multi-family apartments.
  • Unlike the recent NTIA grant program, BEADA doesn’t give priority to any class of grant recipients. The grants can’t exclude cooperatives, nonprofit organizations, public-private partnerships, private companies, public or private utilities, public utility districts, or local governments from eligibility – but none get a preference.
  • There is a challenge process where incumbent ISPs can challenge the validity of a grant area. Interestingly, the NTIA can override States in these challenges.
  • Grant applications must provide at least a 25% matching for the cost of the project. Matching funds can include CAREs funding and ARPA funding – so hang on to those funds for a while! Matching can also come from state grants.
  • Deployed technology must only meet two 9’s reliability, meaning that a network can be out for two days per year and still be considered adequate – that’s a low standard for the industry.
  • Grants must cover every home in a grant coverage area within four years of receiving the grant.
  • Grant recipients must provide at least one low-cost broadband option for eligible households. The NTIA is expressly forbidden to regulate rates in any manner.
  • Interestingly, any fiber built along highways must include access points at regular and short intervals. This money is not for middle-mile fiber.
  • Grant recipients must carry out public awareness programs in grant areas extolling the benefits of better broadband.
  • There is plenty of paperwork. Grant recipients must file semiannual reports tracking the effectiveness of the grant funding.

This grant program dwarfs all previous grant programs combined, so there is going to be a lot of money coming to every State. What is still to be determined is how States will administer these grants – and there will be differences. But the legislation provides enough detail for communities and ISPs to start looking at how to be positioned for these grants.

Rural Cellular Coverage

There has been a lot of recent news about rural cellular coverage and competition. Hopefully this will mean that rural cellular voice and data coverage will be getting better, but I suspect it’s going to matter where you live. But consider some of the changes in the market.

Before talking about the market, I’d like to first talk about rural cellular coverage today. In case you’ve never seen it, here is the FCC’s map showing LTE coverage for voice and data for AT&T, T-Mobile, Verizon, and U.S. Cellular. This map has been created from reports made to the FCC through the same Form 477 process that has resulted in lousy broadband coverage maps. Everything I read says that the cellular coverage maps suffer from the same kinds of problems due to overstated claims by the cellular carriers. I live in Asheville, NC, and I think the claimed coverage of each carrier is overstated, at least around here in the mountains. I happen to use AT&T, and I know of several sizable dead spots that are not reflected on this map. Being a telecom guy, I ask the coverage question a lot, and I’m told of similar rural holes for coverage for every carrier. But most of you will find the map to be interesting because it shows what the cellular carriers claim as coverage in your area.

T-Mobile says it’s going to start aggressively marketing in rural areas. To pursue that goal, the company will be opening 200 new retail stores. This is on top of the retail locations it picked up from the Sprint merger. T-Mobile says it will soon have outlets using the Metro brand name in 2.300 Walmarts. T-Mobile says that it is still on track to meet its goal of offering fixed cellular broadband to 97% of households by the end of 2022. That goal seems aggressive because of T-Mobile’s rural coverage today. Referring back to the FCC map, in my part of the world, T-Mobile has far less existing coverage than AT&T or Verizon.

One of the interesting aspects of T-Mobile’s rural coverage is that the company will be using 600 MHz spectrum in rural areas. That spectrum carries a lot farther than the traditional cellular spectrum, but the lower spectrum also carries less broadband. It’s going to be interesting to see the broadband speeds the company can deliver more than a few miles from a rural cell site.

AT&T also says it’s in the process of boosting rural cellular coverage. Much of its new coverage will be coming from new cell sites that are being activated at towers constructed as part of AT&T’s nationwide roll-out of FirstNet – the nationwide public safety network. The federal funding that is covering the cost of deploying public safety towers is giving AT&T new platforms for cellular coverage. AT&T also uses a relatively low spectrum frequency at 700 MHz that travels farther in a rural setting than its traditional spectrum. A lot of AT&T’s rural coverage comes from its popular Cricket Wireless brand that has a lot of stores in smaller markets. The only downside of Cricket is that it’s widely rumored that AT&T cell sites supposedly give priority to AT&T-branded customers over the lower-priced Cricket customers.

Verizon supposedly has the best rural coverage today of the major carriers, which came as a result of the purchase in 2009 of Alltel. The coverage was enhanced further by the 2020 purchase of TracFone. Verizon announced recently that it is shutting down the 2G and 3G networks operated by TracPhone, which is causing a lot of consternation in rural America. TracFone was known as the last network that was friendly to older flip phones.

What does all of this mean for rural cellphone coverage and competition? The expansion of T-Mobile into rural markets should bring a new competitor to many markets where perhaps only one carrier was available before. The use of the 600 MHz and 700 MHz should expand rural cellular broadband coverage, but we’ll have to wait and see what those spectrum bands mean in terms of broadband speeds. Cutting back on TracFone coverage will hurt some rural pockets where the companies might have been the only cellular option.

Predicting Future Broadband Prices

I’ve been giving a lot of thought lately to the long-term trajectory of broadband prices. This is something that should be considered by anybody who is thinking about competing in a market against a big cable company. It’s not an easy question to get your hands around since there are many factors that might affect future rates. The following are some of the major trends that I think must be considered.

First is pressure on the big ISPs from Wall Street. This is most acutely being felt by the biggest cable companies because they’ve had steady growth in customers and revenues for the last decade as they have slowly decimated the urban DSL market. Their stock prices are predicated on maintaining the historical earnings growth rate, and that will get harder to do every year. I don’t need to be a great prognosticator to see that the day will be coming soon when there aren’t that many more DSL customers left to capture. When customer growth slows, then the only other tool to maintain earnings will be to increase rates.

We already know that the big cable companies are willing to raise broadband rates. Charter and Comcast have both had back-to-back years of raising broadband rates, and it’s about that time of the year for our next holiday present. I calculate the price of standalone Comcast broadband is already at $90. That’s $76 for the basic broadband product and $14 for the cable modem. Charter has imposed back-to-back $5 annual rate increases and is not far behind.

The cable companies are also happy to disguise the true cost of broadband. For example, Comcast must be making a fortune from its data caps. OpenVault reported that at the end of the second quarter of this year that 12.3% of homes are using more than a terabyte of data each month, including 1.5% of homes that are using more than two terabytes monthly. Charter is poised to implement data caps immediately after the end of its agreement with the FCC to not use data caps that came from the merger with Time Warner Cable.

Demographics are also trending the wrong way for cable companies. Population growth rates have slowed, and the U.S. is approaching the zero population growth rates we now see in Japan and some European countries. The prolonged clamp-down on immigration has cut off the other driver of household growth. The large cities where cable companies are centered are also seeing some outflow of people moving to smaller markets as a result of the pandemic, although it’s too early to know if this is a permanent trend.

One of the interesting upsides for the big ISPs is low-income broadband subsidies. If Congress creates a more permanent low-income subsidy with the infrastructure legislation, the big ISPs could get a big burst of new customers and revenues from aggressively pursuing low-income households.

Another issue that should be quietly boosting cable company earnings is the slow death of bunding as millions of homes drop cable TV each year but keep broadband. The cable companies get a bump of $10 or $15 in long-term margin every time a customer breaks a bundle.

The big unknown is competition. In many cities, the big cable companies have become monopolies as DSL dies. But the cable companies are starting to see competition nipping at their heels as other ISPs start filling the void left by DSL. T-Mobile is offering $50 unlimited 100 Mbps broadband from cell towers. Verizon says it’s still on track to roll out 25 million homes with fiber-to-the-curb by 2025. Brand new competitors like Starry are testing the waters and could pop up in dozens of cities quickly. Fiber overbuilders are popping up in markets everywhere to present real competition.

It’s impossible to somehow plot out how these and other factors are affecting the broadband industry in order to guess the future. But since writing a blog lets me pretend to be a pundit, I will venture my best guess. I predict that big cable companies will continue to raise basic rates year after year. People unlucky enough to live in monopoly markets will pay the higher prices. The cable companies will continue to pad revenues with quiet sources of extra revenue like data caps. But in competitive markets, the cable companies will lower rates to remain competitive. The cable companies have unfortunately gotten sophisticated, and similar to the way that airlines have mastered pricing, the cable companies will set rates in each market and for each family based upon algorithms that predict the most they can charge.

These many trends must be creating headaches for the analysts inside of Comcast and Charter. Trying to figure the future for these companies is a mind-numbing challenge. But I have no doubt that that the big cable companies will instinctively go with the old tried-and-true method of raising rates to see what happens.

 

 

Beware the Monopoly ISP

When I do surveys in cities I know that the number one complaint from the public about the broadband situation is a lack of choice. As residents have given up on DSL they finally figure out that the cable company is the only game in town.  I generally tell cities that they should be worried because economics texts all describe the inevitable behavior of monopolies – they invariably will raise rates, cut service, and treat customers with disdain. It’s not always easy to show proof of monopoly behavior because it’s subtle and creeps in over time. An ISP that becomes a monopoly doesn’t turn into an ogre overnight – its bad behavior builds over time through a series of small actions that eventually result in monopoly misbehavior.

Economists say that monopolies always migrate over time to abusive monopoly behavior – it’s almost impossible for a monopoly corporation to not do this. Monopolies rarely act poorly due to devils in the board room that decide to take advantage of monopoly markets. Monopoly behavior comes more often from the employees of the monopoly who implement policies that benefit the company over customers. In a lot of the most egregious cases, the bad actions are driven by employees wanting to make bigger bonuses.

Consider two such monopoly actions taken by big ISPs during the pandemic. It’s not hard to find similar stories about every big ISP, but these stories are a pure demonstration of monopoly power.

Verizon quickly enrolled in the FCC’s Emergency Broadband Benefit (EBB) program when it was announced. This program provides a $50 discount off broadband bills to households who qualify either by being low-income or by being out of work due to the pandemic. Like every big ISP that enrolled in the program, Verizon crowed on its website how it was doing its bit to help folks through the pandemic. Other than perhaps the cost of advertising, the program costs Verizon nothing since the FCC reimburses the company for the discounts given.

But within days of Verizon’s introduction of the EBB plan, stories started coming out from Verizon customers who were told they had to upgrade to a higher-cost broadband product to get the discount. Some marketing folks at Verizon got the bright idea that this was an opportunity for the company to make more money through millions of upgrades. This meant that customers didn’t save the full $50 because they had to pay more for faster broadband. When the EBB discount ends, these customers will find themselves with a broadband package they can’t afford.

When Verizon was called out by the FCC for the practice, rather than do the right thing, the company gave customers 14 days to switch back to the older, less costly plan but didn’t notify any of them of the option. In an example of pure corporate gobbledygook, Verizon tried to justify to the FCC how its actions were not the same as upselling.

In a similar story, Altice has been hit with a class-action suit for not meeting the pledge it made in the Keep America Connected initiative. This was a voluntary pledge during the early days of the pandemic where FCC Chairman Ajit Pai asked ISPs to pledge to not terminate residential and business customers or levy late fees due to the inability to make payments.

Like 800 other IPS, Altice made the pledge, and the Altice USA CEO Dexter Goei was quoted as saying, “Altice USA is proud to do its part in ensuring that customers and business in our service areas have reliable access to the connectivity services that are critically important during this rapidly evolving public health situation.”

One of the complaints in the lawsuit came from a barbershop in the Bronx. Like many businesses, the barbershop was forced to shut its doors by a local ordinance during the pandemic. The barber says that Altice billed him during the 3-months of the mandatory shutdown and then terminated his service at the end of the three months. When he asked if he could pay his open balances to restore broadband and telephone service, the barber was told that he’d have to open a new account and pay a $180 reconnect fee. Further, Altice told him that he couldn’t have the telephone number he’d been using for decades. As a final insult, his new bill was double the old one.

While the Keep America Connected initiative was voluntary, the big ISPs made a big deal about it when they signed on. ISPs were loudly proclaiming how they were pitching in to get America through the pandemic. It’s hard to imagine how Altice could have violated the intent of that agreement any more than the way it treated this barber and many similar businesses.

Monopoly ISPs don’t need to do much soul searching to wonder why people dislike them more than any other type of company in the country. National satisfaction surveys annually rate big ISPs as the least-liked customer-facing business in the country, even lower than the IRS. I wish these were the only two recent cases of monopoly abuses by big ISPs – but the ways the big ISPs mistreat customers could fill a bookshelf. Monopolies invariably abuse customers – and that alone is justification for cities trying to attract new ISPs to their market. The only real cure for monopoly behavior is competition.