Categories
The Industry

Cable Company Cellular Growing

Cable companies are starting to quietly build a significant cellular business to bundle with broadband and other products. Consider the most recent customer count from the eight largest U.S. cellular carriers:

Verizon 143.0 M
T-Mobile 110.2 M
AT&T 101.6 M
Dish 8.5 M
US Cellular 4.9 M
Comcast 4.6 M
Charter 4.3 M
C-Spire 1.2 M

It’s worth noting that AT&T has over 200 million cellular customers worldwide, which makes them the eleventh largest cellular carrier in the world, with China Mobile first with over 851 million customers.

Comcast’s Xfinity Mobile added 317,000 customers in the second quarter of this year to bring the company to a total of 4.6 million customers. Comcast mostly uses the Verizon network to complete calls. However, Comcast demonstrates the major benefit of a cable company being in the cellular business since the company is able to offload a large portion of its outgoing mobile traffic to its WiFi network. Comcast has been experimenting with the use of 600 MHz spectrum to carry some of its cellular traffic. The company purchased $1.7 billion of spectrum in the 2017 incentive auction that freed up spectrum formerly used by television channels. Comcast also purchased $458 million of CBRS spectrum in 2020. The company says it may selectively offload traffic onto licensed spectrum in places where that is cheaper than buying wholesale minutes.

Charter’s Spectrum Mobile added 344,000 mobile customers in the second quarter of the year to bring the company to 4.3 million customers. Spectrum also uses the Verizon network. Charter purchased $464 million of PAL licenses in the CBRS spectrum in 2020. Charter says it intends to place its own radios in high-traffic areas where that will save money. Charter’s CEO Brian Roberts said a few months ago that Charter saw $700 million in new revenues from cellular over the past twelve months.

Altice has been selling mobile services branded as Optimum Mobile for several years and added 33,000 customers in the second quarter, bringing the company to 231,000 total mobile customers. Altice uses the T-Mobile network.

Cox announced the launch of a mobile pilot program on August 29, launching Cox Mobile in Hampton Roads, Virginia, Omaha, Nebraska, and Las Vegas.

All of these companies have a huge potential upside. For example, the mobile customer penetration rate for both Comcast and Charter is under 10%, and both companies believe they can become major mobile players in their markets.

The cable companies face an unusual marketing challenge since each cable company is only in selected urban markets, meaning that a lot of nationwide advertising goes to waste.

The primary reason that Comcast first entered the mobile market was to develop another product that would create a stickier bundle. Comcast figured it would be hard for a customer to leave if that meant finding a new cellular carrier along with a new ISP. Cable companies are still only selling to their own broadband customers, which is a good indication bundling is still a key reason for doing this. It’s also less costly to sell cellular to households that can offload cellular traffic to the cable company broadband network.

The big three cellular carriers have continued to grow in recent years, but the cable companies have definitely made a dent in the market with almost ten million retail mobile customers. The real test for the cellular industry is going to come when Dish finally gets its act together and offers low-cost mobile service in most markets. That’s going to put price pressure on everybody else. If Dish starts a price war, as promised, we’re going to see a real shake-up.

 

 

Categories
Current News The Industry

Satellite Cell Service

T-Mobile and Starlink made a joint announcement recently about an arrangement where Starlink will enable voice and texting capabilities to T-Mobile cellphones by the end of 2023. This is a service that would work with existing cell phones and would supposedly kick in when a phone can’t find a signal from a cell tower. Starlink said the technology would be enabled by new satellites that have significantly larger antennae than the current satellites in the constellation. In the press release, Elon Musk touted this as being able to reach people lost in the wilderness, but the much bigger use will be to fill in cellular coverage in rural areas for T-Mobile.

While the two companies made a big splashy announcement about the arrangement, they are late to the game as other industry players already have similar plans underway.

AST SpaceMobile has been working on deploying satellites aimed specifically at the cellular market. The company plans to launch its first five satellites in 2024. The company’s business plan is to launch fairly large satellites weighing over 3,300 pounds to create a constellation dedicated to cellular coverage. The company has already created partnerships with more than 25 mobile operators around the world, including the giant cellular company Vodaphone.

Lynk is taking a different approach and will launch small satellites around the size of a pizza box. The company has one test satellite in orbit with another schedule this December. The company plans to have 50 satellites in orbit by the end of 2023. Lynk already has 14 commercial agreements in place and will support large corporations and governments as well as mobile providers.

Just yesterday, Apple announced that it will offer a texting service for those lost in the wilderness in a partnership with Globalstar. This service is going to be text only and is going to be exceedingly slow, but it will supposedly work for folks who have the latest iPhone and who also are able to point the phone directly at the satellite. There will be an app that will tell a user where the satellite can be found.

All of these plans raise a lot of questions that we won’t get answered until somebody has a working satellite product. For example, could somebody inside a vehicle connect to a satellite? I have no problem connecting to the Sirius XM satellite service, so this might not be a problem. Will these connections somehow roam and connect back to cellular carriers when the user is in reach of a cell tower? That would be really complicated, and my guess is that this won’t work. Mike Sievert, the CEO of T-Mobile said this project is like putting a cell site in the sky, but much harder – and I believe him. I’ve been trying to picture how the satellites will pick out the right calls because filtering through the many billions of cellphone calls to find the right ones sounds like a huge data processing challenge.

The service would certainly be a boon to somebody lost in the  woods, but this is a much-needed service for a lot of people. My consulting firm does surveys, and it’s not unusual to find rural counties today where 30% or more of homes say they have no cellular coverage at their homes. The national coverage maps of the big cellular companies are a joke in many rural places.

T-Mobile and Starlink said that these connections would be only for voice calls and texting at first but that using cellular data might be on the horizon. That would be a significant accomplishment since a receiver many times larger than a cell phone is needed today to communicate with a satellite.

The real potential for this product is not in the U.S. and Europe where a large percentage of folks can connect today to cellular networks. The real market is the many parts of the world where modern cellular towers are a rarity. Most Americans probably don’t understand or appreciate that there is still a lot of the world where folks are not connected, or perhaps only connected through one universal connection that is shared by a whole community.

Categories
The Industry

Traditional Big ISPs Stagnate

In the first quarter of this year, the big cable companies added 482,000 customers while telcos added over 50,000 customers. In what is a surprise to the industry, that growth has disappeared, and all of the big ISPs collectively lost almost 150,000 customers. That’s a loss of 60,000 customers for the cable companies and 88,000 for the big telcos.

The following statistics have been compiled by the Leichtman Research Group, which tracks the broadband performance of the largest ISPs in the country.

The other big news is that the Fixed Wireless Access (FWA) products of T-Mobile and Verizon added 816,000 customers in the second quarter to bring the sector to net growth of 668,000 customers. This is huge news – FWA is booming while the big ISPs are standing still. The FWA product is home broadband delivered using cellular frequencies. T-Mobile and Verizon are aggressively marketing the product, which is touted to have download speeds over 100 Mbps. The market is going to get even hotter when AT&T and Dish networks enter the market in a big way.

The numbers for the second quarter of 2022:

 2Q 2022 2Q Change % Change  
Comcast 32,163,000 0 0.0%  
Charter 30,253,000 (21,000) -0.1%  
AT&T 15,509,000 (24,000) -0.2%  
Verizon 7,412,000 12,000 0.2%  
Cox 5,560,000 0 0.0%  
Lumen 4,377,000 (93,000) -2.1%  
Altice 4,333,600 (39,600) -0.9%  
Frontier 2,827,000 8,000 0.3%  
T-Mobile FWA 1,544,000 560,000 56.9%  
Mediacom 1,468,000 0 0.0%  
Windstream 1,178,500 2,500 0.2%  
Cable ONE 1,059,000 2,000 0.2%  
Breezeline 717,919 (1,689) -0.2%  
Verizon FWA 700,000 256,000 57.7%  
TDS 500,800 5,600 1.1%  
Consolidated 381,213 1,063 -0.2%  
   Total 109,984,032 667,874 0.3%  
         
Total Cable 75,554,519 (60,289) -0.1%  
Total Telco 32,185,513 (87,837) -0.3%  
FWA 2,244,000 816,000 57.1%  

There is a lot to unpack in these numbers:

  • The cable companies have gained customers every quarter for far longer than a decade, so this net loss for the sector is a big surprise.
  • There is another story underneath the big telco losses – fiber is doing well. AT&T added 316,000 fiber customers in the quarter but still had a small net customer loss. Frontier added 54,000 fiber customers for the quarter and had a small net customer gain. Verizon added 36,000 FiOS customers in the quarter. Lumen added 28,000 fiber customers for the quarter but continued to bleed DSL customers.
  • T-Mobile leaped to become the ninth largest ISP in the country.
  • TDS repeated as the fastest growing traditional ISP.
  • Lumen lost the largest percentage of customers compared to other telcos. Altice was the biggest percentage loser among cable companies.
Categories
Regulation - What is it Good For?

The CHIPS Act and Wireless

The recently enacted CHIPS and Science Act of 2022 is providing a lot of funding to bring more chip manufacturing back to the U.S. This funding fills a big hole in the U.S. supply chain. We have some chip manufacturing in the U.S., but we only make about 12% of the chips that we use in cellphones, cars, computers, and broadband technology.

Making domestic chips became a national priority when we saw during the pandemic that international chipmakers took care of regional demand before U.S. demand. U.S. automakers are still largely on hold due to a lack of chips, and there has been a rumor floating around the broadband industry that we’re going to see another round of chip shortages for broadband gear. It will take some years to turn this new funding into chip factories, but in the long run, this is one of the more sensible things Congress has done in many years.

The CHIPs Act approved $52 billion to bring chip manufacturing back to the U.S. But like all big legislation, not all of the money appropriated goes to the main goal. For example, there is funding in the bill for new research and development in the technical sciences. Today’s blog looks at funding from the CHIPs Act that is being used for the mobile industry. Specifically, the CHIPS legislation:

Appropriates $1.5 billion for the Public Wireless Supply Chain Innovation Fund, to spur movement towards open-architecture, software-based wireless technologies, funding innovative, ‘leap-ahead’ technologies in the U.S. mobile broadband market. The fund would be managed by the National Telecommunications and Information Administration (NTIA), with input from the National Institute of Standards and Technology, Department of Homeland Security, and the Intelligence Advanced Research Projects Activity, among others.

This sounds like funding for wireless product research to find new market uses for 5G. I’m a big believer that the federal government should have a large role in funding basic science research and development. One of the reasons that the U.S. has had technological success in the past is that we funded the basic research that has made the breakthroughs that turned into our current technology industries. National funding for pure research has fallen in recent years to woefully low levels.

But I’m not a big fan of the U.S. government undertaking product research. That is something that ought to be left to the industries that will benefit from the research. This $1.5 billion feels like a handout to the big wireless companies – and they don’t need this money.

Consider dividends. Verizon paid out $10.4 billion in dividends to stockholders in 2021, or almost $2.50 for every outstanding share. In recent shareholder meetings, the company says the goal is to increase dividends in the coming years. AT&T most recently paid $8 billion per year in dividends or $1.11 per share in recent quarters.

T-Mobile is the most cash-flush of the big cellular carriers and told shareholders earlier this year that the company plans to spend $60 billion by the end of 2025 to buy back its own stock.

These three companies don’t need a $1.5 billion government handout, but as often happens, the industries that lobby the hardest often get rewarded with funding. If the $1.5 billion is spent wisely, it might turn into future profits for these companies. But this is research that these companies should be routinely funding directly.

This feels like a residual benefit to these companies from all of the effort they put into persuading the government that we were losing an imaginary 5G war with China. That discussion is still not completely dead, and we still occasionally hear a politician talking about our 5G crisis.

I love the concept behind the CHIPS Act, and I hope it spurs 100,000 new permanent manufacturing jobs and greatly expands the domestic chip supply. But I am not a fan when big legislation is used to pay back industries that spend huge money to lobby politicians.

Categories
Regulation - What is it Good For?

Another BEAD Grant Complication

I’ve been thinking more about the NTIA’s definition of Reliable Broadband Service that was part of the recently issued Notice of Funding Opportunity (NOFO) for the $42.5 billion BEAD grants. That definition says that any grant cannot be used to overbuild a reliable broadband technology that meets or exceeds the 100/20 Mbps speed threshold of the grants. The NOFO said that the grants can’t be used where speeds are adequate for the following technologies: (i) fiber-optic technology; (ii) Cable Modem/ Hybrid fiber-coaxial technology; (iii) digital subscriber line (DSL) technology; or (iv) terrestrial fixed wireless technology utilizing entirely licensed spectrum or using a hybrid of licensed and unlicensed spectrum.

The policy behind this makes sense – the NTIA doesn’t think that valuable federal grant dollars should be used where adequate broadband technology is already in use. That would make them a good shepherd of the federal dollars.

But this particular definition is going to cause some complications the NTIA might not have considered. I’ve been running into rural FWA cellular wireless broadband in rural markets. So far, I’ve only encountered the new technology from T-Mobile and Verizon. But this will also be introduced by Dish Network. AT&T says it also has plans to roll out the faster cellular home product.

The FWA technology is enabled when a cellular company beefs up cell sites to provide home broadband in addition to cell phone service. This is being enabled by the introduction of new spectrum bands. For marketing purposes, the carriers are labeling these new bands as 5G, although the technology is still 4G LTE.

The cell carriers have been offering a weak version of home broadband for years, marketed as a hotspot or jetpack. But that technology shared the same frequencies used for cell phone service, and the broadband has been slow, weak, erratic, and expensive. However, putting home broadband onto new cellular spectrum changes the product drastically.

Recently I heard from a farmer who is getting 200 Mbps download broadband from a rural T-Mobile FWA connection – this farmer sits right next to a large cell tower. According to the NTIA, this farm should not receive any grant subsidy to bring fiber broadband with a grant. But as is usual, real life is a lot more complicated than that. This same farmer says that his nearest neighbors, only a little over a mile away, are seeing speeds significantly below 50 Mbps.

This makes sense because that’s how cellular technology works. Most people don’t realize how quickly broadband signal strength weakens with distance from a cell site. In cities, practically everybody is within half a mile or a mile from a cell site, so we never notice. But in rural areas, most people live too far from a cell site to get decent bandwidth from this technology. Consider the following heatmap of a real cell site.

The fastest broadband speeds would be within a few thousand feet, like with the farmer. The area that might get 100 Mbps broadband is in the orange and yellow areas on the map. The speeds in the green areas are where speeds fall below 100 Mbps, and by the time the broadband signal reaches the light blue areas the speeds are almost non-existent. The purple areas show where a voice signal might carry, but only unreliably.

What does this mean for the BEAD grants? As T-Mobile and the other cell carriers start updating rural cell sites they are going to be putting heatmaps like the one above into the FCC mapping system. It’s worth noting that most cell sites don’t create a roughly symmetrical coverage pattern because the wireless signal gets disrupted by any obstacles in the environment, even small rolling hills. It’s also worth noting that cellular coverage is dynamic and changes with temperature, precipitation, and even wind.

Recognizing cellular broadband coverage (licensed) as reliable broadband will have several consequences. First, this disrupts grant coverage areas since there will be cellular areas in every county that won’t be eligible for grants. This will create a swiss cheese phenomenon where there are areas where grants are allowed next to rural areas that are not allowed. That will complicate the engineering of a broadband solution for the areas that are left. This is the same thing the FCC did with the RDOF awards – chopped up potential grant areas into incoherent, illogical, and costly swiss cheese.

This also might mean this farmer won’t get fiber. His neighbors who can’t get good speeds on T-Mobile might be covered by a BEAD grant, but an ISP might be unwilling to fund the cost to reach this farmer if the cost is not covered by a grant.

I doubt that the NTIA thought of the practical consequences of the new definition, just like I can’t imagine the FCC had the slightest idea of the absolute mess they made with RDOF coverage areas. The only way to justify building a new network in a rural area, even with grants, is to cover large areas with one coherent network – not by building a network that has to somehow avoid RDOF areas and cell towers.

ISPs interested in BEAD awards are now going to have to wait until the new broadband maps come out to know what this might do to their grant plans. I’m thinking that, at least in some cases, this will be the final straw that breaks the camel’s back and convinces an ISPs to walk away and not even try.

Categories
The Industry

The Upcoming Marketing Wars

In April 2019 my daughter and I were watching the NCAA final between Virginia and Texas Tech (and rooting for her school TTU). We noticed about halfway through the game that practically every ad we had seen was about 5G. Verizon was busy showing us speed tests from millimeter-wave cellphones receiving gigabit speeds. Not to be outdone, there were a ton of commercials also from T-Mobile and AT&T.

I found it extraordinary that the cellular carriers would spend that much money to buy premium-rate ads for a major sports event. We now know this was part of an all-out blitz on 5G to put pressure on Congress and the FCC to give them more spectrum.

I think that by this fall we’re going to wish we could go back to the 2019 level of ads because I’m predicting by this fall that all we’re going to hear about is cellular and broadband. A lot has changed in the industry since 2019. In more recent sporting events, I noticed that a lot of the ads were from the cable companies touting low-cost cellular service. The cable companies view bundling with cellular as one of the best ways to retain broadband customers – bundling means that when a customer drops broadband they will also lose cheap cellular service.

Dish network will be hitting the market sometime this summer, promising a rollout in a hundred smaller markets and 25 large markets in June. The company already owns Boost Mobile, but Dish is going to spend a lot of money to convince America to consider it for cellular service. This means mountains of advertising to make us aware that Dish is now a cellular company. Dish promises to be aggressive with pricing, so expect this advertising effort to set off a price war from the other carriers.

T-Mobile has already been blitzing the air this year in an attempt to sell its cellular broadband product. The company picked up 400,000 new cellular broadband customers in 2021, most at the end of the year. T-Mobile has a goal to pick up several million new broadband customers this year. T-Mobile’s ultimate goal is to reach 6 to 7 million FWA customers by 2025.

Verizon is also selling fixed wireless broadband and plans to hit the market hard later this summer. The company has a goal to reach 4 to 5 million FWA customers by 2025.

AT&T isn’t going to hit the national market with a push for FWA until some time in 2023, but there is no way that the company is going to sit by and watch the other cellular carriers lure away its customers. Expect AT&T to also be on the air nonstop.

It’s hard to think that national advertising this fall will be much more than cellular and political ads. I’m warning you now to find an outdoor hobby if you don’t want to hear any more about 5G.

We’re also going to see an unprecedented marketing blitz from cable companies and fiber overbuilders. All of the big telcos are furiously building fiber this year. There are aggressive plans to build fiber underway from AT&T, Verizon, Frontier, Windstream, Consolidated, Ziply, Lumen, and many smaller fiber builders. Much of the construction this year will be in cities and county seats, and that is going to mean a whole lot of advertising.

We’ve not seen a lot of national advertising about home broadband, and the marketing wars will likely be local. That’s going to translate to salespeople knocking on doors and a lot of mailers about fiber broadband.

There was some unexpected growth in cellular customers last year. For example, in the third quarter of last year, there was a net addition of 2.3 million new nationwide cellular customers. Industry analysts are chalking this up to businesses buying cell phones for remote employees and more students buying phones because of remote learning during the pandemic.

This kind of market growth is not sustainable since most people have cell phones. That means that the coming cellular marketing wars will largely be a zero-sum game. The only way for a cellular company to grow will be to take customers from another carrier. That’s going to lead to some real desperation – and even more ads. When you watch the first football game this fall, don’t say I didn’t warn you.

Categories
The Industry

Stock Buybacks

All of the big ISPs brag to the public about how much they spend on their networks. There is barely a press release when they don’t remind the public how much money they are pouring back into making their networks better. Even at the local level, it’s rare to ask a big ISP to a local government meeting where they don’t open the conversation by reminding local politicians how much money they have spent in a given town or county.

The story is often just the opposite when problems with networks are pointed out, and communities ask the ISPs to beef up networks and improve service. That’s when we hear that money for capital spending is tight, but an ISP will make upgrades a priority in the future.

What’s never heard in conversation about capital spending is how much big ISPs spend to buy back shares of their own stock. This is a practice where big ISPs (and many other large corporations) use profits to purchase and retire stock. The transaction reduces the number of shares of outstanding stock and consequently nudges up the announced earnings per share. The first time I encountered the practice, I was flabbergasted.

Let’s consider the Comcast stock buybacks. Comcast paused stock buybacks in 2019, but in 2021 repurchased 73.2 million shares of stock for $4 billion. The company has over 4.5 billion outstanding shares of stock, so the buyback reduced the shares of outstanding stock by 1.6%. Comcast earnings for 2021 were announced as $3.06 per share for the year. Without the stock buyback, the earnings would have been $3.01.

The theory is this small nudge is good for investors. But it’s hard to envision a worse use for cash. Comcast could have gotten a far better return for investors from using that money to extend networks around their current markets, upgrading older networks to keep customers loyal, or marketing to add new customers. Those kinds of changes would result in long-term value gain for shareholders. Comcast recently announced that it is increasing the stock buyback in 2022 to $10 billion. To put that into perspective, Comcast’s capital spending for the last two years was $11.6 and $12.1 billion.

ISPs vary in the amount put towards stock buybacks according to their current cash situation and Board philosophy. Here are a few other stock buyback plans for large ISPs.

  • Charter has actively been buying back its stock. The company repurchased $15.4 billion of its own stock in 2021 and $11.2 billion in 2020.
  • T-Mobile has plans to really step up stock buybacks and plans to repurchase $60 billion of its own stock between 2023 and 2025.
  • AT&T is not currently buying back stock and only repurchased $104 million of stock in 2021.
  • Verizon told investors it would buy back 100 million shares of stock in 2022 – the stock is currently trading at $54 per share.
  • The one that is hardest to understand is Lumen. The company generated $700 million in free cash flow in 2021 and spent $1 billion to buy back its stock. That probably demonstrates the pressure that Wall Street is exerting for stock buybacks.

This makes me wonder if corporations that are engaging in stock buybacks should be allowed to get federal grants. For example, should we have allowed a company like Charter to get $1.2 billion in RDOF funding in 2020 at a time when the company was spending $11 billion to buy back its own stock? Did Charter really need a federal subsidy, or does grant funding just allow a company to even further increase stock buybacks? I don’t have an answer for that other than it just doesn’t feel right.

Categories
Technology

Fixed Cellular Broadband Performance

One of the first in-depth reviews I’ve found for T-Mobile’s fixed cellular broadband was published in the Verve. It’s not particularly flattering to T-Mobile, and this particular customer found the performance to be unreliable – fast sometimes and barely functioning at other times. But I’ve seen other T-Mobile customers raving about the speeds they are receiving.

We obviously can’t draw any conclusions based upon a single review by one customer, but his experience and the contrasting good reviews by others prompted me to talk about why performance on cellular broadband networks can vary so significantly.

I’ve always used the word wonky to describe cellular performance. It’s something I’ve tracked at my own house, and for years the reception of the cellular signal in my home office has varied hour-by-hour and day-by-day. This is a basic characteristic of cellular networks that you’ll never find the cellular carriers talking about or admitting.

The foremost issue with cellular signal strength is the distance of a customer from the local cellular tower. All wireless data transmissions weaken with distance. This is easy to understand. Wireless transmissions spread after they leave a transmitter. The traditional way we depict a wireless transmission shown in the diagram below demonstrates the spread. If two customers have the same receiver,  a customer who is closer to the tower will receive more data bits sooner than somebody who is further after the signal has spread. The customer in the bad review admitted he wasn’t super close to a cell tower, and somebody in his own neighborhood who lives closer to the cell site might have a stronger signal and a better opinion of the product.

There are other factors that create variability in a cellular signal. One is basic physics and the way radio waves behave outdoors. The cellular signal emanating from your local cell tower varies with the conditions in the atmosphere – the temperature, humidity, precipitation, and even wind. Things that stir up the air will affect the cellular signal. A wireless signal in the wild is unpredictable and variable.

Another issue is interference. Cellular companies that use licensed spectrum don’t want to talk about interference, but it exists everywhere. Some interference comes from natural sources like sunspots. But the biggest source of interference is the signal from other cell towers. Interference occurs any time there are multiple sources of the same frequency being used in the same area.

The customer in the review talks about the performance differing by the time of day. That is a phenomenon that can affect all broadband networks and is specific to the local robustness of the T-Mobile network. Performance drops when networks start getting too busy. Every DSL customer or cable company broadband customer has witnessed the network slowing at some times of the day. This can be caused by too many customers sharing the local network – in this case, the number of customers using a cell tower at the same time. The problem can also because caused by high regional usage if multiple cell towers share the same underlying broadband backbone.

The final issue that is somewhat unique to cellular networks is carrier priority. It’s highly likely that T-Mobile is giving first priority to customers using cell phones. That’s the company’s primary source of revenue, so cell phones get first dibs at the bandwidth. That means in busy times that the data left over for the fixed cellular customers might be greatly pinched. As T-Mobile and other carriers sell more of the fixed product, I predict the issue of having second priority will become a familiar phenomenon.

This blog is not intended to be a slam against fixed cellular broadband. The customer that wrote the review switched to cellular broadband to get a less expensive connection than from his cable company. This customer clearly bought into the T-Mobile advertising hype because a cellular broadband signal will never be as reliable as a signal delivered through wires.

We can’t forget the real promise of fixed cellular broadband – bringing broadband to folks who have no alternatives. Somebody that switched to T-Mobile from a 1 Mbps rural DSL product would have written a different and more glowing review of the same product. The bottom line is that anybody buying cellular broadband should recognize that it’s a wireless product – and that means the product comes with the quirks and limitations that are inherent with wireless broadband. I imagine that we’re going to continue to see bad reviews from customers who want to save money but still want the performance that comes with wired broadband. This is another reminder that it’s a mistake to judge a broadband product strictly by the download speed – a 100 Mbps cellular broadband product is not the same as a 100 Mbps cable company connection.

Categories
Regulation - What is it Good For?

The FCC and Broadband Outages

Comcast had a widespread network outage in early November. The problems started in San Francisco and spread the next day to Chicago, Philadelphia, parts of New Jersey, and three other states. The outage knocked out broadband customers along with Comcast cellular customers. Comcast has never disclosed the reason for the outage and announced only that it was due to a ‘network issue’.

In 2020 CenturyLink suffered an even larger outage that not only knocked out CenturyLink customers but spread into other networks, including Amazon, Cloudflare, and Hulu. The problem was blamed on a software update that blocked the establishment of Border Gateway Protocol (BGP) sessions and impeded broadband traffic routing.

T-Mobile also had a major network outage in 2020 that knocked out broadband customers and also cut off some voice calls and most texting for nearly a whole day. T-Mobile blamed the issue on problems with a leased circuit that was compounded by two previously undetected flaws in third-party software. Reports at the time said that the electronics failed on a leased circuit, and then the backup circuit also failed. This then caused a cascade that brought down a large part of the T-Mobile network.

In 2019 CenturyLink had perhaps the largest outage that knocked out much of its network and customers that relied on the Level 3 network for transport. The company blamed the outage on a bad circuit card in Denver that somehow cascaded to bring down a large swath of fiber networks in the West, including numerous 911 centers.

The FCC investigates big outages from time to time and opened an inquiry in October 2020 in a few of the outages listed above. The FCC also recently adopted a Notice of Proposed Rulemaking to investigate the disaster resiliency plans of major telecom providers to take a harder look at how cellular and broadband carriers make repairs after big storms.

Interestingly, the FCC recently fined T-Mobile $19.5 million for the 2020 outage, but not the other carriers. This is not because T-Mobile’s outage was worse than the others. T-Mobile was fined because they are a cellular carrier and still fully regulated by the FCC. But Comcast and CenturyLink are ISPs and under different regulatory rules.

Oddly, the FCC has very little power to do anything about ISP network outages because the FCC has very little regulatory authority over ISPs in general. The FCC abrogated its authority to regulate ISPs when it killed Title II regulation and handed a few vestiges of regulation to the Federal Trade Commission. The FCC only regulates ISPs tangentially through the specific authority given directly by Congress. Any authority the FCC once had as a result of claiming Title II regulatory authority is gone.

The process has finally started to seat a fifth FCC Commissioner, and the industry speculates that one of the early acts with five Commissioners will be to reinstate Title II authority. This effort might be a little more streamlined in the past because federal courts have already ruled that the FCC can choose to regulate or not regulate broadband.

Unfortunately, any move to regulate ISPs and broadband will only last until we have another shift in administration that wants to kill regulation again. We have ended up in an absurd regulatory merry-go-round where regulating or not regulating ISPs depends on the party that controls the White House. It makes no sense to not regulate ISPs at a time when cable companies have nearly total monopoly power in some markets. Overall, broadband might be the most important industry in the country because it powers just about everything else. Local jurisdictions around the country regulate occupations like nail salon technicians, plumbers, and masseuses, and yet we can’t get our act together as a country to regulate an industry where a handful of giant ISPs openly manifest monopoly behavior.

There is a really simple fix for this. Congress could give authority to the FCC to regulate broadband so that future FCCs or administrations could not undo it. It would only take a simple law that says something like, “The FCC shall regulate the broadband industry for the benefit of the citizens of the United States.” Obviously, lawyers could word this to be more ironclad – but giving the FCC the authority to regulate broadband doesn’t have to be complicated.

Categories
The Industry

Big Regional Network Outages

T-Mobile had a major network outage last week that cut off some voice calls and most texting for nearly a whole day. The company’s explanation of the outage was provided by Neville Ray, the president of technology.

The trigger event is known to be a leased fiber circuit failure from a third party provider in the Southeast. This is something that happens on every mobile network, so we’ve worked with our vendors to build redundancy and resiliency to make sure that these types of circuit failures don’t affect customers. This redundancy failed us and resulted in an overload situation that was then compounded by other factors. This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls.

In plain English, the electronics failed on a leased circuit, and then the back-up circuit also failed. This then caused a cascade that brought down a large part of the T-Mobile network.

You may recall that something similar happened to CenturyLink about two years ago. At the time the company blamed the outage on a bad circuit card in Denver that somehow cascaded to bring down a large swatch of fiber networks in the West, including numerous 911 centers. Since that outage, there have been numerous regional outages, which is one of the reasons that Project THOR recently launched in Colorado – the cities in that region could no longer tolerate the recurring multi-hour or even day-long regional network outages,

Having electronics fail is a somewhat common event. This is particularly true on circuits provided by the big carriers which tend to push the electronics to the max and keep equipment running to the last possible moment of its useful life. Anybody visiting a major telecom hub would likely be aghast at the age of some of the electronics still being used to transmit voice and data traffic.

I can recall two of my clients that have had similar experiences in the last few years. They had a leased circuit fail and then also saw the redundant path fail as well. In both cases, it turns out that the culprit was the provider of the leased circuits, which did not provide true redundancy. Although my clients had paid for redundancy, the carrier had sold them primary and backup circuits that shared some of the same electronics at ley points in the network – and when those key points failed their whole network went down.

However, what is unusual about the two big carrier outages is that the outages somehow cascaded into big regional outages. That was largely unheard of a decade ago. This reminds more of what we saw in the past in the power grid, when power outages in one town could cascade over large areas. The power companies have been trying to remedy this situation by breaking the power grid into smaller regional networks and putting in protection so that failures can’t overwhelm the interfaces between regional networks. In essence, the power companies have been trying to introduce some of the good lessons learned over time by the big telecom companies.

But it seems that the big telecom carriers are going in the opposite direction. I talked to several retired telecom network engineers and they all made the same guess about why we are seeing big regional outages. The telecom network used to be comprised of hundreds of regional hubs. Each hub had its own staff and operations and it was physically impossible for a problem from one hub to somehow take down a neighboring hub. The worst that would happen is that routes between hubs could go dark, but the problem never moved past the original hub.

The big telcos have all had huge numbers of layoffs over the last decade, and those purges have emptied out the big companies of the technicians that built and understood the networks. Meanwhile, the companies are trying to find efficiencies to get by with smaller staffing. It appears that the efficiencies that have been found are to introduce network solutions that cover large areas or even the whole nation. This means that the identical software and technicians are now being used to control giant swaths of the network. This homogenization and central control of a network means that failure in any one place in the network might cascade into a larger problem if the centralized software and/or technicians react improperly to a local outage. It’s likely that the big outages we’re starting to routinely see are caused by a combination of the  failure of people and software systems.

A few decades ago we somewhat regular power outages that affected multiple states. At the prodding of the government, the power companies undertook a nationwide effort to stop cascading outages, and in doing so they effectively emulated the old telecom network world. They ended the ability for an electric grid to automatically interface with neighboring grids and the last major power outage that wasn’t due to weather happened in the west in 2011.

I’ve seen absolutely no regulatory recognition of the major telecom outages we’ve been seeing. Without the FCC pushing the big telcos, it’s highly likely nothing will change. It’s frustrating to watch the telecom networks deteriorate at the same time that electric companies got together and fixed their issues.

Exit mobile version