Continued email problems

Rackspace is apparently a disaster, so if you want to reach me, please use dawsonccg@gmail.com.

It’s amazing how reliant we are on the little things like email that works.

 

 

Email Outage

We are, unfortunately one of the many victims of the RackSpace server crash. Our email has been out all day. If you have something critical to get to me, call or text me at 202 255-7689.

Doug

Large ISPs Continue to Stagnate 3Q 2022

For the second quarter in a row, the biggest landline ISPs in the country are largely sitting still in terms of total broadband customers. The largest cable and telephone companies collectively lost 97,000 customers. But Lumen had a significant loss, and without counting Lumen, the other large landline ISPs collectively gained only 23,000 customers for the quarter.

While the traditional landline ISPs are stagnating, the fixed wireless access products (FWA) from T-Mobile and Verizon are still seeing big growth, having added 920,000 new customers in the third quarter, a quarterly growth rate of over 40%. The wireless home broadband is marketed as having download speeds over 100 Mbps.

The following statistics have been compiled by the Leichtman Research Group, which tracks the broadband performance of the largest ISPs in the country. The numbers for the third quarter of 2022:

 2Q 2022 2Q Change % Change  
Comcast 32,177,000 14,000 0.0%  
Charter 30,328,000 75,000 0.2%  
AT&T 15,452,000 (57,000) -0.4%  
Verizon 7,447,000 35,000 0.5%  
Cox 5,560,000 0 0.0%  
Lumen 4,256,000 (121,000) -2.8%  
Altice 4,290,600 (43,000) -1.0%  
Frontier 2,831,000 4,000 0.1%  
T-Mobile FWA 2,122,000 578,000 37.4%  
Mediacom 1,468,000 0 0.0%  
Windstream 1,175,000 (3,500) -0.3%  
Verizon FWA 1,063,000 342,000 47.4%  
Cable ONE 1,062,000 3,000 0.3%  
Breezeline 707,954 (9,965) -1.4%  
TDS 506,500 5,700 1.1%  
Consolidated 381,912 699 0.2%  
   Total 110,827,966 822,934 0.7%  
         
Total Cable 75,593,554 39,035 0.1%  
Total Telco 32,049,412 (136,101) -0.4%  
Total FWA 3,185,000 920,000 40.6%  

There is a lot to unpack in these numbers:

  • The underlying story for the big telcos is that they continue to add customers on fiber. For the quarter, the telcos lost 685,000 DSL customers but added 550,000 fiber customers. While many of the customers in those counts were converted from DSL to fiber, the fiber growth bodes well for the future of the telcos.
  • Verizon FWA grew to have more broadband customers than Cable One and Breezeline during the quarter, jumping up to become the twelfth largest ISP.
  • The biggest loser on the list is Lumen, having lost 2.8% of broadband customers during the quarter. In next quarter’s numbers, Lumen will have split off the twenty easternmost states to Brightspeed. Breezeline (Formerly Atlantic Broadband) was the biggest percentage loser among cable companies, having lost 1.4% of broadband customers during the quarter.
  • TDS continues to be the fastest-growing landline ISP, having grown by 1.1% for the quarter. Next is Verizon FiOS, having grown by 0.5% for the quarter.

A Study of ISP Billing Practices

Consumer Reports undertook a large study where it solicited broadband bills from customers across the country. The beauty of examining bills is the ability to see what ISPs really charge instead of what they say they charge.

The study is not a statistically valid sample since folks voluntarily submitted bills – but Consumer Reports was able to gather over 22,000 bills and found some interesting things.

Some of the things the analysis found are already widely understood. For example, ISPs that bundle multiple services together don’t disclose the actual price paid for each of the services. This means consumers are at the mercy of the ISP to tell them the revised bill that will come after dropping just one of the services. Companies like Comcast have been using the bundling discount as a cudgel to try to persuade folks not to break a bundle by claiming that all discounts were assigned to whatever service is being dropped.

ISPs offer other kinds of discounts. There are discounts for first-time subscribers who buy a service from the web. There are discounts for agreeing to go paperless or agreeing to auto-pay with a bank debt or credit card. There are discounts that are negotiated with customers who threaten to drop service. As might be imagined, these discounts are all over the board, even within the same ISP. Discounts are the Wild West of the broadband world, with some customers getting much deeper discounts than their neighbors.

The study also documents hidden fees that are not usually disclosed in the advertised rate for broadband. Fees like modem rental can sometimes be avoided by a customer willing to buy a modem, but in some cases, that is not an option. The biggest such fee is the median cost of $16 per month charged by Wave Broadband for a modem. Some ISPs have mysterious fees for broadband which are not explained. The biggest headscratcher is the $7.77 Deregulated Administrative Fee charged by Windstream.

The study was particularly critical of data caps. It highlighted Cox, which charges $49.99 to customers who want a guarantee of unlimited data. Consumer Reports saw one bill where Cox charged a customer $100 in a month for going over the data cap.

One of the most interesting findings is that consumers in zip codes where there is only one fast ISP pay an average of $75, while consumers in places with broadband competition average $65. This is reminiscent of a decade ago when the conventional wisdom was that competition lowers rates by around 15%. This still seems to be the case.

The report highlights Altice (Optimum and Suddenlink) as having the highest rates before any discounts. It lists Sonic, a fiber overbuilder from San Francisco, as having the lowest rates.

The report also highlighted some cases where it found prices to be puzzling. For example, the median prices charged by AT&T for various speeds were 12 Mbps for $63, 45 Mbps for $80, 100 Mbps for $60, and gigabit for $80. The report wonders why AT&T would charge more for 12 Mbps than for 100 Mbps. I have my own theory that the big telcos are milking DSL before it dies while trying to drive people off of copper networks.

There was nothing in this report that is a surprise to consumers who are regularly annoyed and angered by the billing practices of the big ISPs. I’m guessing that the reaction of most folks reading this report is, “At least my ISP isn’t the worse one.”

My 2023 Broadband Predictions

You may have noticed that WordPress decided to halfway publish a draft of this blog on Sunday.

This is my annual stab at predicting the major trends in the broadband industry in the coming year.

FCC Mapping Will be a Mess

This might be the least brave prediction I think I’ve ever made. The first iteration of the new map just came out, and there is a lot to like and hate about the new maps. Early reports, like from the State of Vermont, are that the new maps are pretty far off in identifying the locations that can buy broadband. But the more disturbing issue is that the new maps are showing a lot of broadband availability that doesn’t seem to exist – largely due to the FCC still allowing ISPs to report marketing speeds instead of more realistic speeds. I can’t foresee the maps being useful for counting broadband customers for all of 2023.

Supply Chain Inflation Will Slow

The big supply chain issues that caused price spikes in fiber, conduit, and electronics have peaked, and I don’t think we’re going to see component prices continuing to rise. The only wild card that could still impact electronics prices is the continued bizarreness happening in the Chinese economy. Unfortunately, very few vendors will lower prices even if their own supply chain issues are solved, so we aren’t going to see material costs dropping. There will still be labor rate increases since work crews continue to be in short supply and can command premium rates. Increases in Davis-Bacon wage levels will continue to push up the cost of grant-funded labor.

Cable Companies Will Continue to Aggressively Increase Rates

There was some question about how cable companies would react to the fact that broadband customer growth has stagnated. In the second and third quarters of 2022 combined, the big cable companies collectively lost 20,000 broadband customers. Charter decided to raise rates by $5 on November 1, and I think all of the big cable companies will continue to increase rates in 2023. Rate increases bring far more bottom line benefits than any downside from customer losses. If anything, slower growth might make it more imperative to raise rates to satisfy stockholders.

Only a Trickle of BEAD Grant Cycles will Start in 2023

It’s been over a year since the announcement of the BEAD grants, but I don’t think there will be many BEAD grant applications due in 2023. State Broadband offices can request 20% of the BEAD funding after the NTIA approves a State’s initial broadband grant plan, and some of that funding will become available late in the year. There are still major hurdles for States to get the full funding, including workable FCC maps, States developing a broadband grant plan, States getting feedback from stakeholders, and a challenge of the FCC maps used for broadband (which is different than the current challenge about the map fabric). I predict that the bulk of the BEAD grant program awards will happen in 2024, more than two years after the grant program was announced.

FWA Broadband Will Continue to Make Waves.

While the big traditional ISPs are seeing no customer growth, fixed wireless access (FWA) broadband using cellular frequencies is growing explosively. T-Mobile and Verizon together added 816,000 new customers in the second quarter and 920,000 in the third quarter of 2022. These carriers are still in the process of the widespread deployment of the technology, so  I expect to see big growth continuing in 2023 to the detriment of other big ISPs.

We’ll Finally Get a Fifth FCC Commissioner

I whiffed on one prediction last year. The Senate never mustered enough votes to confirm GiGi Sohn, and the FCC has been a Commissioner short for two years. I don’t know of anything that has changed, but I’m still optimistic that the Senate will finally muster enough votes to seat the fifth Commissioner. That will lead to redefining the definition of broadband to 100/20 Mbps – two years too late, and when it’s already time to be thinking about an even faster definition.

Real Movement on Solving the Digital Divide

Congress created two large digital divide grant programs aimed at tackling the underlying issues that lead to homes not having broadband. This has activated non-profits, local governments, and others to finally start getting broadband into more homes. There will be grant awards made in 2023 from the two grant programs, but I predict that communities are going to move forward with the effort regardless of winning these grants.

Electric Grants and Broadband

The U.S. Department of Energy finally announced the first round of grant applications for funding that was created by the Infrastructure Investment and Jobs Act. While these grants are aimed at improving the electric grid, any projects built with these grants could also build some fiber. The grants will total $13 billion. It’s worth noting that 30% of the funding will go to small utilities that sell no more than 4 million MWh of electricity per year.

There will be $10.5 billion in grants from the Grid Resilience and Innovation Partnership, or GRIP grants. Within the GRIP grant program are three separate programs:

  • $2.5 billion will go to grid resiliency grants to provide infrastructure to improve the survivability of the electric grid from weather-related and other events.
  • There are $3 billion for smart grid grants that can be used for projects that add intelligence to the electric grid.
  • Finally, $5 billion in grants is aimed at grid innovation. This grant is looking for creative ideas for improving the electric grid.

The other grant program is the Transmission Facilitation Program, which will provide $2.5 billion to improve the long-haul electric grid between communities.

The first and immediate round of funding for the GRIP program will be for $3.9 billion, with additional rounds of funding being announced next year. Unlike broadband grants, the first-round GRIP grants are on a rapid timeline. It took the DOE over a year to announce the specifics of the grants, but there are almost immediate deadlines coming. The White House has said that it wants to see more of the infrastructure spending being used, and this timeline will see grant awards made in 2023.

  • Anybody interested in applying for the smart grid or resiliency grants must submit a concept paper by December 16 that explains the proposed project. Concept papers for the innovation grants are due January 13. Concept papers for the transmission grants are due February 1. The DOE will have to accept a concept paper in order for an applicant to move on to the next phase of the grant application.
  • Full grant applications for the smart grid, grid resilience, and innovation programs will be due in March, April and May, respectively.

All of these grants could propose building fiber as part of the solution. Fiber is a way to get more brains into the electric grid and as a tool for making networks more resilient. There is no reason why any constructed fiber from these grants couldn’t serve the dual role of supporting a broadband network.

The short timelines for the first round of funding are going to make it a challenge for anybody that doesn’t already have a grant proposal on the drawing board. It seems unlikely that anybody who hasn’t already done so could create a partnership with an electric company and meet the concept paper deadlines. But electric companies can do this quickly, and I would expect that municipal electric companies and electric cooperatives will propose concept papers that will both improve the electric grid and also improve fiber infrastructure.

These grant announcements are a wake-up call for communities that have not already had discussions about how to improve the electric grid. There is still time to create partnerships for future grant cycles, but the time to start these discussions is now.

According to the DOE, these grants are only the down payment for the funding needed to improve electric grids. Jennifer Granholm, the Secretary of Energy, says the country might need to triple transmission capacity by 2050. Unfortunately, there were insufficient votes in the Senate to approve a large amount of proposed additional funding for the electric grid.

How Good are the New FCC Maps?

The long-promised new FCC maps came out recently, and everybody rushed to see what the maps said about their own home. But a lot of folks looked deeper to try to understand the difference between this and earlier maps. There are two ways to judge the maps – the mapping fabric and the broadband coverage story.

The mapping fabric represents the FCC’s attempt to count the number of locations with or without broadband. They chose to do this by trying to put every potential broadband customer on the FCC map. The FCC hired CostQuest to create the mapping fabric, and the company used a variety of data sources to pinpoint locations on the fabric.

The State of Vermont has already sent a challenge letter to the FCC that says that 11% of the locations in the FCC mapping Fabric don’t match Vermont’s own data. Even worse, Vermont says that 22% of locations it knows about are missing from the FCC map.

I looked at my own neighborhood, which is deep inside Asheville, NC. My neighborhood was established one hundred years ago, and as I expected, most homes are shown on the FCC map. But the FCC maps did not show several new homes that were built here in the last few years. My guess is that the FCC maps generally do better in cities than in suburbs and total areas.

Vermont also looked at the broadband coverage claims by ISPs. According to the new maps, over 95% of Vermont homes have access broadband to broadband of at least 100/20 Mbps. The State created its own broadband maps, which show that only 71% of homes in the state could receive broadband at 100 Mbps or faster at the end of 2021. In looking at the data, the difference seems to come from claims on the new FCC maps that satellite and fixed wireless broadband can reach huge numbers of folks – something that is not true in hilly and wooded Vermont. There also are ISPs that have claimed speeds that are faster than what the State believes is being delivered.

Industry folks have said all along that the new maps are not going to be any better than the old ones if an ISP can claim any marketing speed it wants with no repercussions for exaggerating speeds. There seems to be a lot of work still needed if the new maps are going to be used to allocate BEAD funding to states and, more importantly, to define areas that are eligible for grant funding.

On a local level, the new reporting is interesting. For example, in looking around my city, I can now find the little pockets where AT&T has built a few blocks of fiber. There was no way in the past to be this granular. The new maps are going to drive a lot of folks crazy to see that fiber is only a block away.

The new maps also let me look at Charter’s coverage in more detail than ever before. It’s been said for years that the big cable companies don’t serve everybody in cities, and assuming that Charter is reporting coverage accurately, this can now be verified. I found little pockets all around my city where Charter doesn’t serve. The old FCC reporting by Census block normally showed everybody in a metropolitan area having access to cable company broadband. The new map also shows at the edges of the city how Charter hops over some neighborhoods to serve others that are farther out.

My own house shows a lot of ISP options, some of which don’t really exist. For example, there are several WISPs shown as covering my neighborhood. There is no way that is possible from where the towers are located due to the large hills in the city that creates huge wireless dead zones. The cellular broadband speeds reported were a little more accurate. Verizon doesn’t show coverage at my home at all – which is true since there are also zero bars of voice coverage at my end of the block. I’m not sure why T-Mobile even bothers reporting the 0.2 Mbps speeds – that might be true – but isn’t that really zero broadband? The satellite speeds reported at my house are improbable in a city surrounded by a bowl of mountains and a lot of trees.

For those who haven’t looked yet, here is the new map. I’d be interested to hear from anybody who was surprised by what the maps show for your home.

Does New Technology Thrill You?

Today’s blog is not about broadband, or perhaps only peripherally. As I write this holiday weekend blog, I find myself thinking a lot about an article written last month by Shannon Vallor in the MIT Technology Review. She asks the question, “We used to get excited about technology. What happened?”.

The world is full of new technologies, yet I’ve had the same feeling as Shannon that these new technologies don’t excite me as they once did. She recalls a few technologies that brought her wonder and awe, such as her first ride on the San Francisco BART, seeing a Concorde for the first time, or her first Commodore PET.

We all have our own list of technologies that thrilled us or that we recognized instantly as game changers. My list includes things like Alan Shepard in the first Mercury flight, my first DSL connection that got me off dial-up, online music libraries like Napster and Spotify, and seeing the first iPhone.

The technological breakthroughs I loved the most were good for me or good for mankind. The childhood me saw the Mercury flight as the first step towards mankind expanding our boundaries past this planet. DSL liberated me to finally search the whole world from my living room. Online music meant I was no longer constrained to the music I could afford to buy and could explore the forty different genres of music I like. The iPhone gave everybody a portable handheld computer. The many other technologies I loved at first sight had similar benefits.

The article discusses how a lot of new breakthroughs feel small and somewhat tawdry because they are aimed at helping the companies that sell the technology more than the people who buy it. She cites how farmers feel captive to John Deere because of the way it controls self-driving tractors. She talked about how Roombas and smart refrigerators spy on us  – our transaction with technology companies doesn’t stop when we bring the technology home.

I remember going to Epcot when it first opened. I’m the first to admit that Disney’s vision of the future was schmaltzy, but the vision shown in the Epcot globe is how the history of technology is inexorably tied to making people’s lives better. The century before I was born saw amazing new technologies like electricity in homes, automobiles and planes, refrigeration, vaccines against some of the worst diseases, and mass communications through telegraphs, telephones, and radio.

The article talks about how technology breakthroughs today seem to be more about making the developers rich. If there is any one technology trend I’d like to see undone, it is how we’ve decided to reward companies with breakthrough technology as unicorns and make the founders into instant billionaires. I’m having a hard time getting as excited as I once with space when we’re using the latest technologies to provide private space rides to billionaires. It’s disheartening to see drones becoming the next weapons of war that can threaten us all. It’s disturbing to see vaccines going to wealthy countries instead of everybody. It’s scary that a lot of the electronics we bring into our homes are watching us and reporting back to parties unknown.

However, while I share the same unease as Vallor, I also read a lot about science breakthroughs in labs around the world. We are surrounded by breakthroughs that would have amazed us a few decades ago that barely rate a mention in the press. We’re discovering amazing materials that will enable the next generation of energy use and communications. The breakthroughs in biology are amazing, and we’re probably not far from finding a cure for the common cold and many cancers. We don’t seem to be far away from the first working generation of fusion reactors.

I guess I’m still hopeful, but at the same time, I’ve been thinking about reducing the number of gadgets in my life instead of adding more. I say all of this knowing that I might get thrilled with a new technology announced tomorrow. But then again, maybe I won’t.

Businesses Are Ready for the Metaverse

The latest technology on the horizon is the metaverse, which, stated simply, is the creation of online environments. While the primary focus of the metaverse is to create alternate realities, an application with a possible immediate big uptake is vertical presence for business meetings.

Ciena, a manufacturer of fiber optic transmission equipment, recently did a survey worldwide of 15,000 business people to understand the interests and expectations of the metaverse. Here are some of the most interesting findings from the survey:

  • 96% of businesspeople surveyed recognize the value of holding virtual meetings.
  • 78% of survey respondents said they would prefer an immersive experience over current tools like video conferencing. Many talked about having Zoom fatigue.
  • 71% thought that virtual meetings and the metaverse could become part of the everyday practices for businesses.
  • 40% thought that their business was likely to move from traditional collaboration tools in favor of virtual-based platforms in the next two years.

Of course, the respondents recognized the hurdles to bringing the metaverse into the workplace. 38% thought that network performance would be a challenge and would hold businesses back from using the metaverse. Many respondents also worried that there would be limited availability of high-quality software to operate efficiently in the metaverse.

Overall, the survey showed that there is interest in the metaverse in the workplace. That’s not the same as demand, and companies will likely only embrace the technology if it is affordable and is reliable. I have to admit I am intrigued that so many survey respondents thought that using the metaverse in the workplace is right around the corner.

I’m not sure that a lot of the respondents grasped the network challenges required for using the metaverse in the workplace. A big question that still needs to be answered is how much bandwidth is going to be required to use the metaverse.

There are two visions for the metaverse. One would replace face-to-face Zoom images with avatars. It’s possible that this metaverse would use even less bandwidth than video calls. But this begs the question of whether people want to have meetings with avatars rather than with the actual person. I might be the exception, but I like Zoom calls that let me see the person I’m talking to. I know a lot of people are shy or hate Zoom calls for other reasons, such as having to put on a public face for a function that just a few years ago would have been a phone call. Going from cameras to avatars might be comforting to Zoom haters.

But the other vision of the metaverse in the workplace uses a lot more broadband. This is the concept of telepresence, where a person can feel like they are meeting live with the person at the other end. This might mean beaming the meeting into an actual office or holding meetings in virtual offices and conference rooms. Telepresence is going to require a lot of bandwidth in order to project a real-time hologram of a meeting participant. I also have to assume that a telepresence connection is going to require low latency and jitter.

The big challenge for most of the world is upload bandwidth. Companies aren’t going to make telepresence calls over today’s cable technology. The cable companies could solve this by implementing faster upload speeds, and many are tackling that. But most are not looking at upload speeds that equal the symmetrical speeds on most fiber connections.

The other big challenge for the metaverse is that a lot of employees now work virtually, meaning that companies will have to deal with a wide variety of inferior home broadband connections.

Matching Big ISP Tactics

There are three billing practices that are routine for the large ISPs that smart competitors avoid. First is offering special low prices to attract new customers. The second is bundling, which means giving a discount to customers buying multiple products. Third is what has become known as hidden fees, where there are routine monthly fees that are not included in the online advertised price offers to customers.

A lot of smaller ISPs wonder if they should match these same tactics. The argument for copying the tactic is that it allows advertising rates that can be compared to what the big companies advertise. The main argument against matching these tactics is that the practices are deceptive, and customers have made it clear that they don’t like these tactics. Fiber overbuilders tell me that the first customers they win in a new market are those who feel deceived and mistreated by the bigger ISPs.

Big ISP online advertising has felt sleazy for many years. I wrote a recent blog where Charter in Los Angeles offers customers drastically different introductory rates depending upon neighborhood – with the highest rates being offered to the neighborhoods with the highest level of poverty. It’s common to see broadband specials advertised for less than half of the list price. A customer has to click through multiple levels of footnotes to find out the rate at the end of the special – if it is online at all. It’s not hard to think that somebody could be attracted to low rates without understanding that big increases will be coming in a year or two.

Bundling is an interesting pricing strategy. Customers are given a discount for buying multiple products but are never told which products get the discount. If a customer tries to drop one of the bundled products, they inevitably find that the dropped product had all of the discount and the customer usually ends up paying full price for the products they don’t drop. This tactic is intended to bully folks into not breaking the bundle.

Hidden fees are just plain sleazy. A customer buying an online cable product will get socked with a range of hidden fees on the first bill. While they thought they were buying a $40 cable package, the first bill could easily be $60 or $70. The most common hidden fee for broadband is usually a high rate for the cable modem, which can be over $15 per month. Even more expensive are data caps, which can significantly add to the monthly bill.

The majority of the small ISPs I work with don’t use these tactics. They understand that these tactics are what drive consumers to seek them out. Most of the small ISPs I know have the philosophy of charging the same fair rate all of the time.

But I’ve seen ISPs that start with the simple, fair rate philosophy and get sucked into offering discounts to try to win new customers. Their marketing folks become convinced that matching the big company techniques is the only way to get new customers. I’ll grant that mimicking the big guys is probably the easiest sales technique, but acting like the big ISPs is a poor long-term tactic for many reasons.

  • Promotional rates tell customers that rates are negotiable, and once an ISP goes down that path, customers will ask for breaks forever. Many consumers are used to negotiating with the big ISPs and will do so with the small ISP as well.
  • These tactics tell customers that your rates are too high and that the real rate is the discounted rate. Customers who are too timid to negotiate for lower rates feel cheated.
  • Unplanned discounts can be devasting to cash flows and meeting financial objectives. If your business plan and budgets are based upon a specific set of rates, then giving discounts lowers the average revenue per customer. Do the math and consider what happens if the average revenue for all of your customers drops by $5 or $10.
  • Matching the big ISP tactics also attracts customers who will drop an ISP for a small discount elsewhere. Every few years, they will compare you against the competition and will take the best offer. ISPs with fair rates tell me that they rarely lose a customer to special rates – and that might be because they don’t attract customers who get a thrill out of bartering.
  • Finally, special discounts complicate your dealing with customers. The ISP now has to track when special promotions are finished and notify customers that rates will increase. This likely means having to talk with most of your customers, and calls to the call center will skyrocket. It’s important to remember that most customers view the perfect ISP as one they never need to talk with.

I’m a huge fan of keeping things simple because I have seen so many ISPs that thrive with the philosophy. The danger of mimicking the big ISP tactics is that the public will see you as just another untrustworthy ISP.