Category Archives: Improving Your Business

Seasonal Broadband

The 4th of July is a good day to talk about seasonality, as millions of folks visit their second homes for the holiday. A lot of ISPs operate in seasonal markets where customers regularly spend many months away from the market. Northern states are familiar with the annual migration of snowbirds who flee to somewhere warmer in the winter. These same states often see visitors coming in the summer to visit lakes and rural cabins to beat the southern heat. Any community close to the ocean understands the huge difference between the tourism season and the off-season. One of the most challenging seasonal markets is a small college town where residents leave in the summer to often be replaced in the fall with a new wave of students.

Seasonal visitors are a challenge for ISPs. A lot of seasonal visitors won’t buy broadband if they have to pay full price all year. Selling to seasonal customers has gotten more complicated since some bring a portable Starlink unit with them when they travel.

But seasonal visitors increasingly need broadband. A lot of seasonal visitors want to work remotely while moving between residences. Broadband brings access to video and to the many parts of daily life that are now conducted online. Many seasonal communities also have poor cellular service, particularly during tourist season when cell towers get overwhelmed.

Seasonal customers aren’t all the same. Some want to visit their seasonal property occasionally in the off-season for maintenance or for holidays. Many seasonal customers want to maintain security cameras and perhaps indoor sensors when they are away from the property but hate to pay a full rate for broadband for the months they aren’t there.

I’ve seen a wide range of ways that ISPs deal with seasonal customers.

  • The harshest solution is to bill for every month of service at full rates. This is the general treatment of seasonal customers by some of the largest ISPs.
  • Some ISPs let customers turn off service when they leave the market. There might be a reconnection charge to reestablish service when the customer returns for the next season. But that means no cameras or sensors or broadband when visiting the property in the off-season.
  • It’s also common to charge a reduced rate for the off-season. This allows customers to maintain security cameras and leak detectors without paying a full rate.

Some ISPs are getting more creative.

  • I know an ISP that reduces the fee to an off-season rate but automatically bills for a full month if the customer uses broadband in a significant way more than two days in a month during the off-season. They can do that because of the ability to track customer usage by location on their fiber network.
  • I know several ISPs who offer an off-season safety package. The ISP installs security cameras, burglar alarms, and leak detectors and monitors the devices for activity. They will alert the police if the burglar alarm is triggered. They will send a technician to investigate if a leak detector is triggered. They provide easy access to the owner of stored video camera recordings.
  • I know ISPs that have special rates for rental properties to encourage property owners to offer good broadband as an amenity while making it affordable for the whole year.
  • I know an ISP who has mastered the challenges of operating in a small college town. They swarm the campus when classes start each semester to make sure that students and parents know that they are a cheaper alternative to the cable company. They market a reduced rate for each semester for parents who will pre-pay for the semester. This is popular and saves students from being disconnected for non-payment in the final month of each semester when they often run out of funds.

The goal with seasonal packages is to maximize ISP revenue while not making customers feel like they are being overcharged. An ISP who finds the right package or products for their market will have loyal seasonal customers for years.

Supporting Rural Cell Towers

I work with a lot of ISPs that own rural fiber. Some rural network owners have been successful over the last decade in providing fiber to the cell sites located near their networks. A few sell directly to a cellular carrier, but most of these connections are sold to an intermediate carrier that bundles together cellular connections across a large geographic area.

This has been good business. Depending upon the bandwidth being purchased for the cell site, the monthly fee for providing bandwidth to a cell tower has ranged from $500 to several thousand dollars per month. That’s good revenue for a rural ISP that mostly serves residential customers, and that doesn’t typically have a lot of higher-margin business customers.

I’m hearing about requests from cellular companies or intermediate carriers to increase bandwidth at cell sites. I’ve had four separate discussions about this from different parts of the country in the last month. This makes a lot of sense. All of the statistics say that bandwidth usage on cellphones is growing at an even faster pace than bandwidth usage for households.

There is another phenomenon driving the recent need for more bandwidth at cell sites. T-Mobile and Verizon have added almost 7 million FWA cellular broadband customers in just the last few years and have gained over a 6% market share of all national broadband connections. AT&T and smaller cellular carriers are also launching the FWA product.

FWA broadband is a major upgrade from the cellular hotspots that have been sold in rural America over the last decade. Hotspots hit the news everywhere when school systems and employers tried to use hotspots to provide home broadband during the pandemic, only to find that the bandwidth available was not adequate to meet the demand. FWA cellular is using the new spectrum that the cellular carriers have labeled as 5G, and this has meant a significant increase in speeds – with T-Mobile and Verizon providing speeds between 100 Mbps and 300 Mbps within a few miles of an upgraded cell tower.

These new requests for higher bandwidth create a few issues for local ISPs. Some local networks will have to upgrade electronics to be able to deliver a 10-gigabit connection to a cell site. At the same time, there has been a continuous downward pressure on backhaul bandwidth pricing, and it’s likely that the cellular carriers are going to want multi-gigabit connections at a similar, or even lower cost than what they might have been paying for a gigabit or less today.

The real dilemma is that an ISP that increases bandwidth for an FWA site is enabling a last-mile competitor. ISPs have not worried about competition from hotspots because the speeds are relatively slow, and hotspots have tiny data caps that make them too costly for a household to use in any meaningful way. But FWA cellular is both cheap (in the range of $60 per month) and provides unlimited usage. There is always a significant percentage of customers in any market for whom price is more important than bandwidth –  there is always a market for low prices.

If an ISP provides increased bandwidth to a cell tower, the carrier at that tower might use that bandwidth to capture 10% or more of the customers around the tower. It was one thing to support cell towers when they were used for rural cellphone coverage. But it’s a new equation to be asked to provide faster bandwidth to an ISP that will use the bandwidth to win over local customers.

I’ve been having some interesting conversations with ISPs about this new dilemma. Is the extra revenue from selling bigger bandwidth to cell sites high enough to replace what might be lost from last-mile retail sales? ISPs have never been required to enable competitors to compete against them unless their network was partially funded by a grant. Last-mile ISPs have never thought of cellular carriers as competitors, but suddenly they are.

Part of the answer to the question comes from looking at the likelihood that somebody else will bring the backhaul if an ISP declines to increase the bandwidth. At some cell sites. it might be worthwhile for somebody else to build new fiber or microwave routes to supplement the cell site bandwidth. We are also possibly going to see satellites able to fulfill rural backhaul. The business plan for OneWeb has changed from retail broadband to wholesale backhaul. There are several other satellite companies designing satellite constellations aimed directly at providing cell site backhaul.

I don’t have any easy answer for this, and every local situation is a little bit different. Perhaps FWA competition is inevitable, and maximizing sales to cell sites is a way to offset the coming revenue losses. Most rural fiber network owners have not had to worry much about competition. Every network owner, including those providing fiber broadband, has always seen some customers choose other ISPs. But rural ISPs have never feared losing significant customers. Should a network owner embrace the increased bandwidth sale to towers, or should they drag their feet for a while to push off the time for competition? At what point Do they risk losing backhaul sales to cell sites – since the worst result would be losing both the wholesale and retail revenues?

One of the old truisms of rural broadband is there is almost no real competition once somebody builds a fast rural network. But the advent of FWA wireless and faster fixed wireless radios means that rural markets will have competition, at least in some portion of a network.

How Do You Budget?

I work with a lot of Boards and management teams, and once a year I get questions about budgeting. The questions are more pointed this year because everybody is wondering how to reflect the uncertainty of the market in their forecasts.

For most of the last decade, the budgeting process was relatively easy. An ISP would know its expense structure and there wasn’t much chance of unexpected inflation. An ISP could estimate the cost of a construction project that would be good for three or four years. The hardest decisions that had to be made in creating a budget were operational, such as the timing of when the ISP might add staff or start a new construction project. I once built a budget for a client that was still pretty accurate four years later.

But the easy budgeting days are behind us. Inflation has wrecked budgets for the last two years. Nobody built enough expense into budgets. Practically everything an ISP buys that was not tied to a long-term contract saw cost increases. Construction budgets were particularly gruesome since it seemed like the prices for fiber and electronics was increasing every month. It’s been even harder to get a reliable budget for engineering and construction crews. Anybody that finances with debt has seen interest skyrocketing rates.

Sitting in January 2024, it looks like the uncertainty has calmed a bit for now. The Federal Reserve has said that it hopes to drop interest rates a bit in 2024 – but that is no guarantee. Inflation is still a little higher than pre-pandemic, but it’s impossible to know if world events will cause another spike in costs. Supply chains have largely returned to normal, but we’re finally going to see the impact of BEAD grants across the county hitting on roughly the same time frame. The labor shortage didn’t materialize to the degree predicted in 2023, but BEAD grant construction could cause problems in 2025.

The most common question I get is if forecasts should be conservative or optimistic – and that is not an easy question to answer. Management teams always opt for conservative budgets. They want to be able to beat expectations with owners or Boards, and the easiest way to do that is to hedge the forecast to be conservative. That means predicting slower sales, higher expenses, and higher construction costs than expected. It probably means using the highest interest rate that they think the Board will accept in the forecasts. But Boards often have a different goal. Boards often want to know how well the business might perform, which means looking at realistic or even optimistic forecasts.

In the past there was not as much difference between the two scenarios. In the past, the primary topic of conjecture for ISPs that were growing was the likely customer penetration rates a year into the future. Management budgets were made a little conservative to provide wiggle room to account for unexpected events.

One of the dangers of creating an optimistic forecast is that human nature tends to accept and remember the rosiest scenario and not a more conservative one. Management’s biggest nightmare is to be faulted for not meeting an optimistic scenario due to factors outside its control. There is nothing to be done about interest rates that come in higher than hoped or another round of increased material or labor costs from vendors.

I think Boards need to have a discussion about this topic this year. Boards should be the ones to decide the degree to which they want conservative or optimistic forecasts, or both. Boards should have a discussion at the start of the year about how they will feel if actual performance is better than the conservative forecast but not as good as the optimistic one.

I know Boards that only want to see conservative forecasts. They want to be sure that the wheels don’t come off the business and that there will be sufficient cash to operate even with a conservative outcome. Such Boards typically expect management to beat the conservative forecast, but they don’t usually choose a specific earnings target.

But I’ve also worked with Boards who want an optimistic forecast and who then put pressure on management to meet it. I caution such Boards that putting unrealistic pressure on management in an uncertain environment is unhealthy for the business – but there are many companies that operate under this philosophy.

A Board should be making an open and conscious decision of what it wants to see and how it expects management to perform. Too many Boards don’t convey their expectations to management – and that is not a formula for success.

In-kind Contributions for BEAD Grants

The process of winning BEAD grants is expensive, and grant applicants should do everything possible to lower out-of-pocket costs for winning the grant. Of the most interesting ways to lower the cost of accepting a grant is through the use of in-kind matches. In-kind contributions recognize non-cash benefits of property, goods, or services that will benefit a BEAD project. In-kind matches can be used as part of the process of calculating the matching funds being provided by a BEAD grant applicant, and many other grant programs also allow for in-kind matches.

To use a simple example, if a grant applicant must provide a 25% grant match, then any approved in-kind matches can be used to satisfy a portion of that match requirement. If a grant applicant can justify 5% of the cost of the project as in-kind contributions, then the cash matching in this example would be reduced to 20%.

The BEAD grant process explicitly allows for in-kind matches. The use of in-kind matches for any federal program is described in federal regulation § 200.306 – Cost Sharing or Matching. I must warn you that the federal rules for in-kind matches are confusing, even for accountants.

In-kind matches can be contributed by the grant applicant or by a third party like the local government. The Frequently Asked Questions for BEAD provides a list of examples of costs that might be considered as an in-kind match:

  • Employee or volunteer services
  • Equipment
  • Supplies
  • Indirect costs (this one has me scratching my head)
  • Computer hardware and software
  • Use of facilities
  • Access to rights-of-way
  • Pole attachments
  • Conduits
  • Easements
  • Access to other types of infrastructure

How might an in-kind match work? Here are a few examples.

  • There might already be empty conduits in the BEAD grant area that can be used by the BEAD grant winner to save money on construction. If the BEAD grant winner or some other owner of the conduit, like a local government, doesn’t charge for the use of the conduit, a value can be calculated for the benefit from the conduit and used as an in-kind match.
  • A grant applicant might have already obtained rights-of-ways inside the BEAD grant area. Or perhaps a local government is willing to provide the rights-of-ways for free to encourage the construction of fiber. Some of that past or current value of contributed rights-of-way can be recognized as an in-kind contribution.
  • A grant applicant might have made a significant investment in mapping or engineering software in the past. If it can use that software for the BEAD process without a new large charge, then some portion of the value of the software might be considered as an in-kind contribution.
  • A local government might provide free permitting, location services, or traffic control as a way to encourage the construction of a fiber network. The imputed value of those services at market rates can be an in-kind contribution.
  • An ISP might have multiple reels of fiber already in inventory. The cost of that fiber can be used as an in-kind contribution as long as it is not also billed as material for the construction process.

I must note that these examples are all subject to being approved by the State Broadband Office that will be administering a given BEAD grant. No in-kind contribution is automatic, and the normal process is to negotiate the use and the value of each in-kind contribution with the party awarding the grant. But since it looks like State Broadband Offices are going to be rushing the grant application process, it’s worth claiming as many legitimate in-kind matches as possible to help with the initial grant scoring.

Calculating in-kind contributions is worth pursuing because almost every BEAD grant project will benefit from some existing assets or services that can be classified as in-kind contributions. Every dollar recognized as an in-kind contribution reduces the cash contribution needed for the grant matching.

I’ve also note that there are a number of states that are giving extra grant scoring points for applicants who have local contributions towards a BEAd project. Many local governments will be unable to make cash contributions towards BEAD projects, but it’s well worth talking to them now about contributing in-kind matches.

Bundling Cellular with Broadband

The biggest cable companies have been successful in recent years in bundling cellular service with broadband and cable TV. Comcast and Charter have been at this the longest, but most of the next tier of cable companies are also now offering cellular service.

The cable companies launched their cellular products by operating as an MVNO (Mobile Virtual Network Operator). That’s another industry acronym to remember that means that the cable companies purchase and resell cellular minutes, texts, and data from one of the big cellular carriers.

This MVNO business plan works for cable companies for two reasons. First, they know that a huge percentage of cellular usage is made from home. When  a customer is at home with a cellphone, the outgoing cellular calls, texts, and data can use the customer’s WiFi to connect to the cable company broadband – meaning the cable company doesn’t have to pay for the usage to the underlying cellular company.

The biggest cable companies have also selectively started to install their own cell sites in their busiest neighborhoods to totally bypass the cellular carriers. For example, Comcast purchased a lot of spectrum that can be used for its own cellular service. Over time, they will probably move a lot of cellular traffic directly to their own network – although they will always need the MVNO service to cover customers who make connections outside the reach of a cable company cell tower.

The cable companies have collectively been very successful. They are selling cellular service at a low price, with the primary advantage to use cellular bundling to reduce churn – a customer who wants to drop broadband also has to find a new cellular service.

Smaller ISPs are now being offered the same bundling opportunity. The National Content & Technology Cooperative (NCTC) has been offering white-label cellular service to members. This uses an MVNO arrangement that buys cellular minutes, text, and data from AT&T.

Small ISPs share the same primary advantage as cable companies in that they can hand a lot of cellular traffic through the landline network. However, smaller ISPs who buy this service are not likely to ever be able to find the spectrum needed to directly get into the cellular business with their own towers.

Today’s blog is to warn small ISPs about the risks of this business plan. One of the advantages of having been in the industry for a long time is that I have seen similar arrangements come and go several times over the years. Where the big cable companies probably have the economic power to keep these contractual arrangements for many years, smaller ISPs, even collectively, have no negotiating power with the big cellular carriers.

I refer to the MVNO business as arbitrage. This means that an ISP offering the resold cellular services has zero network to back up the business. The small ISP is completely at the mercy of the big cellular companies to continue the relationship – and that cannot be guaranteed.

I recall twenty five years ago that a lot of my clients had AT&T cellular stores and resold AT&T cellular service – until the day when AT&T decided to pull the plug on the business line and stranded my clients with a big inventory of cellphones. I recall numerous clients that had a similar arrangement with Sprint, and some of them went so far as to jointly build towers with Sprint. That relationship also came to an abrupt end. I recall that even before Spring pulled the plug on the MVNO business, it changed the profit-sharing arrangement to the point where small ISP partners made no profit.

My advice to an ISP that enters the MVNO business is to not make it central to your business plan. Use cellular as a bundling opportunity, but know that it’s almost inevitable that the relationship and product will end some day. It might be two years or ten years, but arbitrage opportunities inevitably come to an end – I can’t recall one with staying power. At some point an executive at the underlying cellular company will decide the profits from the arrangement don’t justify the cost and effort and will pull the plug.

Rural carriers should be particularly cautious about putting their name on a cellular network with poor coverage. In much of the country the cellular coverage in rural areas is abysmal. Putting your brand name on a lousy cellular network can hurt your brand name more than the benefit of picking up the cellular bundle.

I am not recommending that ISPs should avoid the cellular opportunity. If it makes money and helps to sell broadband then give it a hard look. My caution is that a small ISP in an arbitrage arrangement has zero market power, and that the arbitrage opportunity can stop abruptly at any time. The folks trying to talk you into the opportunity probably won’t mention this possibility.

Community-Wide WIFI

Somebody sent me an article from BocaNewsNow that talks about the trend that Hotwire is seeing in communities that want broadband everywhere. Residential communities in Florida are investing in outdoor WiFi networks that allow residents to connect to broadband from everywhere on a property, including tennis courts, lakefronts, and common community areas.

Communities are advertising ubiquitous broadband as an attractive amenity, and homeowners associations are investing in the technology at the prompting of residents.

It’s an interesting idea, but not a new one. Folks might remember the municipal WiFi craze of twenty years ago when cities everywhere were considering installing massive outdoor WiFi networks as a way to provide broadband to everybody. This was such a hot topic that there was even a magazine for municipal WiFi and conventions where folks came to learn about it. The largest such experiment was in Philadelphia, but there were many other cities that tried this on a smaller scale.

All of the early attempts for creating massive outdoor WiFi failed. The main reason for the failure was technical. The technology required deploying large numbers of pole or building-mounted radios that operated in a mesh network. The radios were mounted fairly close to each other so that there was a radio every several blocks in all directions. The advantage of a giant mesh network was that a customer walking around a community never left the network and didn’t have to keep logging in to keep the same connection.

But there was a giant downside that was never solved. The mesh radios constantly communicated with neighboring radios so the network could reconfigure to avoid a faulty or overloaded radio. It turns out that large early mesh networks spent more bandwidth communicating between neighboring radios than in providing bandwidth to users. The whole concept crumbled once a few cities tried this on any scale.

The other issue that killed the idea was that home broadband was improving drastically during this same time period. Speeds were climbing from cable companies and telcos, and folks were suddenly able to buy speeds of 6 Mbps to 12 Mbps, which quickly made the 1-2 Mbps speeds on wireless mesh networks feel glacial.

Over the years, outdoor WiFi technology has improved dramatically like other technologies. Since the early days of the technology, the FCC approved the 5 GHz, and more recently the 6 GHz bands of spectrum for use in WiFi networks. Outdoor hotspots that are fed with significant backhaul can now easily deliver speeds that are adequate for most of the kinds of uses of broadband that would be expected outdoors. Folks can watch videos, join Zoom calls, and use the outdoor WiFi network to stay connected.

Hotwire claims that the demand for outdoor WiFi has also grown due to people now working from home. It’s attractive for employees to take a laptop to the pool or a park rather than be tied to a desk all day.

I’ve talked to a lot of cities that have already expanded or are considering expanding public WiFi to parks and other public areas. The pandemic showed a lot of city officials that there are a lot of folks who need broadband access and don’t have it at home for some reason. It’s one of those amenities that, once you have it, you wonder how you lived without it.

Creating Brand Awareness

ISPs are entering new broadband markets at an unprecedented rate. There have always been ISPs expanding into new markets, but I’m seeing ISP expansion at a far greater rate than ever before. A large percentage of new ISPs are entering markets where they have never served and are operating under brand names that may not be familiar to potential customers. Today I want to talk about some basic Marketing 101 concepts that any ISP entering a new market should be aware of.

It’s easy for somebody bringing fiber to a market to assume that the folks in a new market will flock to the new opportunity – but market research has shown that this is not the case. No matter how much folks might want a better broadband alternative, they also need to be convinced that they can trust the new ISP.

Local ISPs that already operate near a new market have some advantage if folks in the community already know their name. But I think ISPs often overestimate how well they are known in a nearby community – it’s likely that a significant percentage of a community will not know them even if they’ve operated in the region for many years. People tend not to pay attention to brand names for companies and products that are not available to them.

Nielsen has been doing market research for many years and offers up some interesting statistics about the effectiveness of advertising aimed at building a brand identity. The two characteristics that Nielsen says are mandatory for somebody entering a new market are baseline brand awareness and brand recall. These metrics measure how well the residents of any community recognize a brand name (brand awareness) and know what the company behind the brand name sells (brand recall). One other important characteristics is how many folks in a community have a negative opinion of the brand name. This last characteristic defines the uphill battle that the big telcos must overcome in rural markets where a large percentage of folks have an ingrained negative opinion of them.

One of the most interesting statistics from Nielsen is a measure of the effectiveness of brand advertising. In a market where less than half of residents have heard of a brand name, a good brand advertisement can raise awareness of the brand by 8% for those folks that hear or read the ad. The effectiveness of brand advertising decreases with the familiarity of the brand. For instance, if 75% of folks in a community already know a brand, then there is only a 3% bump in positive brand building among those viewing an ad.

These statistics point out a few things that an ISP entering a market should consider. First, how well do the folks in the community already know you? When residents hear your brand name, do they know you are an ISP and think of broadband? Do the residents have a positive or negative opinion of your company? These are things that can be measured in statistically valid surveys.

But many ISPs take the conservative approach that folks do not know their brand name and reputation. Even in communities where a significant number of residents might know them, they feel that they need to build brand name awareness among folks that do not know them.

The relatively low success rate of a single brand-building ad should remind an ISP that it has to find multiple ways to get the word out about them entering the market. The percentage of folks that will see any advertisement is small, and the incremental impact of brand-building only applies to those that see your ads. This means that a new ISP should leave no stone unturned. On top of the normal ad channels, ISPs should work hard to get stories about entering the market in the newspaper, on social media, and local TV. New ISPs shouldn’t miss opportunities to have a presence at any local events where lots of people gather.

The bottom line is that an ISP should assume that most people in a new market don’t know who they are. Even if they do, they probably won’t have an opinion if you are a good or bad ISP. In fact, if the community has experience with poorly performing incumbents, a new ISP should assume that folks will be skeptical of all ISPs.

It’s too easy to bring a fiber network to a new market and assume that folks will automatically flock to it. I’ve known ISPs who are shocked that they don’t get the flood of new customers they expected. ISPs need to take a page from Marketing 101 and build awareness of their brand name and plant the seed that it is not the same as the incumbents. ISPs that can make that connection with the public usually fare well.

Safe Software Upgrades

We’ve had some spectacular recent failures of software upgrades gone wrong. The one that got the most press was a software problem at the FAA that knocked out nationwide flights by corrupting the NOTAM system that transmits real-time information to pilots about flight hazards and airspace restrictions. The FCC said the outage was created when personnel unintentionally deleted files while working to correct synchronization between the live primary database and a backup database.

There seem to regularly be outages of Internet platforms caused by software issues. In the last year, there have been outages at Google, Facebook, Twitter, and dozens of other software platforms. The telecom industry has had plenty of outages caused by similar issues that have knocked out large chunks of the Internet backbone or various data centers. Any of these outages that were software-related have one thing in common – with good software upgrade procedures, the outages likely could have been prevented.

Telephone companies have the longest history working with software upgrades that are capable of knocking out networks. The possibility of big voice outages crept into the industry when we replaced electromechanical switches with electronic switches. The big telephone companies developed software upgrade protocols that were designed to minimize outages due to software upgrades. Even when upgrades went poorly, smart telcos adopted processes for quickly flipping back to the original configuration. The frequency and the size of the software outages we keep seeing today are good indicators that a lot of companies are not following the safe practice that have been around for decades.

One of the first things that anybody that touches a core of a network should understand is that there is no such thing as a casual upgrade or casual maintenance of a mission critical system. It’s obvious there are bad practices in place when one technician can delete or modify a file and cause a major outage – it should be impossible for somebody to have access to casually do that.

The processes for safe software upgrades are well known. They require a lot more discipline than many network engineers want to use – but they are safe. The tried-and-true way to make a software upgrade is as follows:

Have a Project Manager for the Upgrade. It is vital to have one person in charge of the upgrade. They can get assistance in planning and doing the upgrade, but they need to be ready and authorized to react if things don’t go as planned.

Develop a Checklist. There should be a step-by-step checklist of all aspects of the upgrade. Make sure to understand every piece of equipment and software that will be affected by the upgrade. Then, most importantly, develop a step-by-step list of the steps required to perform the upgrade.

Break the Upgrade into Manageable Steps. If possible, the upgrade should be done in stages where progress can be measured and tested after each step.

Establish a Baseline / Establish a Go-Back Process. Establishing a baseline means understanding the current network configuration in detail. It means understanding the exact settings of every piece of software and equipment. Once the baseline is in place there should be a go-back process. This is the process of returning software and hardware to the original configuration if something goes wrong during the upgrade. Ideally, the go-back would be something that can be implemented quickly, and if designed well, can be done in minutes.

Make Sure to have Vendor Support. It’s worth considering having a vendor representative on site for major upgrades, or on alert for minor ones. I have seen clients schedule an upgrade over a holiday, not thinking that the needed expertise at the vendor is probably not going to be available.

Pre-test Every Component before the Cut. Safe practices establish a test lab for a complicated upgrade where the new software and/or hardware is tested first in a lab setting instead of live.

Take Every Upgrade Seriously. I often see companies follow most of the above steps for major upgrades only to see them knock out their network for what they think of as simple upgrades or routine maintenance.

It’s easy to define a bad upgrade process as one where a single technician can unilaterally change files and setting in a mission critical system without going through any of the above processes. Every time there is a bad outage we hear reasons for the outage like a corrupted file or bad hardware – nobody ever admits they were too casual with an upgrade, although that’s probably the real reason for the outage.

Competing Against Big Cable Companies

I’m asked at least twenty times a year how a small ISP can compete against the big cable companies. The question comes from several sources – a newly-formed ISP that is nervous about competing against a giant company, a rural ISP that is entering a larger market to compete, or investors thinking of funding a new ISP. These folks are rightfully nervous about competing against the big cable companies. Comcast and Charter together have roughly 55% of all broadband customers in the country, so the assumption is that they are formidable competitors.

It’s more realistic to say that they are decent competitors. They have slick marketing materials to try to lure customers. They have persuasive online marketing campaigns to snag the attention of new customers. They have good win-back programs to try to keep customers from leaving them.

But the two big cable companies have one obvious weakness – their prices are significantly higher than everybody else in their markets. Every marketing push by these companies involves giving temporary low special prices to lure customers – but those prices eventually revert to much higher list prices.

There is a great example of this in the market today. Both Verizon and T-Mobile have been adding large numbers of broadband customers to their fixed wireless FWA products that deliver home broadband using cellular spectrum. The two cellular companies have been highly successful in the marketplace, adding over 2.6 million new broadband customers through the first three quarters of 2022, while Comcast and Charter added about half a million customers during that same time period – mostly at the start of the year.

The FWA wireless product is clearly competing on price. The FWA broadband is not as fast or robust as cable company broadband, but the prices are attractive to a lot of consumers. For example, T-Mobile offers 100 Mbps broadband for a $50 monthly fee for customers willing to use autopay – a price T-Mobile says will never increase. This is far below the prices of the cable companies, which are in the range of $90 per month for standalone broadband.

I thought I’d take a look at how Comcast is competing against the lower-price FWA products. Comcast has two special offers in January 2023 for standalone broadband.

  • In a special offer that ends February 1, Comcast will provide 400 Mbps broadband for $30 per month, which requires autopay. The special price is under a contract for one year, but the special price extends for two years (meaning that if a customer terminates during the first year they have to pay for the remaining months of the contract). The special price for this product was higher in the past and likely has been lowered to compete against FWA.
  • The other offer is ongoing and doesn’t end on February 1. Comcast will provide 800 Mbps download speeds for $60 per month, which requires autopay. This is also a two-year term, with the first year under a contract.

Comcast then adds hidden fees to the special price. Unless a customer brings their own modem, Comcast charges $15 per month for a WiFi modem, a price that was increase by $1 this month. In many markets, Comcast also has data caps, and customers that exceed 1.2 terabytes of usage per month are charged $10 for each additional 50 gigabytes of data used in a month.

For the 400 Mbps product, a customer who brings a modem and who doesn’t exceed the data caps will pay $30 per month if using a bank debit and $35 per month with a credit card debit. Using the Comcast WiFi modem (which most customers do), raises the monthly price to $45 or $50 – right in line with the T-Mobile FWA product. But the kicker comes at the end of the term when the price, before a cable modem, jumps to $92 per month, and $107 with the modem. The result at the end of the 800 Mbps special is similar, with the price rising to $97 per month before a WiFi modem. Anybody buying the special today must also worry about whatever rate increases Comcast adds to the base broadband price by 2025.

The special prices offered by the big cable companies are alluring – customers can get a significant discount for a year or two. But inevitably, the prices will skyrocket – and in the case of the 400 Mbps special will more than double at the end of the discounted special.

ISPs that compete against the big cable companies have learned that all they have to do to compete is to offer fair prices and wait out the specials. Over time, customers who get tired of the pricing yoyo will come around. ISPs with fiber tell me that customers that come to them from a cable company almost never go back to cable. Customers appreciate fair pricing with no games and a reliable broadband product that delivers the promised speeds – that’s how you compete against the big ISPs.

Should You Be Benchmarking?

As recently as fifteen years ago, I was often asked by many of my clients to help them benchmark their ISP against their peers. By this, they wanted to know if they had the right number of employees for their customer base, if their revenues and expenses were in line with other similar ISPs, if they had too much expense from overheads, etc.

I always tried to help them, and I would gather statistics from other ISPs and share it with everybody who contributed information. But after a while, I found something that didn’t surprise me but which surprised most of my clients. It turns out that ISPs were not particularly comparable. They seemed to differ in most of the statistics that my clients wanted to understand. Interestingly, a lot of the folks with significantly different metrics considered themselves to be successful.

I started to dig into these differences, and that’s when I realized that, at least for relatively small ISPs, that benchmarking and comparing metrics between peers had little relevance to overall success. What I found on deeper examination was something I already knew but could now prove – that every ISP is unique.

Some of the reasons that ISPs differed in metrics was due to the way they purposefully operated.

  • ISPs differed significantly in their commitment to fix customer problems. Some of my clients didn’t react to customer outages after hours and on weekends, while some had a philosophy of not going home for the day until customer issues were resolved.
  • Some clients purposefully kept functions in-house rather than outsource for a lower cost due to a commitment to keep jobs for long-time employees.
  • Some ISPs set rates as low as possible to benefit the public, while others strove to maximize profits.
  • Some of my clients used software to automate processes as much as possible, while others kept the same methods in place that had worked for decades.
  • Some clients spent extra money to have pension plans and top-notch health insurance for employees while others were less generous.
  • Some of my clients were fully leveraged with debt to take advantage of growth opportunities, while others took pride in being debt free.
  • Some clients served highly rural areas where a truck toll to visit any customer was a huge time-eater.

These kinds of differences make it nearly impossible to make a side-by-side comparison of two ISPs, even ones with the same number of customers.

There was a time when rural independent telephone companies shared so many common characteristics that it was possible to gather benchmarking data that they found useful. But once these companies started to build networks outside of the regulated core areas, the companies changed. Growth outside the company meant competing without subsidies and only tackling growth that looked both manageable and profitable.

Today there is such a wide variety of ISPs that it’s even more difficult to compare them. Is it even possible to compare an ISP associated with an independent telco, one started by an electric coop, one created by a municipality, and one that is an overbuilder not associated with any other business?

This is not to say that mall ISPs don’t have a lot to learn from each other – but benchmarking is probably the least useful approach to understanding the business. ISPs all benefit from comparing technical solutions, software systems, marketing techniques, and how they market against large competitors. But none of that is benchmarking and just means small ISPs benefit from sharing information with similar peers.