Cognitive Bias when Selling Broadband

One of the most successful ways to sell broadband on a newly constructed network is door-to-door sales. I know of numerous fiber overbuilders who have has great success with this sales method. Companies that sell this way all say that some salespeople do better than others.

I’m not going to cover the giant topic of sales training in a short blog, but I’ve been reading about one aspect of the sales process recently. Social scientists have been doing a lot of research into the topic of cognitive bias – the ways that brains take shortcuts to avoid having to do hard thinking all of the time. When somebody answers the door for a sales call, they often automatically react to the salesperson with various kinds of cognitive bias behavior that has to be overcome to make a sale. We’ve all heard that some people are natural-born salespeople, and that means that they have the talent of recognizing cognitive bias. Luckily, this is something that can also be learned.

Here are some examples of the most common kinds of cognitive bias encountered in door-to-door sales.

Attentional Bias. This is the bias where somebody’s actions are affected by the memory of experiences they’ve had in similar circumstances. A salesperson must react quickly if they hear, “I don’t buy from door-to-door salesmen” before the door is slammed in their face.

Confirmation Bias. This is when a person embraces their existing beliefs. Somebody who thinks they already did a great job in picking their current broadband product and provider might not want to admit that they could have done better.

Status Quo Bias. This is a natural desire to keep things the same. This might be the most common objection to buying faster broadband – “If it ain’t broke, don’t fix it.”

Reactance. Reactance is the natural impulse to do the opposite of what you’re told to maintain a sense of independence. In the sales process, this manifests as somebody who perceives the salesperson as pushy and who then reacts to the salesperson rather than to the sales presentation.

Loss Aversion. This is the brain’s natural tendency to fear losses more than gains. This manifests in a fiber sales if the expected installation and conversion process is perceived to be a bigger hassle than the resultant benefit from better broadband.

Bandwagon Effect. This is a bias that can work in a salesperson’s benefit. People tend to be influenced by what their neighbors do, so being able to tell them that their neighbors have already purchased can overcome reluctance. But this can work against a new market entrant if people perceive that sticking with the brand name incumbent is the consensus choice.

Ambiguity Effect. This is the tendency for the brain to avoid scenarios where the outcome is uncertain. Even if customers accept the benefits of faster broadband, they might worry about network outages or the responsiveness of the new provider in responding to customer service calls. Uncertainty can stop them from making a change.

Selection Bias. This is the natural tendency of the brain to notice more of something when it’s brought to their attention. In the sales process, this might mean that a potential customer will assume that your product has many of the same flaws and problems as their current product.

Mere Exposure Effect. This is the tendency for the brain to accept something if they are already familiar with it. This is why companies do brand advertising in a market along with door-to-door sales. A lot of potential customers are a lot more comfortable considering a new broadband product if they are already familiar with the new provider. This is also why ISPs often do well when moving into neighboring communities where many people already know thier name and have heard about them.

How Smart are Promotional Rates?

I think the big ISPs are recognizing the impact that special promotion rates have on their bottom line. Promotional pricing is the low rates that cable companies offer to new customers to pry them away from the competition. Over the years promotional rates have also become the tool that cable companies use to retain customers. Most customers understand that they have to call the cable company periodically to renegotiate rates – and the big ISPs have routinely given customers a discount to keep them happy.

We’re finally seeing some changes with this practice. When Charter bought Time Warner Cable they found that Time Warner had over 90,000 ‘special’ pricing plans – they routinely negotiated separately with customers when they bought new service or renegotiated prices. Charter decided to end the practice and told most former Time Warner customers that they had to pay the full price at the end of their current contract period.

We’ve seen the same thing with AT&T and DirecTV. The company decided last year to eliminate the special discount on DirecTV and DirecTV Now. When the discount period ends for those products the company moves rates to the full list price and refuses to renegotiate. The practice cost AT&T almost a million customers just in the first quarter of this year. But AT&T says that they are glad to be rid of customers that are not contributing to the bottom line of the company. I’ve seen where the CEOs or other big ISPs like Comcast have said that they are considering changes in these practices.

At CCG we routinely examine customer bills from incumbent ISPs as part of the market research of helping ISPs entering new markets. While our examination of customer bills has never reached the level of equating to a statistically valid sample, I can report that the vast majority of bills we see have at least some level of discount. In some markets it’s rare to find a customer bill with no discount.

The discounts must accumulate to a huge loss of revenue for the big ISPs. The big ISPs all know that one of the only ways they are going to be profitable in the future is to raise broadband rates every year. The growth of broadband customers overall is slowing nationwide since most homes have broadband, although Charter and Comcast are still enjoying the migration of customers off DSL. The ISPs are continuing to lose revenues and margins as they lose cable and landline voice customers. Most US markets are seeing increased competition in broadband services for businesses and large MDUs. There’s not much left other than to raise residential broadband rates if the big ISPs want to satisfy the revenue growth expected by Wall Street.

If the big ISPs phased out promotional discounts it would probably equate to a 5% to 10% revenue increase. This is something that is becoming easier for a cable company to do. Many of them have already come to grips with cord cutting, and many are no longer fighting to keep cable customers. Cable companies are also less worried over time about customers leaving them to go back to DSL – a choice that is harder for consumers to make as the household need for broadband continues to climb.

Most ISPs won’t make a loud splash about killing discounts but will just quietly change policies. After a few years, I would expect customer expectations will reset after they realize that they can no longer extract discounts by threatening to drop service.

I’ve always advised my fiber overbuilder customers to not play this game. I ask clients if they really want to fight hard to win that slice of the market of customers that will change ISPs for a discount. Such customers flop back and forth between ISPs every two years, and in my opinion, companies are better off without such customers. Churn is expensive, and it’s even more expensive if an ISP provides a substantial discount to stop a customer from churning. Not all of my client agree with this philosophy, but if the big ISPs stop providing promotional discounts, then over time the need to do this for competitors will lessen.

This is certainly a practice I’d love to see slip into history. I’ve never liked it as a customer because I despise the idea of having to play the game of renegotiating with an ISP every few years. I’ve also hated this as a consultant. Too many times I’ve seen clients give away a huge amount of margin through these practices, giving away revenue that is needed to meet their forecasts and budgets. It’s dangerous to let marketing folks determine the bottom line because they’ve never met a discount they don’t like – particularly if they can make a bonus for selling or retaining customers.

Consider Rural Health Care Funding

One of the sources of the Universal Service Fund that often is forgotten is the Rural Health Care Program. The FCC recently carried forward $83.2 million that was unspent in 2018 into the 2019 funding pool. In June Chairman Ajit Pai proposed to raise the annual cap on this fund from $400 million to $571 million. That’s where this fund would have been today had the original fund been indexed by inflation since it was started in 1997. He also proposes that the cap on this Fund grow by inflation in the future.

I have a lot of clients who help their customers benefit from the Schools and Libraries Fund, but many of them never think about doing the same thing with the Rural Health Care Fund.

The Rural Health Care Program provides funding to eligible health care providers for broadband and voice services. Eligible health care providers must be either a public or a non-profit entity. The funds can be used for entities such as 1) educational institutions offering post-secondary medical instruction, teaching hospitals and medical schools; 2) community health centers providing care to migrants; 3) local health departments; 4) community mental health centers; 5) non-profit hospitals; 6) rural health clinics; 7) skilled nursing facilities; and 8) consortiums of providers that include one or more of the preceding list.

This program is comprised of two programs: The Healthcare Connect Fund Program and the Telecommunications Program. The Healthcare Connect Program provides support for high-speed broadband connections. Eligible entities can receive as much as a 65% discount on monthly broadband bills for services like Internet access, dark fiber, or traditional telco data services. This works a lot like the E-Rate program for Schools and Library program. The health care facility pays the reduced rate for service and the partner ISP can collect the discount from the Universal Service Fund.

The health care providers can also ask for assistance with telecommunications equipment and can use the funds to help pay for the construction of fiber facilities. This funding can be an interesting way for a rural ISP to get some assistance for paying for a fiber route to reach a health care facility (and then use that fiber to also serve other customers).

The Telecommunications Program works a little differently. In that program the health care facility can buy broadband and telecommunication services at rates that are reasonably comparable to rates charged for similar services in nearby urban areas. That’s likely to mean discounts smaller than the 65% in the Healthcare Connect program. Functionally this still works the same and the ISP can collect the difference between the urban rates and the rural rates.

Just like with E-Rate, the health care provider must apply for this funding. But also like E-Rate, it’s typical for an ISP to help prepare the paperwork. The paperwork will feel familiar to any ISP already participating in an E-Rate situation.

It’s obvious that since $83.2 million is being carried over from 2018 that rural health care providers are not all taking full advantage of this program. I see articles all of the time decrying a crisis in rural health care due to the high costs of providing services in rural America. This program can bring subsidized broadband connection to health care facilities at a time when that is likely a welcome relief.

This funding has been available for a long time, yet I rarely hear clients talking about it. I’m guessing most rural ISPs have never participated although there are likely eligible health care facilities nearby. This likely will require some training for potential customers. School and library associations have done a good job at alerting their members that this subsidy exists – but I’m guessing the same has not been done with rural health care providers. An ISP willing to tackle the filings can gain a great customer while also benefitting their community.

Predicting Financial Success

I’m often asked to provide a rule-of-thumb metric to predict the financial success of a broadband business plan. The two most commonly requested metrics are customer density (how many households are needed per mile of road) or the percentage of customers needed (penetration rate) to make a fiber business plan work.

After having done hundreds of feasibility studies I’ve stopped boiling success down to any simple metric – it’s never that simple. The reality is that there are a number of important variables that have a major impact on operating a successful broadband business plan. Every ISP and every market is different, and one or two variables can have a huge positive or negative impact on a given business plan.

Following are the major variables that can make a difference when building fiber. A similar list can be made for deploying fixed wireless or other technologies.

  • Customer penetration rate. An area with low density might still have great financial results if the penetration rates are high enough. I’ve seen expected penetration rates vary from 40% in some large markets to over 90% in markets with no existing broadband. Changing the expected penetration just a few percentage points can have a big impact on cash flow. This is why we think it’s mandatory to do a survey to understand customer interest in fiber broadband.
  • Labor rates. The cost of staffing varies widely across the country and between companies. and there are places where labor costs twice as much as in other parts of the country. Labor costs also include taxes and benefits which vary widely by state and between ISPs. The staffing structure of the ISP also comes into play since companies vary between lean and staff heavy.
  • Borrowing costs. The interest rates and the term of a loan (15-years versus 25-years) can have a huge impact on a fiber project since the size of the borrowing is usually significant. Things that mitigate borrowing costs such using some equity, getting grants, etc. can have a big positive impact.
  • Prices. Broadband prices can have a big impact. We know that most customers will buy the lowest priced broadband that has a reasonable speed. There is a big difference if this primary product is at $50 versus $60.
  • Cost of the Network. The metric I’m often asked about is the minimum number of households needed per road mile. While customer density is an important factor, there are many other issues that can have a big impact. The cost of building fiber varies widely across the country due to some of the following:
    • The mix of aerial and buried fiber has a giant impact.
    • For buried fiber, the type of soil matters, because the presence of rock adds big costs.
    • Again labor rates, meaning the cost of construction crews. We’ve also seen projects that took federal money that had to pay prevailing wages for rural construction that killed the project.
    • The condition of the poles and the effort and cost needed for make-ready can be a huge factor.
    • The difference between building in the power space versus the communications space on poles can be significant.
    • Choosing PON versus active Ethernet can have a difference, with Active E having larger fiber bundles needing more splicing.
    • One of the biggest impacts is the cost of fiber drops – the two important factors are 1) average distance customers are from the road, and 2) who builds the drops (we’ve seen the labor costs for drops vary by several hundred percent).
    • Building in phases versus building as quickly as possible can sometimes make a big difference.
    • Customer density is important, but the above factors can matter a lot more. Density can also be a tricky number. Consider two examples of companies that would have the same average density but significantly different costs: Company A has no towns and the rural areas average 10 households per road mile. Company B includes one decent sized town but is surrounded by big farms but still averages 10 households per road mile.

Clients always want me to predict the outcome of a business plan before we undertake the needed business models. I’ve learned to not predict. I’ve worked on projects that look to be far more profitable than I would have expected and looked at others that don’t look feasible for some reason. As an example, I recently finished a business plan model where it turns out that the existing poles in the new market were nearly unusable and the alternative of going underground was impractical because of rock. This one factor made it hard to justify building fiber in a market that otherwise would have passed the sniff test using high-level metrics.

ISP Economy of Scale

I’ve worked for numerous small communities over the last few years and the first question they always ask me is if their community is large enough to support a standalone fiber ISP business. What they really want to know is if they can somehow operate their own local ISP and still have affordable broadband rates.

The question about how big an ISP must be (in terms of customers) is really a question about economy of scale. The textbook definition of economy of scale is a business where costs decrease per unit through increased output. The ISP business is clearly an economy of scale business since the cost per customer decreases as an ISP adds more customers.

The example I usually use to demonstrate this is to look at the cost of paying for a general manager for an ISP business. The cost of the general manager is what economists would call a fixed cost – it doesn’t vary as you add or lose customers. However, the amount of the general manager salary that must be covered by each customer gets smaller as the customer base grows larger.

A large part of the costs of operating an ISP are fixed like the GM salary. Costs like operating a business office, doing the needed accounting, buying a billing system, etc. are largely fixed. The largest fixed cost is often the debt service required to pay off the cost of building the network.

I have done hundreds of business plans for communities of all sizes and I have developed a few rules of thumb for operating a traditional ISP – one that has the expected number of employees, charges normal industry rates and has to finance the cost of their network.

  • It’s hard to justify a new standalone ISP with fewer than 2,000 customers.
  • Economy of scale kicks in at that point and the business gets more efficient, when measured on a per customer cost up until an ISP reaches between 20,000 and 25,000 customers.
  • After 20,000 customers the cost curve reaches relative stasis – adding customers increases efficiency but also drives additional fixed costs. For example, companies find they need to hire extra accountants; they might need backoffice positions for functions like personnel or benefits management.
  • At some fairly large size, say 100,000 customers, ISPs tend to become less efficient. Large companies tend to become bureaucratic, hire significant middle management and become less functionally efficient as centralize functions and put them into silos.

There are ways defeat the economy of scale curve to some extent and I have clients who used the following ways to be more efficient than other ISPs of the same size:

Reduce Costs. There are various ways to spend less on needed functions than the average ISP.

  • Use existing excess capacity. A City or an electric company can open an ISP and not have to spend money for a business office since they are already operating one. An existing business like an electric company or a telco might be able to use underutilized customer service reps or line technicians without having to hire a whole new staff for a new ISP venture. A utility or city might be able to use the existing billing systems from their water or electric utilities to support the ISP functions.
  • Reduce Expectations. Small ISPs often can save a lot of money by reducing customer expectations. For instance, they might elect to not have repair service on evening and weekends unless there is a real emergency. Small ISPs also don’t need to offer all of the bells and whistles of larger ISPs and can have a simple product offering. A small ISP might elect to not offer cable TV, which for small ISPs usually has a negative margin. This concept can only be taken so far and works best in communities where an ISP is competing against a decent giant ISP.

Avoid Costs. It’s not easy to avoid the cost of being an ISP, but it can be done.

  • Financing from Other Revenues. I have a client with only 600 potential customers that is successful since they decided to fund their network using property taxes rather than having to pay debt from ISP revenues. This allows them to have broadband rates far lower than surrounding communities by avoiding ISP debt payments. I know municipal and cooperatives power companies that have raised electric rates to help cover the cost of an ISP business since that’s what their customers wanted.

Grow Efficiently. The most obvious strategy to beat the economy of scale curve is to get more customers in an efficient manner.

  • Increase Penetration Rates. For a small ISP the difference between a 50% and a 70% penetration rate can be dramatic. Many ISPs become complacent once they generate enough cash to pay the bills while the smart ones continue to sell hard every year to maximize the customers in their footprint.
  • Expand to Another Market. Most small telcos figured this out years ago and operate a CLEC business outside their regulated footprint.
  • Partnering. There are several examples today of cities that have banded together to create a larger ISP. There are numerous cities and electric cooperatives that are partnering with telcos to gain the economy of scale. We are still seeing small telcos getting purchased by larger ones.

A Roadmap to Better Broadband Grants

I’ve been thinking about the effectiveness of federal broadband grant programs. We’ve had three recent major sets of federal grant awards – the stimulus grants of 2007, the first CAF II grants in 2015 and the recently awarded CAF II reverse auctions. We also have an upcoming e-Connectivity grant program for $600 million. I think there are lessons to be learned from studying the difference in the results between these grants. These lessons apply to State grant programs as well as any new federal programs.

Don’t Reward Slow Broadband Speeds. Probably the most bone-headed decision made by the FCC in my memory was handing out billions in CAF II to upgrade rural copper to 10/1 Mbps. This wasn’t considered decent broadband at the time of this decision and yet these upgrades continue to be funded today. The FCC could still take back the remaining CAF II money and redirect these funds to a reverse auction, which we just saw produced much faster speeds in areas with far less density than the CAF II footprint.

Keep Politics Out of It. The CAF II decision to give all of the funding to the big telcos was purely political and resulted in a huge waste of money that could have created many real broadband solutions. The FCC is supposed to be an independent agency, and it’s shameful that lobbyists were able to kill the reverse auction originally planned for CAF II. We are seeing politics back on the table with the e-Connectivity grants where Congress created a feel-good grant program, but then saddled it with a restriction that no more than 10% of homes in a study area can have existing 10/1 Mbps speeds. The reason for this provision was not even hidden, with the big telcos saying they didn’t want federal grant money to be used to compete against them.

Don’t Fund Inadequate Technologies. AT&T is using LTE cellular broadband to satisfy CAF II. This technology will never provide adequate broadband. In the recent reverse auction we saw money going to high-altitude satellite companies. Regardless of speeds that can be delivered with these satellites, the latency is so poor that it limits the ability to use the broadband for important activities like working at home or taking on-line classes.

Don’t Stress Anchor Institutions over People. The stimulus grants required middle mile providers to pop off of highways to build expensive last mile fiber to a handful of anchor institutions – schools, libraries, etc. While these anchor institutions need good broadband, so do the neighborhoods around them. This requirement added a lot of cost to the middle-mile projects as well as made it harder for anybody else to build a last mile network since the biggest bandwidth users in a community already have fiber.

Build to Industry Practices. The stimulus grants required that fiber builders conduct expensive environmental studies and historic preservation studies. That was the first time I ever saw those requirements in my forty years in the industry. Since telecom infrastructure is built almost entirely in existing public right-of-way these restrictions added a lot of cost but zero value to the projects.

Penalize Companies that Cheat. There needs to be repercussions for companies that cheat on grant applications to win the funding. The biggest area of cheating is claiming speeds that the technology can’t deliver. The FCC follows up grants with a decent speed-test program, but the worst repercussion in failing these tests is to not get funding going forward. A carrier that badly fails the speed tests should have to return the original grant funding. I’m also hearing rumors that the many rural households covered by CAF II will not get the promised upgrades – and if so, the big telcos should be forced to return a proportionate amount of that funding for homes that don’t get the promised upgrades.

The New e-Connectivity Pilot Grants

In March Congress passed a new $600 million grant/loan program to build rural broadband. The project has been labeled as the e-Connectivity Pilot and it’s expected that the specific rules for seeking the funding will be released early on 2019. The USDA sought public comments on the program in September and is now working out the details of how the awards will be made.

Anybody interested in these grants should get serious about it now, since it’s likely that the grant application window might not be any longer than 60 to 90 days. Getting ready means having a detailed and solid business plan as well as already having a source of funding for any parts of a project not covered by these grants. The grants are also likely to include provisions like getting a professional engineer to approve the network design – so designs need to be specific and not generic. It’s likely that the USDA will stick with their existing grant application process – and those forms have always been a bear to complete.

There is one huge hurdle to overcome for this program since an application can’t cover an area that has more than 10% of households with access to broadband speeds of at least 10/1 Mbps. Considering that the CAF II awards and more recent CAF II reverse auctions awards already will supposedly provide this kind of speed to huge swaths of the country, there are not a lot of areas left that will meet this requirement.

Claiming that an area meets the 90% unserved threshold will be also be difficult because grant applications can be challenged by carriers that serves the grant area today. I have to assume that CAF II reverse auction winners will also be able to challenge. The big rub is that the original CAF II award winners still have until 2020 to complete their build-out and they will certainly challenge awards for any CAF II area that has not yet been updated. The CAF II reverse auction winners have ten more years to complete their buildout. The USDA will likely be obligated to reject an application that encroaches on any of the CAF II footprint – even if those areas don’t have broadband today.

This gets even more complicated since the CAF II reverse auction awarded funding to fixed wireless and satellite providers. They were funded to serve specific little pockets of unserved homes, but it won’t be hard for them to claim that the CAF II award dollars will allow them to serve much larger areas than the tiny boundaries they bid on.

The process of proving a study area isn’t served will be further complicated by the USDA’s reliance on the FCC’s broadband maps, which we all know to be highly inaccurate in rural America. This all adds up to mean that an applicant needs to prove the area doesn’t have broadband today and will not be getting it over the next decade from one of the CAF winners. They will also need to overcome any errors in the FCC maps. This is going to be hard to prove. I expect the challenge process to be brutal.

From the instant I saw the 90% unserved test, I’ve assumed that the most likely candidates for these grants will be somebody that is already planning on building broadband across a large footprint. If such an applicant is careful to only identify the scattered homes that meet these grant rules, then this funding can help to pay for a project they were going to build anyway. The other natural set of applicants might be those companies that already took CAF II funding – they could use these grants to fill in unserved homes around those build-out areas. The industry is going to be in an uproar if a lot of this funding goes to the big incumbent telcos (who won’t challenge their own applications).

Another issue to consider is that the USDA can award funding as a combination of grants and loans. These awards will surely require matching funding from an applicant. Anybody that is already planning on funding that matching with bank or other financing might find it impossible to accept USDA loans for a portion of a project. USDA loan covenants are draconian – for example, USDA loans usually require first priority for a default, which will conflict with commercial lenders. It’s always been nearly impossible to marry USDA debt with other debt.

rant applicants should also be aware that the USDA is going to be highly leery of awarding money to start-ups or somebody that is not already an ISP. The agency got burned on such grants awarded with the stimulus grants and has indicated that they are looking for grant award winners to have a strong balance sheet and a track record of being an ISP. This will make it nearly impossible for local governments to go after the money on their own. Chances of winning will be greatly enhanced by public/private partnerships with an existing ISP.

I know my take on the grants sound highly pessimistic. Congress saddled these grants with the 90% unserved test at the coaxing of the big telcos who wanted to make sure these funds weren’t used to compete against them. Past USDA grants had the opposite requirement and could consider awards to areas that didn’t have more than 10% of houses with broadband. However, if you are able to identify a service area that can survive the challenge process, and if you have the matching funded lined up, these grants can provide some nice funding. I’m not taking any bets, though, on the USDA’s ability to award all of the money – there might not be enough grant applications that can make it through the gauntlet.

A New Way to Finance Fiber

I recently was part of a team that brought the financing to build fiber in Dallas, Oregon. The new fiber business is operating under the name Willamette Valley Fiber. Dallas is a community of over 15,000 located near to the state capital of Salem. As the title of this blog suggests, this project was funded in what I am sure is a new way for the industry.

The funding uses what might best be described as private activity bonds. This are municipal-like bonds that are distributed in the public bond market. In this case the bonds, and the network, are owned by a non-profit corporation. The primary benefit to this financing structure is that the City doesn’t have to go onto the hook for the new debt – something that many cities are reluctant or unable to do. Building fiber networks is expensive and many cities are unable to tackle the size of the needed debt. In this case, the City of Dallas, while thrilled to be getting the fiber network, is not associated with or a party to the bond financing.

If there is any one hurdle to the financing structure it’s that these are pure revenue bonds – meaning that they only are supported by the revenues of the project. There are no backdrop guarantees by a City or anybody else to support the bonds if the project doesn’t perform as expected. That means that any business plan funded this way must be solid and conservative to make sure that revenues will cover costs. That leads to a few key characteristics for a project to be funding in this way:

  • Bond financing generally will have higher up-front costs than other kinds of financing, but they are usually offset by lower interest rates. The high up-front costs mean this kind of financing is only cost effective for projects the size of Dallas or larger.
  • It’s essential that there are no cost overruns from construction because there is no party, like an underlying City, that can step in to make up for any cash shortfalls. This means that engineering must be done before funding, and that a design-builder must be found that’s willing to build the network for a guaranteed price. This means tying down not only fiber costs, but the costs of drops and electronics.
  • It’s also mandatory to understand the community, and that means doing surveys and other market research to make sure that the community is receptive to buying from a new fiber network. It’s easy to just assume that fiber sells, but one of our products at CCG Consulting is doing surveys and we’ve seen major differences from market to market, sometimes even within the same region.
  • It’s also mandatory to have a cost structure that minimizes expenses. The best way to do that is to find an ISP operator who’s already successfully operating a fiber business. There are significant expense saving when an ISP opens an additional market. The fiber business is largely an economy of scale business and there are huge benefits to an operator for spreading joint and common costs across an additional market.

This means that the best structure for this kind of financing is to find an existing ISP willing to tackle operating the new market. That operator will benefit financially by allocating costs to the new market, and the new venture benefits by lower costs. As an example, if an ISP opens up a new market that doubles their size, the cost for something like the salary of their CFO effectively is halved for the original business as half of the CFO’s cost is allocated to the new market. The new market benefits by getting a CFO for half of the cost compared to hiring one.

In Dallas the operator is MINET, a municipal ISP that is owned jointly by the nearby cities of Monmouth and Independence Oregon. MINET has been effective as an ISP with a market penetration in their own markets of nearly 85%. The Dallas expansion offers the opportunity to double their customer base, meaning that they can allocate a high percentage of existing costs to the Dallas venture – a win-win for both parties.

Our team is interested in developing more fiber ventures that meet the above criteria. I’d like to hear from communities that want fiber and that already know of a nearby quality ISP that would be interested in operating the business.

I’m also interested in hearing from existing ISPs that can meet our criteria. We’re only interested in ISPs with a track record of success. An ISP can benefit two ways from such a venture – they can gain economy of scale and allocate a lot of existing expenses away from their current business. An ISP-operator also can benefit from profit sharing if the new venture is successful.

You can contact me at blackbean2@ccg.comm if you think you have a project that can benefit from this kind of financing.

Fiber in Apartment Buildings

For many years a lot of my clients with fiber networks have avoided serving large apartment buildings. There were two primary causes for this. First, there has always been issues with getting access to buildings. Landlords control access to their buildings and some landlords have made it difficult for a competitor to enter their building. I could write several blogs about that issue, but today I want to look at the other historical challenge to serving apartments – the cost of rewiring many apartment buildings has been prohibitive.

There are a number of issues that can make it hard to rewire any apartment. Some older buildings had concrete floors and plaster walls and are hard to drill for wires. A landlord might have restrictions due to aesthetics and not want to see any visible wiring. A landlord might not allow for adequate access to equipment for installations or repairs, particularly after dark. A landlord might not have a safe space for an ISP’s core electronics or have adequate power available.

But assuming that a landlord is willing to allow a fiber overbuilder, and is reasonable about aesthetics and similar issues, many apartment owners now want fiber since their tenants are asking for faster broadband. There are new ways to serve apartments that were not available in the past that can now make it possible to serve apartments in a cost-effective manner.

G.Fast has come of age and the equipment is now affordable and readily available from several vendors. A number of telcos have been using the technology to improve broadband speeds in apartment buildings. The technology works by using frequencies higher than DSL and using existing telephone copper in the building. Copper wire is mostly owned by the landlord, and they can generally grant access to the telephone patch panel to multiple ISPs.

CenturyLink reports speeds over 400 Mbps using G.Fast, enabling a range of broadband products. The typical deployment brings fiber to the telecom communications space in the building, with jumpers made to the copper wire for customers wanting faster broadband. Telcos are reporting that G.Fast offers good broadband up to about 800 feet, which is more than adequate for most apartment building.

Calix now also offers a G.Fast that works over coaxial cable. This is generally harder to use because it’s harder to get access to coaxial home runs to each apartment. Typically an ISP would need to get access to all of the coaxial cable in a building to use this G.Fast variation. But it’s worth investigating since it increases speeds to around 500 Mbps and extends distances to 2,000 feet.

Millimeter Wave Microwave. A number of companies are using millimeter wave radios to deliver bandwidth to apartment buildings. This is not using the 5G standard, but current radios can deliver two gigabits for about one mile or one gigabit for up to two miles. The technology is mostly being deployed in larger cities to avoid the cost of laying urban fiber, but there is no reason it can’t be used in smaller markets where there is line-of-sight from an existing tower to an apartment building. The radios are relatively inexpensive with a pair of them costing less than $5,000.

It’s an interesting model in that the broadband must be extended to customers from the roof top rather than the basement. The typical deployment would run fiber from the rooftop radio, down through risers and extended out to apartment units.

The good news with stringing fiber in apartments is that wiring technology is much improved. There are now several different fiber wiring systems that are easy to install, and which are unobtrusive by hiding fiber along the corners of the ceiling.

Many ISPs are finding that the new wiring systems alone are making it viable to string fiber in buildings that were too expensive a five years ago.   If you’ve been avoiding apartment buildings because they’re too hard to serve you might want to take another look.

Relying on Cellular Broadband (Part II)

One of my recent blogs talked about the reliability of cellular data as a substitute for wireline broadband. Almost immediately I had an example of a wireless outage shoved in my face. I was in Phoenix at an all-day meeting. When I left at about 4:00 I tried my Uber app and it wasn’t working. The app cycled through but would not find a driver. This was inconvenient because I was standing in the 100-degree sun, so I immediately looked for shade. I tried a few more times. Giving up on Uber I tried Lyft and got the same results. Now I’m figuring a data outage, but since Android phones are sometimes squirrelly, to be safe I rebooted my phone.

That didn’t work and I was standing waiting in hot weather to get a ride to my hotel which was 20-miles away. Uber, Lyft and taxis were out of the question. Luckily my voice was still working, so I called my wife who ordered an Uber for me. But had she not been available I’m not sure how I would have gotten to my hotel. I’m picturing the huge number of other people this also inconvenienced. How many people landed at an airport and couldn’t get a ride? How many people were driving and suddenly lost access to their mapping software? How many businessmen were traveling and couldn’t read or respond to email?

When I got back to a landline connection I looked at the AT&T outage website and it was lit up like a Christmas tree. It looked like the east coast was totally out, but almost every other NFL city also showed an outage. Phoenix, which I knew to be out, didn’t even show on the map as having a problem, and it’s possible that the whole nationwide AT&T network had a data outage. A few days later I checked and AT&T had said nothing about the cause of the outage. Their outage website shows a 17-hour outage that day, without specifying the extent or the reason for the outage.

There is obviously something shoddy in the AT&T national network if an event of any kind can knock out the whole nationwide data network for that long. It’s hard to believe that the company would not have redundant backup for every critical system that is needed to keep the network functioning. There are only a few possible explanations. Possibly some critical component of data routing failed, such as their DNS system that routes Internet traffic for cellphones. The company might also have gone too far with software defined networking and created some new points of failure that could affect the whole network. Or the company had a major fiber cut that feeds the site of one of those key network systems. There is no excuse for any of these possibilities, and a company with nearly 160 million customers ought to have redundancy for every critical component of their wireless network.

I contrast this to the hundreds of companies I know with landline broadband networks. All of my clients worry about total network failure and they work hard to avoid it. Unless they are geographically isolated, most of my clients have redundant routes between their network and the Internet. They generally have redundancy of key routers and switches to keep critical functions operational. Most of my clients have almost no outages that are not caused in the last mile. Local broadband networks are always susceptible to cable cuts in the last mile. But those cuts, by design, only knock out customers who are ‘downstream’ from the cut. It’s becoming extremely rare for my clients to have a total network outage, and if they do they usually take steps to stop it from happening a second time.

The press is in love with wireless right now and there are dozens of articles every month declaring how wireless is our future. Cellphones are going to become blazingly fast and 5G will fill in the gaps where cellular isn’t good enough. I’ve written enough blogs about this that you probably know that I think we are still a number of years away from seeing such wireless technologies.

But this outage makes me wonder about whether people will ever fully trust wireless technologies if they are operated by the big ISPs. The big ISPs are cavalier about network outages and they seem to suppose that their customers will just accept them. If my ISP clients had a 17-hour outage they would have taken steps after the outage to made amends with customers. They would have explained the cause of the outage and talked about their plans to make sure that it didn’t happen again. They likely would have given every customer a day’s credit on their bill for the downtime.

It astounds me that something like this outage could happen. If I was the head of AT&T, heads would have rolled after this was fixed. There is no excuse for a company with a $23 billion annual capital budget to have a network that is vulnerable to a widespread outage. The only reason the company could have such outages is that they don’t place value on redundancy. Until the big ISPs can make their wireless networks as reliable as landline networks I will never consider using them for broadband. I can’t see customers sticking with a 5G network that has a 17-hour outage. Broadband is now critical to many of us and I expect outages to be measured in minutes, not in hours or days.