ISP Economy of Scale

I’ve worked for numerous small communities over the last few years and the first question they always ask me is if their community is large enough to support a standalone fiber ISP business. What they really want to know is if they can somehow operate their own local ISP and still have affordable broadband rates.

The question about how big an ISP must be (in terms of customers) is really a question about economy of scale. The textbook definition of economy of scale is a business where costs decrease per unit through increased output. The ISP business is clearly an economy of scale business since the cost per customer decreases as an ISP adds more customers.

The example I usually use to demonstrate this is to look at the cost of paying for a general manager for an ISP business. The cost of the general manager is what economists would call a fixed cost – it doesn’t vary as you add or lose customers. However, the amount of the general manager salary that must be covered by each customer gets smaller as the customer base grows larger.

A large part of the costs of operating an ISP are fixed like the GM salary. Costs like operating a business office, doing the needed accounting, buying a billing system, etc. are largely fixed. The largest fixed cost is often the debt service required to pay off the cost of building the network.

I have done hundreds of business plans for communities of all sizes and I have developed a few rules of thumb for operating a traditional ISP – one that has the expected number of employees, charges normal industry rates and has to finance the cost of their network.

  • It’s hard to justify a new standalone ISP with fewer than 2,000 customers.
  • Economy of scale kicks in at that point and the business gets more efficient, when measured on a per customer cost up until an ISP reaches between 20,000 and 25,000 customers.
  • After 20,000 customers the cost curve reaches relative stasis – adding customers increases efficiency but also drives additional fixed costs. For example, companies find they need to hire extra accountants; they might need backoffice positions for functions like personnel or benefits management.
  • At some fairly large size, say 100,000 customers, ISPs tend to become less efficient. Large companies tend to become bureaucratic, hire significant middle management and become less functionally efficient as centralize functions and put them into silos.

There are ways defeat the economy of scale curve to some extent and I have clients who used the following ways to be more efficient than other ISPs of the same size:

Reduce Costs. There are various ways to spend less on needed functions than the average ISP.

  • Use existing excess capacity. A City or an electric company can open an ISP and not have to spend money for a business office since they are already operating one. An existing business like an electric company or a telco might be able to use underutilized customer service reps or line technicians without having to hire a whole new staff for a new ISP venture. A utility or city might be able to use the existing billing systems from their water or electric utilities to support the ISP functions.
  • Reduce Expectations. Small ISPs often can save a lot of money by reducing customer expectations. For instance, they might elect to not have repair service on evening and weekends unless there is a real emergency. Small ISPs also don’t need to offer all of the bells and whistles of larger ISPs and can have a simple product offering. A small ISP might elect to not offer cable TV, which for small ISPs usually has a negative margin. This concept can only be taken so far and works best in communities where an ISP is competing against a decent giant ISP.

Avoid Costs. It’s not easy to avoid the cost of being an ISP, but it can be done.

  • Financing from Other Revenues. I have a client with only 600 potential customers that is successful since they decided to fund their network using property taxes rather than having to pay debt from ISP revenues. This allows them to have broadband rates far lower than surrounding communities by avoiding ISP debt payments. I know municipal and cooperatives power companies that have raised electric rates to help cover the cost of an ISP business since that’s what their customers wanted.

Grow Efficiently. The most obvious strategy to beat the economy of scale curve is to get more customers in an efficient manner.

  • Increase Penetration Rates. For a small ISP the difference between a 50% and a 70% penetration rate can be dramatic. Many ISPs become complacent once they generate enough cash to pay the bills while the smart ones continue to sell hard every year to maximize the customers in their footprint.
  • Expand to Another Market. Most small telcos figured this out years ago and operate a CLEC business outside their regulated footprint.
  • Partnering. There are several examples today of cities that have banded together to create a larger ISP. There are numerous cities and electric cooperatives that are partnering with telcos to gain the economy of scale. We are still seeing small telcos getting purchased by larger ones.

A Roadmap to Better Broadband Grants

I’ve been thinking about the effectiveness of federal broadband grant programs. We’ve had three recent major sets of federal grant awards – the stimulus grants of 2007, the first CAF II grants in 2015 and the recently awarded CAF II reverse auctions. We also have an upcoming e-Connectivity grant program for $600 million. I think there are lessons to be learned from studying the difference in the results between these grants. These lessons apply to State grant programs as well as any new federal programs.

Don’t Reward Slow Broadband Speeds. Probably the most bone-headed decision made by the FCC in my memory was handing out billions in CAF II to upgrade rural copper to 10/1 Mbps. This wasn’t considered decent broadband at the time of this decision and yet these upgrades continue to be funded today. The FCC could still take back the remaining CAF II money and redirect these funds to a reverse auction, which we just saw produced much faster speeds in areas with far less density than the CAF II footprint.

Keep Politics Out of It. The CAF II decision to give all of the funding to the big telcos was purely political and resulted in a huge waste of money that could have created many real broadband solutions. The FCC is supposed to be an independent agency, and it’s shameful that lobbyists were able to kill the reverse auction originally planned for CAF II. We are seeing politics back on the table with the e-Connectivity grants where Congress created a feel-good grant program, but then saddled it with a restriction that no more than 10% of homes in a study area can have existing 10/1 Mbps speeds. The reason for this provision was not even hidden, with the big telcos saying they didn’t want federal grant money to be used to compete against them.

Don’t Fund Inadequate Technologies. AT&T is using LTE cellular broadband to satisfy CAF II. This technology will never provide adequate broadband. In the recent reverse auction we saw money going to high-altitude satellite companies. Regardless of speeds that can be delivered with these satellites, the latency is so poor that it limits the ability to use the broadband for important activities like working at home or taking on-line classes.

Don’t Stress Anchor Institutions over People. The stimulus grants required middle mile providers to pop off of highways to build expensive last mile fiber to a handful of anchor institutions – schools, libraries, etc. While these anchor institutions need good broadband, so do the neighborhoods around them. This requirement added a lot of cost to the middle-mile projects as well as made it harder for anybody else to build a last mile network since the biggest bandwidth users in a community already have fiber.

Build to Industry Practices. The stimulus grants required that fiber builders conduct expensive environmental studies and historic preservation studies. That was the first time I ever saw those requirements in my forty years in the industry. Since telecom infrastructure is built almost entirely in existing public right-of-way these restrictions added a lot of cost but zero value to the projects.

Penalize Companies that Cheat. There needs to be repercussions for companies that cheat on grant applications to win the funding. The biggest area of cheating is claiming speeds that the technology can’t deliver. The FCC follows up grants with a decent speed-test program, but the worst repercussion in failing these tests is to not get funding going forward. A carrier that badly fails the speed tests should have to return the original grant funding. I’m also hearing rumors that the many rural households covered by CAF II will not get the promised upgrades – and if so, the big telcos should be forced to return a proportionate amount of that funding for homes that don’t get the promised upgrades.

The New e-Connectivity Pilot Grants

In March Congress passed a new $600 million grant/loan program to build rural broadband. The project has been labeled as the e-Connectivity Pilot and it’s expected that the specific rules for seeking the funding will be released early on 2019. The USDA sought public comments on the program in September and is now working out the details of how the awards will be made.

Anybody interested in these grants should get serious about it now, since it’s likely that the grant application window might not be any longer than 60 to 90 days. Getting ready means having a detailed and solid business plan as well as already having a source of funding for any parts of a project not covered by these grants. The grants are also likely to include provisions like getting a professional engineer to approve the network design – so designs need to be specific and not generic. It’s likely that the USDA will stick with their existing grant application process – and those forms have always been a bear to complete.

There is one huge hurdle to overcome for this program since an application can’t cover an area that has more than 10% of households with access to broadband speeds of at least 10/1 Mbps. Considering that the CAF II awards and more recent CAF II reverse auctions awards already will supposedly provide this kind of speed to huge swaths of the country, there are not a lot of areas left that will meet this requirement.

Claiming that an area meets the 90% unserved threshold will be also be difficult because grant applications can be challenged by carriers that serves the grant area today. I have to assume that CAF II reverse auction winners will also be able to challenge. The big rub is that the original CAF II award winners still have until 2020 to complete their build-out and they will certainly challenge awards for any CAF II area that has not yet been updated. The CAF II reverse auction winners have ten more years to complete their buildout. The USDA will likely be obligated to reject an application that encroaches on any of the CAF II footprint – even if those areas don’t have broadband today.

This gets even more complicated since the CAF II reverse auction awarded funding to fixed wireless and satellite providers. They were funded to serve specific little pockets of unserved homes, but it won’t be hard for them to claim that the CAF II award dollars will allow them to serve much larger areas than the tiny boundaries they bid on.

The process of proving a study area isn’t served will be further complicated by the USDA’s reliance on the FCC’s broadband maps, which we all know to be highly inaccurate in rural America. This all adds up to mean that an applicant needs to prove the area doesn’t have broadband today and will not be getting it over the next decade from one of the CAF winners. They will also need to overcome any errors in the FCC maps. This is going to be hard to prove. I expect the challenge process to be brutal.

From the instant I saw the 90% unserved test, I’ve assumed that the most likely candidates for these grants will be somebody that is already planning on building broadband across a large footprint. If such an applicant is careful to only identify the scattered homes that meet these grant rules, then this funding can help to pay for a project they were going to build anyway. The other natural set of applicants might be those companies that already took CAF II funding – they could use these grants to fill in unserved homes around those build-out areas. The industry is going to be in an uproar if a lot of this funding goes to the big incumbent telcos (who won’t challenge their own applications).

Another issue to consider is that the USDA can award funding as a combination of grants and loans. These awards will surely require matching funding from an applicant. Anybody that is already planning on funding that matching with bank or other financing might find it impossible to accept USDA loans for a portion of a project. USDA loan covenants are draconian – for example, USDA loans usually require first priority for a default, which will conflict with commercial lenders. It’s always been nearly impossible to marry USDA debt with other debt.

rant applicants should also be aware that the USDA is going to be highly leery of awarding money to start-ups or somebody that is not already an ISP. The agency got burned on such grants awarded with the stimulus grants and has indicated that they are looking for grant award winners to have a strong balance sheet and a track record of being an ISP. This will make it nearly impossible for local governments to go after the money on their own. Chances of winning will be greatly enhanced by public/private partnerships with an existing ISP.

I know my take on the grants sound highly pessimistic. Congress saddled these grants with the 90% unserved test at the coaxing of the big telcos who wanted to make sure these funds weren’t used to compete against them. Past USDA grants had the opposite requirement and could consider awards to areas that didn’t have more than 10% of houses with broadband. However, if you are able to identify a service area that can survive the challenge process, and if you have the matching funded lined up, these grants can provide some nice funding. I’m not taking any bets, though, on the USDA’s ability to award all of the money – there might not be enough grant applications that can make it through the gauntlet.

A New Way to Finance Fiber

I recently was part of a team that brought the financing to build fiber in Dallas, Oregon. The new fiber business is operating under the name Willamette Valley Fiber. Dallas is a community of over 15,000 located near to the state capital of Salem. As the title of this blog suggests, this project was funded in what I am sure is a new way for the industry.

The funding uses what might best be described as private activity bonds. This are municipal-like bonds that are distributed in the public bond market. In this case the bonds, and the network, are owned by a non-profit corporation. The primary benefit to this financing structure is that the City doesn’t have to go onto the hook for the new debt – something that many cities are reluctant or unable to do. Building fiber networks is expensive and many cities are unable to tackle the size of the needed debt. In this case, the City of Dallas, while thrilled to be getting the fiber network, is not associated with or a party to the bond financing.

If there is any one hurdle to the financing structure it’s that these are pure revenue bonds – meaning that they only are supported by the revenues of the project. There are no backdrop guarantees by a City or anybody else to support the bonds if the project doesn’t perform as expected. That means that any business plan funded this way must be solid and conservative to make sure that revenues will cover costs. That leads to a few key characteristics for a project to be funding in this way:

  • Bond financing generally will have higher up-front costs than other kinds of financing, but they are usually offset by lower interest rates. The high up-front costs mean this kind of financing is only cost effective for projects the size of Dallas or larger.
  • It’s essential that there are no cost overruns from construction because there is no party, like an underlying City, that can step in to make up for any cash shortfalls. This means that engineering must be done before funding, and that a design-builder must be found that’s willing to build the network for a guaranteed price. This means tying down not only fiber costs, but the costs of drops and electronics.
  • It’s also mandatory to understand the community, and that means doing surveys and other market research to make sure that the community is receptive to buying from a new fiber network. It’s easy to just assume that fiber sells, but one of our products at CCG Consulting is doing surveys and we’ve seen major differences from market to market, sometimes even within the same region.
  • It’s also mandatory to have a cost structure that minimizes expenses. The best way to do that is to find an ISP operator who’s already successfully operating a fiber business. There are significant expense saving when an ISP opens an additional market. The fiber business is largely an economy of scale business and there are huge benefits to an operator for spreading joint and common costs across an additional market.

This means that the best structure for this kind of financing is to find an existing ISP willing to tackle operating the new market. That operator will benefit financially by allocating costs to the new market, and the new venture benefits by lower costs. As an example, if an ISP opens up a new market that doubles their size, the cost for something like the salary of their CFO effectively is halved for the original business as half of the CFO’s cost is allocated to the new market. The new market benefits by getting a CFO for half of the cost compared to hiring one.

In Dallas the operator is MINET, a municipal ISP that is owned jointly by the nearby cities of Monmouth and Independence Oregon. MINET has been effective as an ISP with a market penetration in their own markets of nearly 85%. The Dallas expansion offers the opportunity to double their customer base, meaning that they can allocate a high percentage of existing costs to the Dallas venture – a win-win for both parties.

Our team is interested in developing more fiber ventures that meet the above criteria. I’d like to hear from communities that want fiber and that already know of a nearby quality ISP that would be interested in operating the business.

I’m also interested in hearing from existing ISPs that can meet our criteria. We’re only interested in ISPs with a track record of success. An ISP can benefit two ways from such a venture – they can gain economy of scale and allocate a lot of existing expenses away from their current business. An ISP-operator also can benefit from profit sharing if the new venture is successful.

You can contact me at blackbean2@ccg.comm if you think you have a project that can benefit from this kind of financing.

Fiber in Apartment Buildings

For many years a lot of my clients with fiber networks have avoided serving large apartment buildings. There were two primary causes for this. First, there has always been issues with getting access to buildings. Landlords control access to their buildings and some landlords have made it difficult for a competitor to enter their building. I could write several blogs about that issue, but today I want to look at the other historical challenge to serving apartments – the cost of rewiring many apartment buildings has been prohibitive.

There are a number of issues that can make it hard to rewire any apartment. Some older buildings had concrete floors and plaster walls and are hard to drill for wires. A landlord might have restrictions due to aesthetics and not want to see any visible wiring. A landlord might not allow for adequate access to equipment for installations or repairs, particularly after dark. A landlord might not have a safe space for an ISP’s core electronics or have adequate power available.

But assuming that a landlord is willing to allow a fiber overbuilder, and is reasonable about aesthetics and similar issues, many apartment owners now want fiber since their tenants are asking for faster broadband. There are new ways to serve apartments that were not available in the past that can now make it possible to serve apartments in a cost-effective manner.

G.Fast has come of age and the equipment is now affordable and readily available from several vendors. A number of telcos have been using the technology to improve broadband speeds in apartment buildings. The technology works by using frequencies higher than DSL and using existing telephone copper in the building. Copper wire is mostly owned by the landlord, and they can generally grant access to the telephone patch panel to multiple ISPs.

CenturyLink reports speeds over 400 Mbps using G.Fast, enabling a range of broadband products. The typical deployment brings fiber to the telecom communications space in the building, with jumpers made to the copper wire for customers wanting faster broadband. Telcos are reporting that G.Fast offers good broadband up to about 800 feet, which is more than adequate for most apartment building.

Calix now also offers a G.Fast that works over coaxial cable. This is generally harder to use because it’s harder to get access to coaxial home runs to each apartment. Typically an ISP would need to get access to all of the coaxial cable in a building to use this G.Fast variation. But it’s worth investigating since it increases speeds to around 500 Mbps and extends distances to 2,000 feet.

Millimeter Wave Microwave. A number of companies are using millimeter wave radios to deliver bandwidth to apartment buildings. This is not using the 5G standard, but current radios can deliver two gigabits for about one mile or one gigabit for up to two miles. The technology is mostly being deployed in larger cities to avoid the cost of laying urban fiber, but there is no reason it can’t be used in smaller markets where there is line-of-sight from an existing tower to an apartment building. The radios are relatively inexpensive with a pair of them costing less than $5,000.

It’s an interesting model in that the broadband must be extended to customers from the roof top rather than the basement. The typical deployment would run fiber from the rooftop radio, down through risers and extended out to apartment units.

The good news with stringing fiber in apartments is that wiring technology is much improved. There are now several different fiber wiring systems that are easy to install, and which are unobtrusive by hiding fiber along the corners of the ceiling.

Many ISPs are finding that the new wiring systems alone are making it viable to string fiber in buildings that were too expensive a five years ago.   If you’ve been avoiding apartment buildings because they’re too hard to serve you might want to take another look.

Relying on Cellular Broadband (Part II)

One of my recent blogs talked about the reliability of cellular data as a substitute for wireline broadband. Almost immediately I had an example of a wireless outage shoved in my face. I was in Phoenix at an all-day meeting. When I left at about 4:00 I tried my Uber app and it wasn’t working. The app cycled through but would not find a driver. This was inconvenient because I was standing in the 100-degree sun, so I immediately looked for shade. I tried a few more times. Giving up on Uber I tried Lyft and got the same results. Now I’m figuring a data outage, but since Android phones are sometimes squirrelly, to be safe I rebooted my phone.

That didn’t work and I was standing waiting in hot weather to get a ride to my hotel which was 20-miles away. Uber, Lyft and taxis were out of the question. Luckily my voice was still working, so I called my wife who ordered an Uber for me. But had she not been available I’m not sure how I would have gotten to my hotel. I’m picturing the huge number of other people this also inconvenienced. How many people landed at an airport and couldn’t get a ride? How many people were driving and suddenly lost access to their mapping software? How many businessmen were traveling and couldn’t read or respond to email?

When I got back to a landline connection I looked at the AT&T outage website and it was lit up like a Christmas tree. It looked like the east coast was totally out, but almost every other NFL city also showed an outage. Phoenix, which I knew to be out, didn’t even show on the map as having a problem, and it’s possible that the whole nationwide AT&T network had a data outage. A few days later I checked and AT&T had said nothing about the cause of the outage. Their outage website shows a 17-hour outage that day, without specifying the extent or the reason for the outage.

There is obviously something shoddy in the AT&T national network if an event of any kind can knock out the whole nationwide data network for that long. It’s hard to believe that the company would not have redundant backup for every critical system that is needed to keep the network functioning. There are only a few possible explanations. Possibly some critical component of data routing failed, such as their DNS system that routes Internet traffic for cellphones. The company might also have gone too far with software defined networking and created some new points of failure that could affect the whole network. Or the company had a major fiber cut that feeds the site of one of those key network systems. There is no excuse for any of these possibilities, and a company with nearly 160 million customers ought to have redundancy for every critical component of their wireless network.

I contrast this to the hundreds of companies I know with landline broadband networks. All of my clients worry about total network failure and they work hard to avoid it. Unless they are geographically isolated, most of my clients have redundant routes between their network and the Internet. They generally have redundancy of key routers and switches to keep critical functions operational. Most of my clients have almost no outages that are not caused in the last mile. Local broadband networks are always susceptible to cable cuts in the last mile. But those cuts, by design, only knock out customers who are ‘downstream’ from the cut. It’s becoming extremely rare for my clients to have a total network outage, and if they do they usually take steps to stop it from happening a second time.

The press is in love with wireless right now and there are dozens of articles every month declaring how wireless is our future. Cellphones are going to become blazingly fast and 5G will fill in the gaps where cellular isn’t good enough. I’ve written enough blogs about this that you probably know that I think we are still a number of years away from seeing such wireless technologies.

But this outage makes me wonder about whether people will ever fully trust wireless technologies if they are operated by the big ISPs. The big ISPs are cavalier about network outages and they seem to suppose that their customers will just accept them. If my ISP clients had a 17-hour outage they would have taken steps after the outage to made amends with customers. They would have explained the cause of the outage and talked about their plans to make sure that it didn’t happen again. They likely would have given every customer a day’s credit on their bill for the downtime.

It astounds me that something like this outage could happen. If I was the head of AT&T, heads would have rolled after this was fixed. There is no excuse for a company with a $23 billion annual capital budget to have a network that is vulnerable to a widespread outage. The only reason the company could have such outages is that they don’t place value on redundancy. Until the big ISPs can make their wireless networks as reliable as landline networks I will never consider using them for broadband. I can’t see customers sticking with a 5G network that has a 17-hour outage. Broadband is now critical to many of us and I expect outages to be measured in minutes, not in hours or days.

Fiber Networks as Collateral

One of the challenges of getting a new fiber network funded is to satisfy all of the requirements of bankers in order to make them comfortable to make a loan. In my experience one of the hardest hurdles for traditional bankers to overcome is their desire to have safe collateral for every loan. Collateral is typically a hard asset that can be liquidated to pay off the loan should the borrower be unable to do so. Bankers are used to lending into businesses where there are hard assets they know are good collateral. Loans for assets like buildings or large construction equipment are considered safe since the assets can be easily resold. Unfortunately, fiber networks don’t make for good collateral.

I’ve worked with half a dozen different clients recently where the bank asked this question. The bankers want to understand the ‘value’ of an installed fiber network should the loan go into default. It doesn’t take much web research for them to find failed fiber projects where the fiber was sold for pennies on the dollar. The truthful answer to the question is that once installed, fiber has little intrinsic value as an asset.

Companies without good collateral assets have a much harder time borrowing money and this issue is not unique to fiber network. For example, firms that sell labor hours like engineers, CPAs and lawyers often encounter the same resistance with banks when trying to buy money to grow.

The right response to the collateral question is to convince the bank that they are asking the wrong question. The value in a fiber network is not in the fiber, it’s in the revenues that are generated from that fiber. Some bankers understand this and I know several fiber overbuilders who have convinced a bank to provide a cash-based line of credit. These lines of credit are typically short-term loans that can be used perpetually to build fiber. The borrower draws the funds to build fiber and repays the loan as quickly as possible as revenues are generated. Lines of credit keep renewing as the borrower pays down the balance. Over time, as the cash flows of the business grows, banks are generally willing to expand the borrowing limit, enabling the fiber builder to build faster. Eventually the loans get too large to qualify for a line of credit, but by this time the business has grown to the point where they can qualify for more traditional loans based upon their balance sheet.

I always try to look at a loan from the banker’s perspective. Building fiber sounds risky. It’s not hard for a banker to go to Google and find numerous fiber projects that didn’t pan out as promised (starting with Google Fiber!). The are no immediate revenues from that fiber until it’s completed and even after that there is no guarantee that any given fiber build will generate enough revenue to cover the debt payments. A banker has little choice but to consider building fiber to be a high-risk undertaking.

The way to overcome the perceived risk of fiber is to package the loan request in a form that bankers will understand. Bankers understand cash more than anything else, and so the best way I’ve found to avoid the collateral discussion is to focus the whole loan discussion on cash flows. The best way to make a banker feel safe about revenues is to pre-sell to customers. It’s a lot easier to ask for money for a specific fiber project if the bank can see a guaranteed revenue stream.

It’s important to remember that bankers don’t understand the fiber business. During the course of a given month they might consider loans from a dozen different industries and they can’t possibly understand the nuances of each one. I have clients who can’t understand why bankers aren’t wowed by their business plan projections – the simple reason is that they have no basis for knowing if the assumptions made in the projections are good or bad. When companies borrow from an industry-friendly place like CoBank their loan application is reviewed by somebody who understands our industry – but a local bank can’t be expected to ever understand enough about the broadband business to trust a projection.

This means that a borrower needs to package a loan request in terms that a banker will understand. All banks understand cash flows and they will be most impressed by a demonstration of sufficient revenues to make loan payments. A loan application that is boiled down to such simple terms has a much higher likelihood of being considered. If you instead let the loan discussion go down the rabbit hole and concentrate on collateral than you are likely not going to get the loan – because fiber networks make lousy collateral. You’ll be more successful by concentrating on things bankers understand, like cash flow, than you will be in trying to convince them to agree with your awesome business plan. Bankers hear all day about can’t-fail opportunities and they know many of them fail.

How Safe is Your Network?

Last week Comcast suffered a major broadband outage. The worst imaginable set of events occurred when there two simultaneous fiber cuts on major legs of their backbone – one between Chicago and New York and one between Ashburn, Virginia and South Carolina. In case you don’t know, Ashburn is the home of the major Internet POP serving Washington DC and surrounding cities.

This is a network planner’s worth nightmare. Planners always try to build redundancy into fiber routes so that the network won’t crash from a single fiber cut. Modern backbone electronics can be set to automatically forward traffic in both directions around a ring so that service isn’t interrupted in the case of a fiber cut or failure of ring electronics somewhere along the ring. But rings using this technology can’t withstand two simultaneous cuts.

What was a bit surprising to me was the failure of a large part of the Comcast network with fiber cuts that were so far apart. It seems unlikely that the company has a fiber ring that sends all Internet traffic in such a large circle. It’s more likely that the company has centralized one or more of their routing functions, such as DNS routing in one place on the network and the fiber cuts might have isolated that key function, which would shut down their Internet product.

Redundancy is a big concern for most smaller network owners. Lack of redundancy was one of the major issues that drove Cook County, Minnesota to build their own fiber network. There is no cable provider in the county and their entire telecom network was provided by CenturyLink. Tourism is the major driver of the economy and a decade ago there was a cut in the CenturyLink fiber from Duluth that isolated the county during peak tourist season. That meant that the Internet, telephones, and cell phones didn’t work. Businesses couldn’t take credit cards, restaurants and hotels couldn’t take reservations, and family members on vacation couldn’t communicate with each other. This prompted the County to pursue a fiber network that included creating redundancy traffic in and out of the county. The network was ultimately built and operated by the local power cooperative, and today there is greatly reduced chance of a major telecom outage.

Even where there is redundancy there can be outages. One of my clients operates a large statewide fiber network that stretches for hundreds of miles. They followed good engineering practice and scheduled an upgrade of the ring electronics after midnight. While one of the nodes was being upgraded the fiber was cut on a different part of the network when a truck knocked down a telephone pole, and the whole network went dark. Fiber cuts in the middle of the night are somewhat rare, but they happen.

Whenever possible fiber engineers also build redundancy into a local fiber network. They might build a ring connecting the fiber huts serving neighborhoods so that the network keeps functioning with a cut along the ring. It’s nearly impossible to design such redundancy in the last-mile loop, but fiber cuts in the last mile only isolate homes associated with the specific fiber.

But just like with Comcast and Cook County, many local networks have a hard time creating redundancy outside of their immediate network. In geographically remote areas it’s often impossible to find a second secure route to the Internet, leaving a network, or whole communities vulnerable to a fiber cut somewhere outside their area.

Unfortunately, it’s getting easier for fiber providers to run into the same kind of issue that hit Comcast. We are migrating numerous functions to the cloud and having redundant fiber routing does not always mean that there is an automatic redundant connection made to a key cloud server. I have clients that are now relying on the cloud for all sorts of services such as VoIP, cable TV programming, DNS routing for the Internet, the use of cloud-based operational software, etc. These ISPs may have a redundant path to the Internet, but still have only one path to get to the company providing their cable TV signal or DNS routing.

The Comcast outage should prompt companies to look again at redundancy. Don’t assume that every function in the cloud is redundant even if you have a redundant connection to the Internet.

The Value of Persistence

One of the most common questions I am asked by those getting ready to build a fiber network is “How can we know that we will get enough customers to make this work?”. They are always hoping there is some magic pill that will let them gain a huge market share to assure their success. Over the years of watching clients launch fiber into hundreds of markets I can assure you there are no magic pills. I’ve come to the conclusion that the two biggest hurdles in the business are first, getting the new network financed, and second, selling to customers.

But there are traits of successful fiber overbuilders that can be duplicated.

Pre-sales campaign. Fiber overbuilders need to take advantage of the one-time buzz that happens when you first bring fiber to a neighborhood. Somewhere between 3 and 6 months before network launch you need to blitz neighborhoods to let them know fiber is on the way. Ideally you will touch every potential customer, and this means a personalized sales approach. This means throwing neighborhood events like ice cream socials or cook-outs. It means distributing door-hangers followed by knocking on every door to let customers know that fiber is on the way.

There is a well-understood maxim that any fiber overbuilder can get 30% of a market just by showing up. There are always customers that either are hungry for better broadband or who simply hate the incumbent providers. An aggressive pre-sales campaign will attract these customers by letting them know you are coming. But pre-sales will also attract another 10% or more of the market, meaning an initial penetration on day-one of 40% or even much higher.

It’s easy to forget that the easiest thing for any potential customer is to do is nothing. If your sales approach is passive – mailers and newspaper ads, you won’t overcome customer inertia that will make it easy for many potential customers to do nothing and to keep the ISP they are already using.

Knowledge Sells. The most successful fiber overbuilders send out knowledgeable salespeople to knock on doors and talk to potential customers. The key word there is knowledgeable. I’ve seen numerous companies hire temporary salespeople, give them a few hours of training and then wonder why they aren’t selling. Successful companies retain permanent salespeople or even send out their own staff to sell door-to-door. When you engage a customer in person it’s essential that the salesperson can comfortably answer all questions about products, prices, the technology and can clearly talk about why a potential customer should switch service.

So don’t take the cheap and easy path of sending out college kids in the summer to sell your network. A knowledgeable sales team will be two or three times more effective at closing sales -so if you are going to make the effort to send out salespeople, send out the right ones.

Persistence. The number one key to long-term success is persistence. Too many fiber builders will blitz a neighborhood one time when it’s first built. They move on to blitz the next new neighborhood and revert to using passive sales techniques in the older market. Such neighborhoods might slowly gain customers over time, though growth often comes as much from word-of-mouth from existing customers as it does from passive sales techniques.

Smart ISPs are persistent. I have clients who knock on doors every year. They have realistic expectation of adding perhaps 2% or 3% per year from a door-knocking campaign in a mature market – but over a decade that translates into a 20% to 30% higher market penetration than they might have had using passive sales techniques. Persistence can pay off for any kind of ISP – I know both commercial and municipal ISPs who have grown to customer penetration rates north of 75% by plugging away at sales year after year.

I know many fiber overbuilders languishing with 40% to 50% penetration rates while having a superior network, better prices and better customer service. They are suffering from what I’ve always called the ‘build-it-and-they-will-come’ mentality and they think their obvious advantages will somehow draw customers to them. What they haven’t done is make the effort to look each potential customer in the eye and explain those advantages. I highly recommend that the general manager and top executives of every ISP take a few days every year to knock on doors. They will quickly learn that a lot of households never heard of their company even though they have had fiber in a neighborhood for years. It’s a humbling experience that quickly demonstrates the value of talking to prospective customers in person.

How to Talk to Bankers

I spend a lot of time assisting clients in finding financing and in doing do I’ve learned a lot about what bankers are looking for from any prospective borrower. Here are some of the key takeaways I’ve learned over the years from talking to bankers:

Be Ready with a Worst Case Scenario. Borrowers invariably create a rosy best-look business plan to demonstrate how well they will perform with the borrowed money. But bankers have learned from hard experience that things often don’t go as well as planned. While bankers certainly want to see the optimistic business projection they are more interested in your worst case scenario, so a smart borrowers will prepare a worst case scenario along with the best case one.

The bankers wants to hear about everything that might go wrong with your plan – project delays, slow sales, higher than expected cost of construction – and then to understand how the borrower plans to cope with each potential snag. They want to be shown that the borrower will be able to repay the loan even if things go wrong. A banker is going to be far more impressed to see ta plan that considers the challenges and has a solution for every contingency. I’ve seen bank loan applications fail when the borrower was unable to answer simple questions about how their plan might fail.

Don’t Talk in Acronyms. Telecom borrowers are invariably highly technical people who understand the nuances of building and operating complex networks. The banker can understand this by looking at your credentials, experience and references. Most bankers don’t want to hear about the detailed nuts and bolts about how the technology works, and they are going to care a lot more about the products to be sold on the network and the plan to effectuate those sales.

I’ve sat on meetings and calls between borrowers and bankers where the response to a simple technical question elicits a fifteen minute spiel on the nuances of the technology. Bankers are not impressed with this, and in fact it can be worrisome if it they perceive the borrower as somebody who can’t explain their business in plain English – because the banker will know that’s what customers are going to want to hear. My advice is to tone down the technology unless the banker specifically wants to hear the details.

Understand the Market. I’ve had numerous clients over the years who have had the philosophy of “build it and they will come,” meaning that they were so positive of the superiority of their proposed network that they just assumed that people will buy their products. The vast majority of the business failures I’ve seen over the years were due to this blindness of the market.

Bankers are going to want to see evidence that people are ready to buy from the new network. In larger markets that might mean a statistically valid survey. In smaller ventures that is going to mean pre-sales and having a list of customers who are ready to buy service. Bankers also want to see a comparison of proposed prices and the prices of the competition  – I am often surprised by proposed new ventures that haven’t taken the time to look at actual customer bills in their proposed market. Do the homework and make the effort to understand the market before asking for funding.

Understand What Bankers are Looking For. Every lender is different and early in the process you need to ask them how they will judge your loan application. I recommend a two-stage process for getting a loan. The first meeting should be to understand what the bank is looking for. Have the banker describe the borrowing process and then use a second meeting to make a formal presentation of the business plan in a way that meets their requirements.

If the bank is interested primarily in collateral, then walk into the presentation ready to talk about that. If they are more focused in seeing a business plan that meets some set of financial metrics like debt service coverage ratio, then walk into the presentation ready to answer those questions.

Bankers talk in lingo just like our industry, and it’s vital to make sure that you understand what they want from you. I’ve seen many borrowers who don’t understand a bank’s requirements and who then never answer the basic questions the bank asks of them. It’s not uncommon for a borrower to be intimidated by the banking process and to be afraid to show that they don’t understand the banking lingo. In the end, if you don’t understand what your banker wants, then it’s likely you won’t be making the right proposal and the chance of getting a loan are greatly diminished.