Banking Challenges for Fiber Builders

I’ve often mentioned in this blog that it’s gotten harder to finance fiber infrastructure. Today I want to discuss a few of the specific issues that fiber builders face when trying to find bank financing. There are two traditional sources of funding for the industry – the Rural Utility Service (RUS) and CoBank.  However, many fiber builders don’t qualify for this funding since both institutions favor established mature companies. Any company that doesn’t fit the profile of these two lenders must turn to the only other source of funding – local and regional banks. Following are some of the issues I see when trying to borrow from banks.

Familiarity with the Industry. Local banks often are leery about lending to telecom companies because they are not familiar with the business and they fear lending into an unknown industry. Local banks are much more comfortable lending to businesses they understand and make loans to car dealers, retail stores and the other kinds of local businesses that have been their long-term core borrowers.

Amount of Borrowing. Every bank has some pre-determined maximum amount they are willing to lend to any one borrower and it’s easy for a fiber overbuilder to quickly hit this limit. I’ve rarely met a fiber overbuilder who doesn’t see endless opportunities for expansion and it’s not hard to hit a bank’s maximum lending limit.

Loan Terms. Local banks are often uncomfortable with the longer-term loans needed to finance fiber.  Banks prefer to make loans for relatively short periods of time, with their preference being short loans of 2 – 5 years. Fiber builders are often forced to only chase projects that fit the short loan terms – which means cherry picking only the best opportunities. In doing so they will be passing up opportunities that would thrive and produce good returns with a longer loan terms of 5 – 15 years.

Collateral. Banks are often uncomfortable with a fiber network as collateral. It’s not hard to blame them for this. A fiber network, once in the ground or on the pole does not automatically have a liquidation valley equal to the cost of the construction. The real value of a fiber network is the revenues from customers who are added to the network – and banks have a hard time accepting this concept. A little research will show bankers that failed fiber ventures have often liquidated the physical fiber network for pennies on the dollar, and that rightfully frightens them.

Quantifying Risk. It can be difficult for a bank to understand the downside risks of building a fiber project. One of the key steps to making a loan is to understand the likelihood of the borrower not meeting the proposed business plan, and bankers have a hard time quantifying and getting comfortable with the potential downsides of the proposed business.

Meeting Metrics. Many banks are driven by metrics – meaning that they look for key financial performance metrics from a borrower. It’s hard to meet the typical metrics for a new fiber network. When a network is first built it boosts the balance sheet – but revenues then lag a few years behind until the new network has enough customers to meet expected metrics. This cycle of early losses followed by eventual gains does not fit easily into the expectations of a metric-driven bank.

Unfortunately, any one of these issues can convince a bank that the fiber loan is too risky or doesn’t fit their comfort zone. Many banks are comfortable with infrastructure loans, but there are infrastructure loans that better meet their expectations. Consider a loan to build an apartment complex. There is the same period of zero revenues while the buildings are constructed, but the expectation is that the borrower will then quickly reach full revenues within a relatively short period after the end of construction. An apartment building also provides comfortable collateral because there is an established market for selling repossessed buildings. Bankers in general understand the apartment complex operating model and are comfortable with the variables of operating an apartment building.

Fiber overbuilders need to be prepared to tell a story that can get a banker comfortable with each one of these concerns. I always advise fiber builders that they must put themselves into the banker’s shoes and look at their own business plan as a skeptic. I’ve often seen fiber builders who point to a business plan that eventually makes a lot of money and who can’t understand why a banker doesn’t see their plan the same way they do. Many of the misgivings that a banker might have about funding a fiber project are legitimate and the borrower must convince the banker that the overall level of risk is small – a tall task.

Migrating the Voice Switch

We’re seeing switch vendors ending support for the first generation of soft switches and this is forcing a lot of companies to consider how to continue safely operate their telephone products. Almost all of my clients are seeing an erosion of their voice customer base which makes it hard to justify investments in new switch hardware.

There are alternatives to buying new hardware that should be considered before ponying up for a new switch. Some of the options include:

Migrating to the Cloud. There are numerous options for migrating some or all of your voice services to the cloud.

The simplest migration path is to use a call feature server in the cloud. This is the device that supplies all of the voice features and is the core of a soft switch. A migration to the cloud will eliminate the call feature server hardware and software at the carrier end and replace it with a lease for the capability from a cloud-based vendor like Alianza / Level 3.

At the other extreme you can abandon the whole voice switch and move everything to a cloud-based VoIP service. This eliminates the switching hardware and software, and also eliminates the cost of interconnection and the purchase of things like SS7. There are options between the two extremes and it’s possible to outsource only some switch functions.

Sharing with a Neighbor. I’ve been preaching for years that neighboring carriers ought to partner for voice services. The typical voice switch is capable of processing huge volumes of calls and there can be significant savings when companies share the cost of the core hardware, software, interconnection costs and technician labor associated with a softswitch.

What to Watch Out For. There are possible gotchas to look out for in any switch migration. For example, a carrier that still relies on access charge revenues needs to be careful that a transition doesn’t drastically change access billing. Obviously losing access revenue is a negative, but a migration that drives access charges higher can also be negative and can draw challenges and possibly even bypass by long distance carriers.

Another wrinkle to be aware of is the ability to maintain special switching arrangements like EAS or regional long distance plans that are mandated by regulatory bodies. With good planning such arrangements can be accommodated – but address them up front.

Traditional ILECs also need to be aware of changes in settlements. Switching subsidies and related access charges have largely been phased out, but any change to rate base and access billing is something that should always be run past settlement consultants.

If planned properly and with a little creativity a carrier can save money by outsourcing switching while still meeting all regulatory requirements including network structures like host/remote complexes and even the tandem function. But if done poorly a carrier can put related revenues at risk while possibly messing up the ability of customers to make calls.

I don’t normally use this blog to directly market CCG Consulting services – I know I rarely read marketing blogs from others. But this kind of migration has hidden perils to those who aren’t careful – if you are going to do it, then make sure it’s done perfectly. There are so many moving parts in a switch migration, and often a lot of dollars at stake that you must get it right the first time. The CCG staff has migrating and upgrading switches for decades and we can help you to save money on your switching function while maintaining cash flows and meeting regulatory requirements.

 

Selling Wholesale 5G

Frontier announced the other day that it was interested in selling off much of the Verizon FiOS networks it had recently acquired in 2016. Apparently, the company is over-leveraged and needs the cash to make a healthier balance sheet. But regardless of the reason, that puts a sizable pile of last mile fiber networks onto the market.

I read a summary of a report by Cowan Equity Research that suggests that there is increasing value for fiber networks now based upon the potential for selling wholesale connections to 5G providers. As I think about this, though, I’m betting that a lot of fiber network owners will be extremely leery about allowing 5G providers onto their networks.

Without looking at the Frontier specifics, consider an existing last mile fiber network that already passes all, or nearly all of the homes and businesses in a community. Every fiber business plan I’ve ever created shows that any last mile fiber network requires a substantial customer penetration in order to be financially viable. The smaller the footprint of the network, the higher the needed customer penetration rate.

Consider how a 5G provider would gain access to an existing fiber network. They’d want to gain access for each 5G transmitter and would pay some fee per unit, or else a fee to lease the whole network. That fee would have to be low enough for the 5G provider to make a profit when selling broadband. I’m guessing that the Cowan group assumes this will provide an attractive second revenue stream for an existing fiber network.

That assumption ignores the fact that the 5G company will be competing directly against the fiber owner for retail broadband customers. It’s not hard for me to envision a scenario where the fiber network owner will lose margin by this transaction. They will be trading high margin retail customers for low-margin 5G wholesale connections.

I saw one market analyst that guessed that a Verizon 5G gigabit offering would capture 30% of the customers in a market. The only way for that to happen would be for the 5G provider to take a big chunk out of the customer base of both the incumbents in the market as well as the fiber owner.

There are markets where selling wholesale 5G might be a good business plan. For example, I’ve seen speculation that Google Fiber and other large overbuilders hope to achieve a 30% market share in large NFL-sized cities. I could foresee a scenario where Google Fiber might increase profits by offering both retail broadband and wholesale 5G connections.

But in smaller markets this could be a disaster. If the fiber network is in a smaller town of 50,000 people, the existing fiber network might need a 45% or 50% customer penetration to be profitable. It’s not hard imagining a 5G scenario that could drive the network owner out of business through loss of higher-margin retail customers. I can’t see why owners of fiber networks in smaller markets would allow a direct competitor onto their network. While the new source of 5G revenue sounds enticing, the losses from retail margins could more than offset any possible gains from the wholesale 5G revenues.

The Frontier example offers yet another possibility. Verizon is famous for cherry-picking with its fiber networks. They will build to one street and not to the one next door. They will build to one apartment or subdivision but not the one next door. Verizon seems to have stayed very disciplined and built only to those places where the cost of construction met their construction cost criterion. I could foresee somebody owning a cherry-picking network to leverage it to get to the homes that are not directly on the fiber routes. We still don’t yet understand the factors that will determine who can or cannot be served from a 5G network, but assuming that such a network will extend the effective reach of fiber this seems like a possible business plan.

But there are fiber networks owned by telcos, municipalities and fiber overbuilders that might look at the math and decide that having a 5G provider on their network is a bad financial idea. I have a difficult time thinking that cable companies will allow 5G competitors access to fiber that’s deep in residential neighborhoods. My gut tells me that while Wall Street foresees an opportunity, this is going to be a lot harder sell to fiber owners than they imagine.

The Community Reinvestment Act and Broadband

The Community Reinvestment Act (CRA) is a federal law that’s been on the books since 1977. The law encourages banks to reinvest some portion of their portfolio in their local communities. The law specifically wants banks to make loans that benefit low and moderate-income neighborhoods. Over the years banks have met the CRA thresholds by investing in assets like low-income housing.

Recently the Federal Reserve, which monitors CRA lending at member banks has suggested that improving local broadband would qualify as CRA investment as long as the projects benefit the target parts of the community. This decision will make it easier for banks to make loans to local broadband providers in their community.

It’s worth looking at the history of bank lending for infrastructure to put this announcement into perspective. There was a time when banks were a major lender for infrastructure projects. If you look  more than 50 years local banks lent to projects to build community infrastructure like cable TV networks, water systems, electric power grids, city halls, etc. These are considered as infrastructure loans if they have long loan terms of 20 to 30 years, much like home mortgages. Even then banks didn’t loan much for really long-life assets like roads, bridges and dams – but they were still a major lender to things we would consider as basic infrastructure.

But for various reasons banks stopped lending for infrastructure. Part of this was due to the turbulence in interest rates in the early 70s. All interest rates bounced around for a while and at one short period of time home mortgage rates were four times higher than today. While interest rates eventually settled back down, the swings in interest rates scared many banks from tying up high dollar loans for 25 or 30 years.

This same time period also saw requirements from the federal government for banks to hold more cash in reserve. Many local banks before then would loan out most of their cash, with the hope that most of the loans were solid. But there were enough loan failures in the 70s to shake the confidence of the banking system and to dissuade banks from lending most of their cash.

What really put the cap on this kind of lending was the massive bank consolidation that saw a significant portion of local banks get gobbled up by larger banks. Before all of the consolidation there was hardly a town or county in the country that didn’t have a local bank that was interested in making local loans. But as those banks disappeared, borrowing for local businesses of all types became harder.

What might this change by the Federal Reserve mean for broadband projects? At a minimum it means that local banks are a lot likelier to listen to the story of somebody that wants to borrow. Now that loans for broadband infrastructure will meet banks CRA obligations they are going to pay particular attention to such loans.

But this is unlikely to open up the floodgates of bank investment in broadband infrastructure. Even if it’s easier to talk about loans borrowers still need to deal with the fact that most banks have a lending limit for an individual loan, particularly for somebody who hasn’t borrowed from them before. Building fiber is expensive and if the bank’s maximum loan size is something under $1 M (could be a lot less), then such loans won’t go very far if trying to expand a fiber network. This is not to say it’s impossible. I know small ISPs that have a revolving line of credit that they can borrow for expansion as they pay off existing loan amounts. But this is almost the opposite of infrastructure financing since such loans generally are paid off in a few years, at most.

It’s probably going to become a lot easier, though for borrowing for smaller broadband projects. This might be building wireless networks to serve parts of a town. These loans might support public hot spots or broadband to low-income housing, as long as there is a revenue stream sufficient to repay the loans. Such loans might also fund small fiber builds needed to connect to a business park, to cellular towers or to a small segment of the community.

There is another avenue that borrowers ought to consider, which is a bank consortium. This is where a group of banks go together to make a loan that is larger than what any of them would tackle alone. This generally requires a bank that is local to the borrower to act as the broker and leader of the deal. This is a lot of work for the primary local bank, and so it takes a sympathetic and willing local bank partner. But the changes in the CRA rules means that it might now be easier to talk banks into joining a consortium. It’s worth a try for somebody that don’t have another path for borrowing.

Be a little bit leery of anybody that tells you that this a world changing decision. Banks are still incredibly conservative and this won’t change their expectation for the metrics they will want a borrower to meet or the collateral they will expect to support a loan. But it ought to open the doors to have conversations with bankers that might not have been possible a few years ago.

Net Neutrality – Time to Reassure Your Customers

The recent net neutrality decision by the FCC has created an amazing amount of fear for broadband subscribers who are worried that they will be losing access to popular aspects of the Internet. There is also general confusion in the public from numerous rumors circulating on social media – some potentially true and many others false.

And I think this worry and confusion creates a good opportunity for smaller ISPs to let customers know that you will continue to uphold net neutrality, even if it is no longer required. This is an easy pledge for small ISPs to make because it’s difficult for small ISPs to violate net neutrality rules even if they want to. The net neutrality rules were aimed at the largest ISPs, the ones that have enough market power to put pressure on web content providers, or ones that might implement intrusive requirements on customers.

It’s also a good time to tell customers of plans to continue to protect their privacy – something that the public probably associates with the net neutrality headlines. While the two topics are not the same, I am sure that many people equate net neutrality and privacy.

In the short run I recommend contacting customers and making a big splash about the topic. Perhaps send a heartfelt email or even mail a paper letter to customers that pledges a continuation of net neutrality and respect for customer privacy.

Small ISPs that are competing directly with the big ISPs also ought to consider making this one of the highlights of any sales or marketing campaign. This is a differentiation from the big ISPs that customers will value that really doesn’t cost a small ISP anything. It should be easy to promise not to block Internet traffic, throttle customer broadband speeds or force paid prioritization of Internet traffic. It also should be easy to pledge to not share customer data.

If the current reversal of the net neutrality rules lasts for a while (something I am doubtful about) this could get a little more complicated. I am positive, for example, that at some point over the next few years that bigger ISPs or data brokers are going to offer to pay small ISPs for access to customer data. Small ISPs ought to reject such offers because the benefit of maintaining customer privacy is worth more than payments from selling customer data.

I also suspect that small ISPs will eventually get offers to take part in programs or products that would violate net neutrality rules. You might be offered software that will create bundles of Internet products, like the ones likely offered by the big ISPs. You might be offered cheaper backhaul bandwidth that includes some blocking and prioritization of traffic. Again, my guess is that maintaining a totally open Internet product is worth more than can be gained by implementing such future products.

The big ISPs are unwittingly handing their competitors a chance to take the high road and it would be silly not to take advantage of this opportunity. I know that if I had an option to buy broadband from a small ISP I would jump at the opportunity as long as they were making this pledge. I currently have broadband from Charter. They haven’t said what they might have in mind due to the end of net neutrality, but I find it impossible to believe that they won’t copy things done by the other big ISPs that prove to be profitable. As a consumer my real fear about the end of net neutrality is that the public won’t be told what their ISP is doing. For example, you might experience slowdowns of some kinds of web traffic and not know that you were being throttled. The big ISPs are already quietly monetizing customer data.

Even if some of the net neutrality rules should be put back in place I think any marketing advantage from the topic will still favor small ISPs. Small ISPs will be able to claim for many years that you never lobbied to end net neutrality and you never violated customer trust, even after the net neutrality rules were killed.

Why Not More Collaboration?

I recently was working with some electric cooperatives that collaborate to perform many of the routine functions for operating their companies. This is a fairly common business model for small electric companies, and many cooperatives and municipal electric companies collaborate together.

I’ve always wondered why smaller telcos and ISPs don’t work together more with their neighbors. I know of a few telco cooperatives that work together. There are also small holding companies around that own multiple small telcos that effectively force this structure on their companies. There is some small degree of collaboration for companies that belong to statewide fiber network groups, but this is usually fairly limited in scope.

What do I mean by collaboration? It means sharing the cost of performing the routine functions of operating an ISP to achieve efficiencies and savings. When I look at the companies that collaborate I see some of the following functions being done together:

  • Accounting. Collaboration means hiring fewer accountants and also saving money on accounting software.
  • Benefits Administration. Assuming that benefits can be aligned it’s far more efficient to share the cost of acquiring, monitoring and administering benefits.
  • ISP Functions. For companies that do ISP functions in house there are significant savings from sharing email platforms, security monitoring, spam filters, etc.
  • Legal. Some of the routine legal costs such as negotiating interconnection agreements, pole agreements, vendor contracts, etc. can be less costly when done for several companies at the same time.
  • Software Sharing. There can be significant savings on software licensing fees for companies acting as one entity.

There are also possible economy of scale savings from collaboration, where looking like a larger company can save money. Almost everything larger ISPs do is cheaper per customer than for smaller ISPs, so collaboration can let a group of companies look like one larger entity. Possible collaborative functions include:

  • Billing. Larger ISPs are able to perform billing functions in house at a lower cost than outsourced services.
  • Engineering. Smaller companies working together can probably hire an engineer or two for a lower cost than each of them outsourcing the function.
  • NOC Monitoring. Companies working together might be able to afford 24/7 NOC coverage for less than what they are paying today for partial coverage.
  • Help Desk. Collaborating on a shared help desk can also mean better coverage hours for a lower cost per customer.
  • Purchasing. Collaborating can mean using a full-time buyer, which is more efficient that using mainline employees to do the purchasing function. There are also economy of scale savings from buying some network components and other supplies together.
  • Dispatch / Fleet Management. Again, looking like a larger company should equate to savings on vehicle maintenance and insurance. There is also the opportunity to afford a better fleet management tool to reduce windshield time for technicians.
  • Customer Service. I don’t know many telcos that share the customer service function, but I see electric companies with significant savings from operating one group for multiple companies.

Finally, there are significant savings of the collaborating companies that have physically interconnected networks:

  • Voice Switch. The savings per customer are significant for sharing one voice switch for multiple companies. There is also a significant potential savings for switch functions like SS7, database management, interconnection trunks, wholesale long distance, etc.
  • Cable TV Headend. There is also significant savings from sharing one cable headend.
  • Major Network Routers. The major core electronics used to power FTTP or other kinds of networks can be used to support multiple companies if all are on the same network using the same technology.
  • Internet Backbone. Companies acting as one can buy a larger broadband data pipe for a lower cost per gigabyte and well as more easily establishing redundant routes to the Internet.

Neighboring ISPs can gain some of these savings by simply working together. But many of the savings (such as the economy of scale savings) probably require a more formal corporate structure. A common structure in the electric world is to create a service corporation that is owned by, and works for all of the owner / partners. With margins tightening across the small ISP industry this is something that any small ISP ought to think about.  Even if you don’t jump in and share everything, starting with a few functions can make a significant difference to bottom line.

How Do You Plan?

Today’s blog is not specifically about a telecom topic, but is something that affects every one of my clients. Today I ask the question: how do you plan for the future of your business? I ask this question because it’s something I see many of my clients struggle with.

What do I mean by planning? To me it means having a process for identifying and setting goals and then having a process for implementing the goals. Goals can be anything, but most of the goals that my clients identify are either to fix existing problems or else to implement something new in their broadband business.

I know this is a problem for many broadband companies because I see them facing the same problems year after year or I see them taking a really long time to implement a new product or build into a new market. I have many clients that are frustrated by this.

It’s usually fairly easy to diagnosis the reasons why plans don’t get implemented. One of the major reasons that plans go awry is that companies get wrapped up in the day-to-day operation of the business and taking the time to make planned changes slips to the bottom of the priority list. I also often see that companies try to tackle too many changes at the same time. I have clients who hold an annual strategy session and then try to implement a dozen changes in the company. This rarely works and half of the changes fall to the floor and end up on the to-do list the following year. And for some companies, plans don’t get implemented because the company doesn’t have any real planning process and good ideas just hang in the air.

I will be the first to tell you that planning is hard. I am sometimes as guilty of not taking the time to plan as my clients and it’s easy to fall into the trap of reacting to fires every day rather than taking the time to plan for the future. But I have a few clients who are really good at the planning process, and interestingly. In observing companies who are good at planning, I notice the following similarities:

  • The planning process is formal. There are scheduled planning sessions that are a top priority for the company. There are planning or implementation meetings held regularly to help set goals and then to make sure that the goals are being met.
  • The planning process is mandatory. I’ve seen companies set planning meetings only to have half of the planned attendees beg off to take care of daily fires. If planning is not mandatory then the planning process tends to fizzle out over time.
  • It includes the whole company. This is not just something that the top few guys in a company should do. Since it generally takes all parts of the company to implement new ideas or to fix problems, then every group ought to have some input to the process. The top people night have the biggest role in choosing the direction of the business, but if the whole company doesn’t feel vested in the process then plans tend to slip in importance to the daily work routine.
  • Goals are published. Everybody in the company ought to know what the short and long-term goals are, and they should understand their role in implementing solutions.
  • The process needs to be organized. Companies that are good at planning keep a running list of tasks they want to accomplish. This is essentially a company to-do list. They list will include both major and minor goals. A major goal might be something like entering a new market or implementing a new product. Minor goals might include things like developing a needed new management report or finding a way to streamline a specific process.
  • Plan for successes. The best way to keep a formal planning process going is by getting wins. Just like it feels good to cross something off your personal to-do list, a company benefits organization-wide if there are constant small wins by crossing things off the company to-do list. This means setting quarterly goals for minor tasks along with longer time frames for major tasks.

As a matter of disclosure, this blog is not intended to drum up work. While I often help companies set goals and priorities, our firm does not offer a formal product for establishing a planning process. There are plenty of firms I know who offer this service and I’d be glad to make a recommendation. I’ve seen that bringing in an outsider to help create a formal process can be money well spent. If your company struggles with setting and implementing goals then seeking help might be one of the best investments you can make – it’s really investing in yourself.

 

 

Broadband Speeds are a Local Issue

You might think that the big ISPs deliver the same broadband products everywhere. But I’ve been seeing evidence that broadband speeds are definitely a local issue. One of the products that we’ve been using to help clients assess a new market is to get a lot of people in the potential market to take speed test. We’ve mostly been using the Ookla speed test, but probably any speed test is sufficient as long as everybody in a market takes the same test.

The results of these speed tests surprised me a bit because they showed a wide variance in the products of the major ISPs. For example, I’ve seen markets where Comcast is delivering a little more download speed than they are advertising. But I also saw tests results from a Comcast market where the speeds were about 20% less than advertised. I’ve seen the same thing with AT&T where there are markets that get only half of the advertised speeds and other markets where they were mostly delivering what they are promising. I’m not sure if there is any better demonstration that speeds are a local issue than by seeing that the big ISPs don’t deliver the same speeds in every market.

There is a long list of reasons that can account for the differences in speeds. A big one is the age and quality of the network cables. Older telco copper and older coaxial cables can cause a lot of problems with quality. The size of customer nodes is always an issue. If everything else is equal, a cable company node serving 100 customers is going to have better broadband speeds than one serving 200 customers.

The other big issue that affects customer performance is what I call network choke points. A chokepoint is any place in a broadband network that restricts the flow of data to and from customers. There can be a choke point directly within a neighborhood if the nodes are too large. There can be a chokepoint between a node and the core network if the electronics for the connection are undersized. There can be a chokepoint on local network rings if they don’t provide enough bandwidth. There can be electronics chokepoints at a headend if a router or other major piece of electronics is overwhelmed. And finally, there can be an overall chokepoint in a network if the data pipe going to the Internet is too small.

Chokepoints don’t have to always be a problem. Many chokepoints only appear during the busiest hours of usage on the network, but don’t impede data speeds when data traffic volumes are smaller. And this means that chokepoints are often hyper-local. They might affect one neighborhood but not the one next door, and only at some times of the day. I’m guessing that the slowest results I saw in the big ISP speed tests were during the peak evening hours.

These chokepoints obviously don’t only affect the large ISPs and plenty of smaller ISP networks have chokepoints. I’ve seen numerous network chokepoints appear in recent years due to the explosive growth of the use of broadband. A network that may have been functioning perfectly a few years ago will develop chokepoints as the amount of total bandwidth on networks overwhelm some portion of a network.

ISPs often are challenged to keep up with the upgrades needed to avoid chokepoints, because generally the only ways to relieve chokepoints is to replace cables or to upgrade electronics, which can be expensive. Smaller ISPs often don’t have the immediate capital available to fix chokepoints as they appear. The big ISPs tend to ignore chokepoints as they appear and to make large fork-lift upgrades periodically instead of making the constant small upgrades needed to keep the network working perfectly.

I always advice my clients to keep a running list of all of their chokepoints. With good network engineering and monitoring practices a company can see chokepoints coming long before they materialize and hopefully can plan to make the needed upgrades before they degrade the customer experience.

 

FCC’s Recommendations to Avoid Network Outages

The FCC’s Public Safety and Homeland Security Bureau just released a list of recommended network practices. These recommendations are not a comprehensive list of good network practices, but rather are compiled by analyzing the actual network outages reported to the FCC over the last five years. Telcos are required to notify the FCC of significant network outages and every item on this list represents multiple actual network outages. It’s easy to look at some of the items on the list as think they are common sense, but there obviously there are regulated telcos that triggered had outages due to ignoring each of these network practices.

Following are some of the more interesting recommendations on the list:

Network Operators, Service Providers and Property Managers together with the Power Company and other tenants in the location, should verify that aerial power lines are not in conflict with hazards that could produce a loss of service during high winds or icy conditions. This speaks to having a regular inspection and tree trimming process to minimize damage from bad storms.

Network Operators and Property Managers should consider pre-arranging contact information and access to restoral information with local power companies. This seems like common sense, but I’ve been involved in outages where the technicians did not know how to immediately contact other utilities.

Network Operators, Service Providers and Public Safety should establish a routing plan so that in the case of lost connectivity or disaster impact affecting a Public Safety Answering Point (PSAP), 9-1-1 calls are routed to an alternate PSAP answering point. A lot of the recommendations on the FCC’s list involve 9-1-1 and involve having contingency plans in place to keep 9-1-1 working in the case of network failures.

Network Operators, Public Safety, and Property Managers should consider conducting physical site audits after a major event (e.g., weather, earthquake, auto wreck) to ensure the physical integrity and orientation of hardware has not been compromised. It’s easy to assume that sites that look undamaged after big storms are okay. But damage often doesn’t manifest as outages until days, weeks or months later.

Network Operators and Service Providers should verify both local and remote alarms and remote network element maintenance access on all new critical equipment installed in the network, before it is placed into service. I’ve seen outages where equipment was installed but the alarms were not tested. You don’t want to find out that an alarm isn’t working when it’s needed.

Network Operators, Service Providers, Public Safety and Property Managers should engage in preventative maintenance programs for network site support systems including emergency power generators, UPS, DC plant (including batteries), HVAC units, and fire suppression systems. This might easily be the biggest cause of network outages. ISPs get busy and don’t test all of the components critical to maintaining systems. A lot of outages I’ve been involved with were due to failures of minor components like fans or air conditioning compressors.

Network Operators, Service Providers, Public Safety, and Equipment Suppliers should consider the development of a vital records program to protect vital records that may be critical to restoration efforts. Today there is often software, databases and other vital records that must be restored in order first to get equipment up and functioning. Electronics records of this type need to be kept in a secure system that is separate and doesn’t rely on the network to be functioning, but that also can be accessed easily when needed.

Network Operators, Service Providers, Public Safety and Property Managers should take appropriate precautions to ensure that fuel supplies and alternate sources of power are available for critical installations in the event of major disruptions in a geographic area (e.g., hurricane, earthquake, pipeline disruption). Consider contingency contracts in advance with clear terms and conditions (e.g., Delivery time commitments, T&Cs). This is a lesson most recently experienced after the recent hurricanes where local gasoline supplies dried up and several utilities without their own private fuel supply were stranded along with the rest of the public.

This FCC list is a great reminder that it’s always a good idea to periodically assess your disaster and outage readiness. You don’t want to discover gaps in your processes during the middle of an outage.

When a Consultant Says ‘No’

Doug Dawson, 2017

One of my competitors recently held a webinar where they told a group of municipalities that they should never accept ‘no’ from a consultant who is evaluating fiber business plans. This is about the worst advice I think I have ever heard for many reasons. I think perhaps this consultant meant that one shouldn’t be afraid to be creative and to look at alternative ideas if your first ideas don’t pan out. But that’s not what they said.

Building and operating a fiber network is like any other new business venture and sometimes a new business venture is just not a good idea. This is why anybody launching a new business of any type does their homework and kicks the tires on their ideas to quantify the opportunity. A feasibility study means going through the process of gathering as many facts as possible in order to make an informed decision about a new opportunity.

The advice in this webinar was given to municipalities. Somebody giving this same advice to for-profit ISPs would be laughed out of the room. Established commercial ISPs all understand that they have natural limitations. They are limited in the amount of money they can borrow. They understand that there are natural limits on how far they can stretch existing staff without harming their business. They understand that if they expand into a new market and fail that they might jeopardize their existing company. My experience in building business plans for existing ISPs is that they are as skeptical of a good answer as a bad one and they dig and dig until they understand the nuances of a business plan before ever giving it any real consideration.

But municipalities build fiber networks for different reasons than for-profit ISPs. Existing ISPs want to make money. They also undertake expansion to gain economy of scale, because in the ISP world being larger generally means better margins. But cities have a whole other list of motivations for building fiber. They might want to solve the digital divide. They might want to lower prices in their market and foster competition. They might want to promote economic development by opening their communities to the opportunities created by good broadband.

These are all great goals, but I have rarely talked with a municipality that also doesn’t want a broadband business to at least break even. I say rarely, because there are small communities with zero broadband that are willing to spend tax dollars to subsidize getting broadband. But most communities only want a fiber business if the revenues from the venture will cover the cost of operations.

Sometimes a strong ‘no’ is the best and only answer to give to a client. Clients often come to me determined to make one specific business plan idea work. For example, many communities don’t just want a fiber network, but they want a fiber network operating under a specific business model like open access. That’s a business model where multiple ISPs use the network to compete for customers. Open access is an extremely hard business plan to make work. I’ve often had to show municipalities that this specific idea won’t work for them.

Or a commercial ISP might want to enter a new market and want to make it work without having to hire new employees. My advice to them might be that such an expectation is unrealistic and that over time they will have to hire the extra people.

My advice to clients is that they should be just as leery of a ‘yes’ answer as a ‘no’ answer. For example, every one of the big open access networks has an original business plan on the shelf that shows that they were going to make a lot of money – and those business plans were obviously flawed. If they had challenged some of the flawed assumptions in those business plans they probably would not have entered the business in the way they did. It’s a shame their original consultant didn’t say ‘no’.

I’ve always said that ‘dollars speak’ and any new business has to make financial sense before you can think about meeting other goals. Every business plan contains hundreds of assumptions and it’s always possible to ‘cook’ the assumptions to find a scenario that looks positive. I have created business plans many times for commercial and municipal clients where an honest look at the numbers just doesn’t add up. I’ve had a few clients ask me to create a more rosy forecast and I’ve always refused to do this.

I personally would be leery of a consultant that doesn’t think that ‘no’ can be the right answer for doing something as expensive as launching a fiber venture. Sometimes ‘no’ is the right answer, and if somebody tells you ‘no’ you ought to listen hard to them. It makes sense to kick the tires on all of the assumptions when you hear ‘no’ and to get a second opinion, if needed. But it’s important to kick the tires just as hard when you get ‘yes for an answer.