The FCC recently voted unanimously to adopt a Notice of Proposed Rulemaking to investigate the disaster resiliency plans of major telecom providers. The FCC noted that it has been taking too long after recent natural disasters to restore cellular and broadband services and wants to know if there are any steps the agency can take to improve recovery times after network outages.
As part of the investigation, the FCC wants to hear more about carriers’ existing recovery plans, and hopefully, the FCC will compare the actual behavior after recent disasters with the formal recovery plans. The FCC also wants to examine policies such as requiring more backup power.
One of the things I’ve noticed through the years in the industry is that small ISPs plan for disasters better than large ones. When I visit a small telco that sits in a known flood plain, I expect to find huts built on stilts to survive floods. I expect to find huts and buildings in hurricane zones built to withstand hurricane winds. I expect to find a robust system of backup generators, including many portable ones that can be deployed quickly when needed. If I go down the road a few miles to an area served by a big telco, I don’t expect to find any of these things.
Disaster recovery preparedness is not cheap. It costs more to build huts on stilts or to build a fortress for a hut to withstand storms – but in the long run, the prudent solution is also the lowest cost one. It costs far more to try to restore service after a big disaster. Big carriers will always say that resiliency is important, but you’ll have to dig hard to find examples of where they spent the extra money up-front to build things right.
Small companies are also more prepared in other ways. For example, small telcos, and electric coops both have formed pacts for aiding each other after disasters. When a bad storm hits a small company there is generally a swarm of technicians from around the country who converge to begin making repairs within hours after the end of the storm. Big companies bring in outside help as well, but not with the immediacy and vigor that I’ve witnessed many times with smaller companies.
Disaster recovery also means a lot of more subtle ways to be ready for trouble. Something as simple as having sufficient spare circuit cards, or a yard full of spare fiber and poles, or plenty of fuel for vehicles can make a huge difference in restoring service quickly.
Resiliency means a lot more than disaster recovery. Resiliency means planning ahead so that disasters don’t knock out service. Being prepared might mean a lot of little things like placing metal poles at key network intersections. It means designing fiber rings with automatic rollover that don’t lose service from a single fiber cut. It means burying fiber in places where damage can be expected, even if it costs more. Resiliency means adhering to a tree-trimming program.
If the FCC investigates the issue in the normal way, it will hear from a string of witnesses from big carriers that all swear their companies take resiliency seriously and who vow that it is a priority. Each big carrier will have a four-volume disaster recovery manual that looks impressive. But what the big carriers don’t have is a track record of recovering from disasters quickly – which is why the FCC created this docket.
I have a suggestion for the FCC if they want to do this right. Bring in frontline workers from smaller telcos, cellular companies, and electric cooperatives and have them examine the disaster and resiliency plans from the big carriers. Let these frontline workers visit the huts and pole yards of the big carriers. I’m sure they would find dozens of holes in the big company plans.
Unfortunately, I don’t think the FCC can force big companies to do the right thing. the FCC might be able to require a few things like requiring more backup generators. But the FCC can’t pass rules that are detailed enough to make sure that the big carriers do all of the little subtle things right. That only comes from companies that care for their customers as much as they care for the bottom line.
Dear Doug:
The problem here is that the larger carriers, and their C-suite and accountants, prefer to award themselves bigger salaries and bonuses while skimping on redundancy and fortifying back-up plans. It is good that the FCC is going to look into this.
Spending revenues on advertising to claim that one or another weather event is a “once-in-a-hundred-years event” (… that occurs once a decade) is no excuse for what is essentially a continued lack of preparedness.
And electric companies seem to have the same problems — they continue to waste their revenues on executive salaries and bonuses, and band-aid solutions while essentially neglecting the welfare of their customers.
Hope they all wake up!!