The recent bombing in Nashville is a reminder that our telecom infrastructure is always at risk from terrorism or major natural disasters. The Nashville bombing is a telecom company’s worst nightmare where a deranged bomber parked a powerful bomb outside of the building with the express intent to wipe out the AT&T communication hub. We’ll have to wait to hear the full details of the damage done, but it seems that AT&T was able to restore most local service within three or four days.
We’ve had other major outages. The biggest came on 911, 2001 when terrorists knocked down the World Trade Center towers. This had a secondary impact of damaging the major Internet hub located across the street from the towers. This was a major Verizon tandem office along with being a CLEC hotel and a switching point for the Internet. The collapsing towers not only damaged some of the electronics at the site, but the continuing power outages eventually resulted in overheated equipment and continuing failures.
The third big disaster I recall was the Howard Street Tunnel fire in Baltimore in 2001. A rail crash inside the tunnel resulted in an intense fire that melted the fiber optic cables that delivered Internet traffic between Washington DC and the northeast corridor.
In addition to these major news-event outages, I’ve seen numerous smaller outages caused by hurricanes, floods, and tornadoes where telecom buildings and huts were largely obliterated. The most unusual outage I recall was when brazen thieves stole several miles of large copper wiring off the poles near Sugar Land, Texas.
Anybody who works in a telecom network understands how fragile the network is, at least locally. We do our best to hide electronics inside buildings or behind fences. We’re careful not to create maps showing the locations of key switching and fiber connection sites. But the fact is that a determined person that understands a network can do a huge amount of damage in a single night in most cities. They wouldn’t need a camper full of explosives to cause major damage.
We’ve come a long way since 2001 in planning ahead of time for major disasters. From what I’ve read about Nashville, AT&T brought in a few dozen temporary cell sites to restore cellular coverage quickly. In 2001 I recall that Verizon was proud about delivering a portable switch inside of a trailer – but it took weeks to restore phone service to the cables that hadn’t been damaged.
We’ve also become adroit at quickly switching traffic around damaged facilities. The 2001 tunnel fire destroyed fibers for which there was no alternate routing. Today, most carriers have multiple routing options and the ability to electronically divert traffic away from outages. We now have companies like Cloudflare and ThousandEyes which can react instantly to network problems and reroute traffic as needed. We had nothing like this in 2001.
But the Nashville bombing reminds us that we can’t forget about security when designing networks. I know of fiber networks where large OLT huts are sitting unprotected and open to the public – the network owner is largely counting on the fact that nobody knows what the hut is for. To save money and speed up construction we’ve changed from using concrete block buildings with secure doors to smaller and more fragile metal cabinets.
The damages in Nashville ought to be a reminder to network owners to review the physical safety of their network. Little steps like physical barriers like fences and hedges can make a difference. In today’s world, there is no reason not to have security systems with cameras and motion detectors that can notify law enforcement when somebody is visiting a hut in the middle of the night. We don’t need to be paranoid about security – we have hundreds of thousands of telecom sites that are safe and undisturbed day after day. But the Nashville bombing is a reminder that somebody with a grudge or a nutty idea can cause a lot of damage to our networks, which are a lot more fragile than we want to admit.