Every network faces periodic upgrades of electronics or key components. We have found that cutovers are the time when any network is the most vulnerable.

There are tried and true processes that can be used to minimize the chance or duration of network outages during upgrades. The following is a list of steps that we recommend for any network upgrade that puts customer service at risk. Following these steps is never a promise of 100% safety, but we have never seen a company that upgrades in this methodical and planned manner have major problems.

1. Have a Project Manager for the Upgrade. It is vital to have one person in charge of the upgrade. They can get assistance in planning and doing the upgrade, but they need to be the one ready and authorized to react if things don’t go as planned.
2. Develop a Checklist. You should develop a step-by-step checklist of everything to consider for the upgrade. Make sure that you understand every piece of equipment and software that will be affected by the upgrade. And then, most importantly, develop a step-by-step list of the steps required to perform the upgrade.
3. Break the Upgrade into Manageable Steps.  If possible, the upgrade should be done in stages where progress can be measured and tested as each step progresses.
4. Establish a Baseline / Establish a Go-Back Process. By this we mean that you need to completely understand the current network configuration, in detail. You need to know the exact settings of every piece of equipment. And once you understand the current network configuration develop a go-back process. This would be the steps needed to get the network back to the original configuration if something goes wrong during the upgrade. Ideally the go-back would be something really fast and we sometimes have seen this programmed such that it can be done in minutes.
5. Understand the Traffic Flow (and then monitor during the cutover). During the upgrade process you might not get the same kind of alarms that you normally would expect. Also, changing traffic patterns due to the cutover can skew traditional measurements if you are rearranging the network. So it’s vital to understand your traffic flow before the upgrade. Then, have somebody monitor the traffic during the cutover since this might be the only way you will know that you have knocked customers out of service.
6. Make Sure you have Vendor Support. For a major upgrade you should consider having a vendor representative on site. Otherwise, make sure ahead of time that somebody will be able to help you if you run into unexpected problems. I have seen clients schedule an upgrade over a holiday, not thinking that the needed expertise at the vendor is probably not going to be available.
7. Pre-test Every Component before the Cut. Definitely test any new equipment before you introduce it into the network to make sure that it is operating properly. For complicated upgrades you ought to consider setting up a test lab where you can test the new equipment against components that will remain in the network for interoperability.
8. Take Every Upgrade Seriously. I often see companies follow most of the above steps for major upgrades only to see them knock out their network for some simple upgrade like introducing new cards or something they thought was a simple upgrade. Any change that can knock down your network might knock down your network, so take every upgrade seriously.
9. Define What Success Looks Like. You should establish the needed tests ahead of time that will let you check that every aspect of the network is working as planned. You don’t want to do an upgrade that is almost right only to find that you have created future problems. So establish a detailed test plan.

If you have questions about upgrades or want help developing an upgrade plan contact Derrel Duplechin of CCG at (337) 654-7490.