10 Tips To Prepare For Telecom Disasters

With the beginning of summer, it's important to refresh our memories on Telecommunications Disaster Recovery. Here are some tips I've prepared on how to plan for, react to, and prevent telecommunications "disasters".

1) Conduct Frequent "Disaster Drills"

How can we effectively prepare for a disaster, when we aren't 100% sure how complex systems such as voice networks function when a disaster strikes, is a No. 1 priority. Even though it's a tedious process, the act of taking each system down individually during non-production hours is a big help. These "disaster drills" should be well documented for every scenario tested. With the interconnected nature of modern-day voice networks and PBXes, it's hard to estimate what adverse affects a single component failure will have on the remaining environment. Answer those questions in a controlled drill environment before reality strikes.

2) Make Documentation Part of Every Day

Every software update, patch, configuration change must be documented. This too is a tedious processes, but widely-available documentation is the cornerstone of preventative planning and disaster recovery. Frequently updated changelogs, MACs, and hardware replacement logs allow for a faster way to identify and isolate the problem.

3) Prioritize

Dialtone is incredibly important in the modern-day business world. This will likely not change in the near future. Before a disaster strikes, identify which areas, users, or groups need immediate restoration of service. In other words, plan to bring up the NOC's terminals before the lobby courtesy phones go online. Don't be afraid to restore service in segments - some early functionality to critical users is better than none.

4) Know Your Vendor

Whether it's the manufacturer of your PRI cards, or your upstream dialtone provider, a good relationship with a vendor can be a night and day difference. When it comes to ordering emergency analog trunks when your PRI infrastructure dies, to providing a full-scale temporary network to service inbound calls, the vendor is incredibly important. Include the vendor in your disaster recovery planning process so they understand your needs, and are able to deliver in the event of an emergency.

5) Keep Spares On Hand

It's become a fact of life - equipment fails. If you're a large enough organization, ensure that you have replacement spares on hand to replace failed equipment. Don't be caught waiting for an overnight package when service could have been restored immediately.

6) Backups, Backups, Backups

Backups too are a tedious process. For large systems with lots of moves, adds, and changes, it's important to keep daily backups (at a minimum) to restore known-good configurations. Some systems even allow for redo-logs, allowing system changes to be rolled back after problems are encountered.

7) Cross-Train Telecom Job Roles

Ensure that for every critical job function, a backup contact is in place. If the only person who knows how to restore a switch configuration from backups is on vacation in Hawaii, then a cross-trained backup should be able to effectively assume responsibility at the organization. Frantic cell phone calls to the telecom manager thousands of miles away will only elevate the team's blood pressure.

8) Perfecting Disaster Communication

Let's face it: we're (mostly) technically-focused people. When there is a major outage, we want devote all attention to resolving the problem. However, we forget to communicate with our users, co-workers, and executive-level counterparts. Designate one person to provide "Disaster PR" in the event of a large outage. Ensure that communication is effective with the users affected, to the engineers working to restore service, to the CIO and CEO. Effective communication can lower stress levels and hostility between all parties involved.

9) Redundancy Means "Ready"

Now is the perfect time to diversify your telecommunications infrastructure. Adding remote gateways in geographically diverse locations ensures that barebones service can remain active even when a significant failure occurs at the primary site. Systems that automatically failover to remote equipment allows staff to focus on the outage resolution, and not on preparing the remote infrastructure.

10) Know Your Infrastructure Well

Finally, the more you know about the architecture of your voice network, the more prepared you are for a diaster. Share this knowledge with others, and frequently question your vendor or provider about topics, concepts, or architecture you don't understand. Knowledge is indeed power!