What to do When Plan B Fails

The 2019 Polar Vortex and extreme winter weather with wide spread power outages throughout the Great Lakes area put many organization’s recovery plan to the test, including ours at CU*Answers. You develop and plan for interruptions, but what do you do when Plan B is not enough; when Plan B fails? Our primary data center lost utility power multiple times during the severe winter storms on February 7th and 8th. Each time the generator kicked on like clockwork providing an alternate source power. During one of the utility power outages, on a busy Friday morning, the primary generator experienced an equipment failure and powered down. Our Plan B was in jeopardy.

Operating on backup UPS battery power, recovery teams were able to initiate an emergency failover of data center operations to our secondary site located in Yankton, SD. At the same time, salvage teams were diagnosing and attempting to repair the generator. They were successful, with minutes to spare on the power supplied on the UPS. Not taking any chances, our facilities team was able to locate and secure an auxiliary generator with enough capacity to run our data center in the event of another Plan B failure.

When you perform a test of your recovery plan (your Plan B), do you inject multiple scenarios, or do you make the assumption that your Plan B will always work? Do you plan for the day when the controls you put in place do not work the way you expect them to? After our experience from this past February, you can be sure that our plans and our training exercises will not always follow a predictable scenario.

 

Danielle Caliendo MPA, CBCP
Business Continuity Coordinator, CU*Answers