I’ve written previously on this blog about the theory of leveraging cloud computing to move virtual machines and critical computing services out of the way of natural disasters. Today I’m going to talk about doing it in real life.
Over Monday night of this week, 02/22/10, the location of our south data center began to see the effects of the recent winter storm that left 1 million people in the north east without power when all was said and done. Heavy rain was causing local flooding (nothing for the data center to worry about) and wind gusts over 40mph. By Thursday, the continuous downpour was bringing extensive flooding and the wind was steadily increasing.
At 9:30pm on Thursday night, I got an alarm from the data center and immediately headed in. At this point, the downpour was sheeting sideways in steady 60mph wind, gusting to hurricane strength 90+mph. As I neared the data center, I could see the entire industrial/commercial complex that we’re in was totally dark, right in the heart of the city, total power brownout (120VAC legs were down to 35VAC) (no wonder the BlackBerry was hollering at me!).
In light of the extreme weather conditions and to prevent equipment damage to the electrical brownout, I decided to evacuate and shut down the south data center. The order of business was to transfer all critical cloud services to the backup cloud in our north data center (which was on the other side of the snow line and didn’t get the extreme winds), DNS, email, critical customers, and then backup email service for the customers that there simply wasn’t enough time to move across. After that, it was a nice orderly shutdown, which finished with the emergency lights running out of battery and leaving me in the dark.
Friday was sort out the chaos time. Within 10 minutes of main power being restored, uplinks and internal network were running. Within 30 minutes, cloud servers were operational, critical services were being transferred back, and customer virtual machines restarted. Within 2 hours, all virtual machines, all co-located servers were fully restored and operating.
While I wasn’t able to keep all services running through the night, all critical data center services, email and backup email, and critical customer services continued to run, all without a hiccup, thanks to the ability to easily move virtual machines from one data center to the other.
The best kind of disaster is the one you can avoid!
Call or email me or visit the SwiftWater Telecom web site for hurricane resistant green data center and cloud computing services.