Avoiding data center disaster with cloud computing.


I’ve written previously on this blog about the theory of leveraging cloud computing to move virtual machines and critical computing services out of the way of natural disasters. Today I’m going to talk about doing it in real life.

Over Monday night of this week, 02/22/10, the location of our south data center began to see the effects of the recent winter storm that left 1 million people in the north east without power when all was said and done. Heavy rain was causing local flooding (nothing for the data center to worry about) and wind gusts over 40mph. By Thursday, the continuous downpour was bringing extensive flooding and the wind was steadily increasing.

At 9:30pm on Thursday night, I got an alarm from the data center and immediately headed in. At this point, the downpour was sheeting sideways in steady 60mph wind, gusting to hurricane strength 90+mph. As I neared the data center, I could see the entire industrial/commercial complex that we’re in was totally dark, right in the heart of the city, total power brownout (120VAC legs were down to 35VAC) (no wonder the BlackBerry was hollering at me!).

In light of the extreme weather conditions and to prevent equipment damage to the electrical brownout, I decided to evacuate and shut down the south data center. The order of business was to transfer all critical cloud services to the backup cloud in our north data center (which was on the other side of the snow line and didn’t get the extreme winds), DNS, email, critical customers, and then backup email service for the customers that there simply wasn’t enough time to move across. After that, it was a nice orderly shutdown, which finished with the emergency lights running out of battery and leaving me in the dark.

Friday was sort out the chaos time. Within 10 minutes of main power being restored, uplinks and internal network were running. Within 30 minutes, cloud servers were operational, critical services were being transferred back, and customer virtual machines restarted. Within 2 hours, all virtual machines, all co-located servers were fully restored and operating.

While I wasn’t able to keep all services running through the night, all critical data center services, email and backup email, and critical customer services continued to run, all without a hiccup, thanks to the ability to easily move virtual machines from one data center to the other.

The best kind of disaster is the one you can avoid!

Call or email me or visit the SwiftWater Telecom web site for hurricane resistant green data center and cloud computing services.

Vern

swiftwater telecom rcs cloud computing logo

Advertisements

2 responses to “Avoiding data center disaster with cloud computing.

  1. how about redundant 5kw generators? site and engineering looks great! i know the latest storm was an outlier and you may have me as a customer soon – i am saving for colo…keep on writing, good topical content! thanks vern!

  2. That’s certainly always an idea, but being in here in that mess to refuel them would have been questionable. We do have an advantage by being right on top of 2 generating stations (last winter when 80% of the area was out of power, we hummed right along without a hiccup). In the last 5 years, total down time for commercial power here has been about 4-5 hours, total down time for us has been zero.

    Let me know when you’re ready, we’ll be happy to have you aboard 🙂

    Vern

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s