Tonight I was reading about DC area data center restoration after the recent major snow storm. Would you know how to unfreeze your data center?
So, you finally manage to make it through the drifts with snowshoes and sled dogs, only to find a data center that’s dark, silent, and cold as a meat locker. How do you get going again without damaging anything worse than it already is?
1. Disconnect all power to the facility. Try to restore too much at once and you’ll be blowing breakers or damaging equipment if the commercial power is still unstable.
2. Restore major transformers first, then UPS or DC power plants, these are the major draw on the commercial power. Do not restore anything else until the UPS batteries or DC power plant battery string are fully charged. Remember, there are 3 parts to the data center power facilities, commercial AC, battery back up, and generators. Don’t start anything unless 2 of the 3 are available.
3. Before warming the data center up, inspect all weather exposed equipment (cooling, generators) to make sure there’s no snow infiltration and all vents, intakes, and exhausts are clear. Snow will clear a LOT easier when it’s powdery and dry, let it turn to water and you’re going to have a real problem. Apply power to a piece of mechanical equipment that has moving parts frozen or obstructed by snow and you’re going to have a worse problem. It’s also a good idea to avoid sucking snow inside via the cooling system.
4. Get the heat going. Make sure mechanical equipment is up to a reasonable temperature before energizing. Make sure generator starting batteries are warm and charged. Once the generators will run, then you can start restoring the IT equipment.
5. Start restoring equipment from the edge in (uplink equipment, edge routers first). Take the equipment one step at a time, shut down and come back to or diagnose anything that doesn’t start immediately. Never leave a piece of malfunctioning equipment powered up, you could end up with more heat than you expect when it toasts. Gradual restoration will also take it easy on the power systems.
Obviously this isn’t an exhaustive list of everything that would need attention in a frozen data center. Frozen pipes are a whole other chapter (it’s a good idea to shut the water off to frozen pipes, you don’t need a flood in the data center when they thaw with breaks in them.
Here’s hoping you never need to do this, but if you do, these 5 steps will help you get thawed out without making anything worse.