I was a little surprised to revisit my previous post on the Peer 1 data center electrical accident and note that 3 people had rated the post poor. I thought I’d take a little bit and expand on some of the things I said previously and examine why this falls under “we’re our own worst enemies” for data centers.
Any time you open an energized piece of power equipment and expose live conductors, you roll the dice with the reliability of your facility. One mistake or one slip of a conductive tool and you can have lights out, systems out, electrical and physical facility damage, angry customers, and a spot as a trending topic on Twitter (in a bad way). The consequences of even a small goof can be catastrophic (witness the transformer explosion back some time ago in The Planet’s data center or most of the highly publicized data center electrical outages of 2009).
Given the potential negative results, working with live power is not something to be taken lightly. Are there ever reasons for working inside of live data center power equipment? Certainly. Addition of circuits to panels and replacement of some limited components are a couple of good reasons.
What we find happening here is live power work being done for trivial reasons. If a data center power system requires “maintenance” work in live power equipment (ala the Peer 1 accident), then that power system design is severely broken. Witness the recent data center outage where a PDU was shorted while doing a manual “maintenance” test of phase rotation (phase rotation means that the phases of 3 phase power are in the right A-B-C order). Presuming it really NEEDED to be done, a $50 phase monitor would have accomplished the same thing without risking tech safety and risking the operating status of the data center.
The next thing is employee risk. Any time you expose an employee to live power, you place them at risk. It’s not just shock risk, it’s the risk of arc flash injury should a tool slip or a component fail under load. Arc flash can cause anything from light to severe burns, blindness, burns from molten metal, and injury from metal shrapnel ejected at high speed. From the description of the accident, this is what I believe happened in the Peer 1 case. Aside from the economic impact, having an employee injured in an arc flash event because of inadequate protection (in the US, follow NFPA 70E) will bring the governmental labor department having authority in your location (in the US, OSHA) down on you like a ton of rocks.
Finally, it’s also worth noting arc flash risks from other forms of data center power than AC. 48VDC power (which I have supported right along for green data center usage) doesn’t pose a shock danger but the stored energy of the battery string is a significant arc flash hazard. Drop an uninsulated tool into the battery buss bars and it’s probably going to disappear (I’ve seen a DC arc from failed cable insulation cut a 1″ threaded rod completely off).
So, what is the takeaway from this? Protect your data center facility and the health and safety of your employees by eliminating trivial reasons to be exposed to live power. If it is classified as “maintenance”, it shouldn’t be done with the power on, period. When an employee has to be exposed to live power, provide all of the appropriate protection in case the worst happens.
Your customers will thank you for it, your employees will thank you for it, your wallet will thank you for it, and you won’t become the next cautionary example to hit Twitter. The easiest disaster to deal with is the one you avoid.