Wednesday data center tidbits: Intuit faceplants again.

This afternoon, Intuit’s web sites are showing a message that they are unavailable due to a commercial power failure and that they are operating on backup power. This begs the questions. Wasn’t the last (self inflicted) power failure episode painful enough to do something about it? Just what ARE they doing with their “backup power”? Not running their customer facing web sites with it, that’s for sure.

Squeaky red bozo noses to Intuit for egregious data center operations screwups.

Email or call me or visit the SwiftWater Telecom web site for green data center services today.

Vern

Monday data center tidbits.

The “duh” award for most obvious statement of the day comes from this article about the next steps for green IT:

“That capability could save energy because a computer that’s off, most experts agree, is more efficient than one that’s in sleep mode ..”

Um, DUH! I’m glad we had experts to reveal this.

From the same article, we get duh #2, the idea that data center servers that will accept high AC voltages such as 480VAC are more efficient. Until they make chips that use 500V, the voltage STILL has to be stepped down. All you’re doing with the scenario is moving the inefficiency (and all the heat load) inside the server, the last place in the world that you want it.

If you want to get rid of the transformer penalties of AC, go DC in the data center and forget shuffling the penalty around.

Email or call me or visit the SwiftWater Telecom web site for green data center DC power services!

Vern

More green data center DC power myths.

I’ve just been reading about the value of DC power in the data center. in this post, I’m going to correct some of the strange myths and egregious errors that are spread about DC power.

“But the downside is that DC power can require much larger wires to carry the current, thus creating power buildup and arcing that can endanger IT equipment and staff.”

Where do I start on this one. Power does not “build up” in the large conductors of DC power systems, any more than power “builds up” at your unused AC outlet. Not using the DC power doesn’t mean it’s going to build up, over flow, and arc. Mark this one as completely ridiculous.

“.. added that moderately high-voltage DC power poses some safety concerns, where the power can build up and arc.”

Once again, DC power doesn’t “build up”. Any high voltage distribution system that’s inadequately insulated has the potential to do this whether it’s AC or DC. I’ve seen catastrophic failures in common 240VAC and 480VAC circuits that resulted in melted bus bars, holes burned in armored cable, and total destruction of equipment (look at the Fisher Plaza fire where they destroyed 4000A aluminum bus bars).

What does affect the tendency to arc is jacking the voltage up and that’s just the same whether it’s AC or DC, there’s no great mystery to that. There’s a tendency to want to use high voltages to reduce wire size (higher voltage=less current=smaller wire). I myself much prefer to use long time industry standard 48VDC power, which is touch safe and has an extremely small arc over distance.

“”Four-hundred volts DC may be more dangerous than 400-volts AC,” he said.”

In a properly constructed DC power system, there’s no appreciable difference in safety between AC and DC, expect for some possible difference in coming in contact with the energized conductors. This point is really moot since you do NOT want to be coming in contact with either live 400VDC or 400VAC. Either way, someone is getting hurt.

Is there any safety hazard in the DC power plant? Yes, the battery string (exactly the same hazard that a battery equipped AC UPS has as well). Short the unfused battery leads or drop an uninsulated tool into the bus bars and you’re going to be missing some wire or a tool, completely. Batteries can put out 9,000+ amps of current in a fraction of a second in a short condition.

Finally, there’s the efficiency issue. Comparing a DC power plant to a conventional double conversion (or “online”) UPS is not in doubt. The UPS systems that fare better against DC are the type with “eco mode” (the old “standby” UPS). In normal operation, the eco mode UPS powers the load directly from the AC power, so it doesn’t use the power hogging inverter. Of course, this means the data center equipment has to stand the switch over to battery and the efficiency looks just as bad as a double conversion UPS when it’s running from battery.

Call or email me or visit the SwiftWater Telecom web site today and we’ll design a green DC power plant for your data center that will be SAFE and EFFICIENT!

Vern

5 steps to unfreezing the frozen data center.

Tonight I was reading about DC area data center restoration after the recent major snow storm. Would you know how to unfreeze your data center?

So, you finally manage to make it through the drifts with snowshoes and sled dogs, only to find a data center that’s dark, silent, and cold as a meat locker. How do you get going again without damaging anything worse than it already is?

1. Disconnect all power to the facility. Try to restore too much at once and you’ll be blowing breakers or damaging equipment if the commercial power is still unstable.

2. Restore major transformers first, then UPS or DC power plants, these are the major draw on the commercial power. Do not restore anything else until the UPS batteries or DC power plant battery string are fully charged. Remember, there are 3 parts to the data center power facilities, commercial AC, battery back up, and generators. Don’t start anything unless 2 of the 3 are available.

3. Before warming the data center up, inspect all weather exposed equipment (cooling, generators) to make sure there’s no snow infiltration and all vents, intakes, and exhausts are clear. Snow will clear a LOT easier when it’s powdery and dry, let it turn to water and you’re going to have a real problem. Apply power to a piece of mechanical equipment that has moving parts frozen or obstructed by snow and you’re going to have a worse problem. It’s also a good idea to avoid sucking snow inside via the cooling system.

4. Get the heat going. Make sure mechanical equipment is up to a reasonable temperature before energizing. Make sure generator starting batteries are warm and charged. Once the generators will run, then you can start restoring the IT equipment.

5. Start restoring equipment from the edge in (uplink equipment, edge routers first). Take the equipment one step at a time, shut down and come back to or diagnose anything that doesn’t start immediately. Never leave a piece of malfunctioning equipment powered up, you could end up with more heat than you expect when it toasts. Gradual restoration will also take it easy on the power systems.

Obviously this isn’t an exhaustive list of everything that would need attention in a frozen data center. Frozen pipes are a whole other chapter (it’s a good idea to shut the water off to frozen pipes, you don’t need a flood in the data center when they thaw with breaks in them.

Here’s hoping you never need to do this, but if you do, these 5 steps will help you get thawed out without making anything worse.

Call or email me today for data center engineering, operations, DC power plant engineering, or operations services for your data center!

Vern

Balance the data center with green DC power.

This morning I was reading about preventing cascading failures in the data centervia power load balancing. I’ve written previously about not allowing data center failures to get out of control and cascade into a much larger event, in this post I’ll be talking a little bit about the importance of power balancing.

At the simplest, power load balancing is just what it says. Power loads should be balanced at distribution points such as breakers and balanced across redundant feeds to equipment. This insures that a configuration that works fine under normal conditions doesn’t overload and trip in failover conditions. Beyond the obvious of making sure that the power configuration will reliably work under all load conditions, there are also power quality issues as well.

Most industrial/commercial locations are supplied by 3 phase AC power in either a delta or wye configuration, 3 hot phases, and 1 neutral (wye only). Single phase equipment is supplied from either 1 hot phase and the neutral or 2 hot phases. The issue will this type of power is that load imbalance of more than 10% between the hot phases can cause significant misbehavior of the transformer not to mention efficiency problems as well. We just had a landlord correct a load 3 phase load balancing issue at one of our facilities, 95% of the distribution load was on 1 phase of the breaker. This imbalance caused what should have been 120VAC to drop as low as 100VAC.

So where does data center DC power enter into this? A modern modular DC power plant collapses the 3 phase power into one DC power feed. Since the rectifier modules operate in parallel, the load on each AC circuit will be perfectly divided between them. Compare this to AC powering equipment where the load of the equipment is constantly changing, meaning that you could never balance this type of load with any precision. With a DC power plant, as long as you run modules in multiples of 3, you’re guaranteed to maintain perfect load balance.

Efficient in operation, efficient in use of your AC power capacity, making the power balancing act look easy, DC power is the choice for the green data center.

Vern

Wednesday data center tidbits.

First up this morning is the story about Bell Labs and Green Touch committing to reduce the power consumption of data networks by 1000x. I’m not sure that kind of a gain is achievable but I’m fairly certain this is low on the bang for the buck scale.

Second up is the revelation that Google is seriously considering walking away from China over rampant cyber attacks on its data centers. This surprises me not in the least. 99% of the attacks on my servers here are from China, with the remaining amount from Korea and Eastern Europe.

Finally we have an article on preventing cascading failures in the data center. Balancing power loads is absolutely critical to data center reliability, especially when 3 phase AC power is involved. Imbalance on 3 phase can cause some severe power quality issues. After having multiple breaker trips at one of our locations (no downtime thanks to an excellent DC power plant in the data center!), the landlord discovered that 95% of the load was being carried by only 1 phase of the 3 phase feed on that breaker.

Vern

Xen Cloud Platform command of the week.

This post is my once a week effort to detail useful commands in the Xen Cloud Platform virtual cloud computing system that aren’t included in the manuals.

Today’s command is xe host-evacuate. The host-evacuate command has the form xe host-evacute uuid=. As the name suggests, host-evacuate migrates all the virtual machines off an XCP host.

This command could be used to take a host down for service or to migrate VMs before a hosts backup power expires. I use it as part of the procedure to automatically condense the cloud during low usage periods and shut down unneeded hosts.

12 days of data center Christmas #12, cloud powered virtual machines

12 days of data center Christmas #11, DC power engineering specials

Vern, SwiftWater Telecom

Data center questions and answers.

I’ve culled these data center related questions from search engine traffic to the blog this week.

The first question is about 277VAC power. 277VAC is the low voltage component of 480Y277 (hot to neutral). 277 is used to provide lighting in industrial plants that generally use 480 3 phase power. Since most data center equipment universal power supplies top out at 250VAC, 277 really isn’t useful for anything except lighting.

The second question is about using 12VDC as DC power in the data center, vs 48VDC. 12VDC would require substantially more current to provide the same energy as 48VDC, meaning larger power plant, larger batteries, and larger conductors. Add to this that 12VDC is a non-standard voltage for this use and little to no data center equipment is going to support it. Unless you’re Facebook or Google and you can afford to have custom equipment made, stick to 48VDC.

The next question is what the best DC power plant batteries are. Rather than answer that, I’ll run down the criteria. Data center DC power plant batteries should be sealed (to prevent off gassing of flammable hydrogen), constructed to withstand possible deep discharge cycles without damage, and low sulfur content (to extend the battery life and prevent premature failure from sulfation). I usually use common sealed deep discharge 120AH batteries in the strings.

The final question is server room separation distance from the electrical equipment. This is a timely question with recent catastrophic electrical failures in data centers. The only thing in data center electrical equipment that represents a significant threat is an oil filled transformer and those should NEVER be placed inside the facility due to the risk of sudden catastrophic failure. In general, the server equipment should be as close as possible to the power equipment to reduce power loss in the cables, especially if you’re running a 48VDC power plant to supply your servers.

Vern, SwiftWater Telecom

data center cloud computing

Greening the data center, the hard way.

Tonight I was reading about Facebook following Google’s lead in putting batteries in servers in the data center. This just goes to show you’re never to big to have or share a boneheaded idea.

The idea is simple. Remove the data center UPS and add a backup battery to every server. This is far from a green solution, not to mention a serious maintenance nightmare.

The conventional double conversion UPS is one of the biggest energy wasters in the data center. In a double conversion UPS, incoming AC is rectified to DC for the batteries, inverted back to AC for the servers, and converted back to DC internally. Each conversion adds loss to the system and generates extra waste heat that requires energy to dispose of, so eliminating the double conversion UPS can make a major difference in data center power efficiency.

Now we come to the hard way to do it, putting a battery in every server. Now, consider for Facebook, that means in excess of *30,000* batteries. Consider the cost of all those batteries, the amount of toxic heavy metals involved in the manufacture of those batteries, and the near impossibility of maintaining them. Add to that the cost of proprietary retrofits to servers or complete replacement with proprietary servers and this is starting to sound like something out of a horror movie.

So, how does this even actually work for Facebook? It’s simple. Facebook and Google both have such massive amounts of redundant servers that there isn’t much impact if a few hundred to a thousand of them are broken at the same time. Just don’t think about what having a thousand extra servers running does to energy efficiency (this is why PUE is such a crock).

So, how do we eliminate the energy sucking power hog that is the double conversion UPS and not create a nightmare in the process? Simple, DC power distribution. In the data center DC power plant and DC power distribution setup, AC power is rectified to DC and the DC power is supplied directly to the servers. One conversion instead of three is a definite power saver. In this kind of plant, the batteries are simply strings that concentrated with the DC power plant and connected across the DC power distribution.

In the data center DC power plant, you have a minimum number of batteries all concentrated in one location, making maintenance a snap. Also, batteries used in DC power plants are typically designed to last far longer than anything sized to use in an individual server, drastically reducing the expense of replacing all those batteries, not to mention the ecological nightmare of all those little packages of toxic heavy metals. The added benefit is that the heat generated by the servers themselves is also reduced.

Next on the plus side is that servers and power supplies based on the standard 48VDC power, while not as common as AC powered servers, are certainly easily available, eliminating the expense of proprietary hardware design.

Finally, you’re not going to have ANY servers fail from dead backup batteries when push comes to shove. This means no more running hundreds to thousands of extra servers required to compensate for those dead battery failures.

Talk about trying to do it the hard way.

Vern, SwiftWater Telecom

data center DC power plant engineering

Cookie cutter container data centers, take 2

Some time ago, I wrote about a different perspective on container data centers. Tonight I’ve been reading about Mike Manos’s take on containers and I’d like to talk a little bit about some of his points.

Mike thinks that data centers have been moving at an evolutionary snail’s pace. Green power, free air cooling, green cooling techniques in general, DC power distribution, I think the pace of evolution is hopping right along and I sure don’t think moving to pre-built modular Lego data centers is going to spark that.

Mike says that “In the past, each and every data center was built lovingly by hand by a team of master craftsmen and data center artisans. Each is a one of a kind tool built to solve a set of problems. “
This is true, because the people that build this way recognize that “one size fits all” fits everything (sort of) and fits nothing optimally.

Data centers are not nail hammers. Nail hammers generally work equally as well to drive a nail whether they were handmade by a blacksmith or mass produced in a factory. A nail hammer works the same in the tropics, at the North Pole, or on the moon. Cookie cutter data centers may leave you driving tacks with a sledgehammer or driving spikes with a flyswatter. Myself, I’d prefer not to deal with either.

Show me a container data center that can deal with such green techniques as free air cooling or DC power. I’m not aware of either and these are two of the hottest green techniques going.

The watchword for container data centers seems to be simplicity. It’s true, you can dumb down the data center so things consist of nothing except unplugging and plugging huge boxes. Just hope the vendor did all their engineering right because you’re not going to be able to tweak anything once it’s plugged in (not to mention you won’t even have a good sense if things are working the best they can be or not).

So, are containers all bad? Not at all. Containers are useful for huge installations with massive amounts of identical servers. Maybe you’re willing to trade off optimization for only needing 4 people to install a 2000 server container. It’s certainly handy for Microsoft to run a single purpose 700,000 sq ft data center with only 45 people.

On the other hand, if you need flexibility to handle different types of hardware and new technologies, to truly be green, and to be as optimal as possible for your particular set of needs, a Lego data center isn’t going to cut it.

Remember, no sledgehammers on tacks.

Vern, SwiftWater Telecom

data center facility engineering