Finally, proof positive that PUE is garbage.

Vern Burke, SwiftWater Telecom
Biddeford, ME

I’ve just been reading a piece about Microsoft removing fans from their data center servers and that having a negative effect on their PUE numbers. I’ve written on this blog before about the problems with PUE, now we have proof that it needs to be put out of it’s misery.

In a nutshell, PUE is the ratio of power consumed by the IT equipment of the data center, versus the entire power consumed by the data center. A PUE of 1.0 would indicate a data center where all the power is being consumed by the IT equipment. A PUE greater that 1.0 indicates a data center where a certain amount of power is being consumed by other than IT equipment, the biggest chunk of which is cooling.

The problem I’ve written about before with PUE is the failure to take into account the actual work being accomplished by the IT equipment in the data center. Throw in a pile of extra servers just simply turned on and idling, not doing anything useful, and you’ve just gamed your PUE into looking better.

The problem shown here is even more damning. Microsoft determined that data center energy consumption could be reduced by removing the individual cooling fans from its servers and increasing the size of the data center cooling system. Since the increase in power for the data center cooling systems is less than the power required for the individual server fans, the data center accomplishes the same amount of work for less total energy consumption, an efficiency win in anyone’s book.

The side effect of this is that, even though the total energy consumption for the data center is reduced, transferring the energy usage from the fans (part of the IT equipment number) to the cooling (part of the non-IT equipment number) makes the PUE for the data center look WORSE.

Gaming the metric simply made it inaccurate, which was bad enough. Any efficiency metric that shows a net gain in data center efficiency (same amount of work accomplished for less energy consumed) as a NEGATIVE is hopelessly broken. This also has the side effect of making a mockery of the EPA’s Energy Star for data centers, since that award is based directly on the data center’s PUE.

Misleading, inaccurate, and now totally wrong, this sucker needs to go where all the other bad ideas go to die.

SwiftWater Telecom Green Eco Cabinet Filler Panels, insulated, lightweight, inexpensive

Jackass: data center edition

Vern Burke, SwiftWater Telecom
Biddeford, Maine

I was just reading a piece about data center temperature levels, cooling, and the ASHRAE recommendations. It’s not the ASHRAE recommendations that are the problem here.

First of all, is anyone really still running servers that can’t handle the ASHRAE recommended maximum inlet temperature of 80 degrees for 20 minutes out failing the hardware? My oldest machines in service are older Rackable Systems 2x dual core Opteron 1U boxes with 2004 bios dates on the Tyan motherboards. These servers are not only perfectly happy at the ASHRAE maximum recommendation, they will run rock solid for months at high 90 degree inlet temperatures in free air cooling configurations.

The next thing is, a 30 degree inlet to outlet temperature differential is “good”? Really? My old Rackables with their highly congested 1U cases produce 13 degrees differential between inlet and outlet. If you have a server gaining 30 degrees at the outlet, you have a server that has a severe problem with its internal airflow. Of course, restricting airflow will make the server fans work harder, driving up the amount of power they require.

So, the original poster pulled a truly jackass stunt:

1. He “accepted” and commissioned a new data center without properly testing the systems.

2. He ran live servers in a data center with no power backup for the cooling systems, only the servers themselves.

3. He allowed servers to run to failure in a completely defective environment. Of course, we’re never told what the actual inlet temperature was on the servers when they failed, it could have been far higher than the ASHRAE recommended maximum.

The problem here isn’t the inlet temperature recommendation, it’s a defective data center design combined with defective operations (did anyone do a maintenance plan before running that fire alarm test?).

I guess if you can’t figure out that backup power for the servers isn’t adequate to be running anything without backup power for the cooling, then you should unload lots of money running your data center temperature low enough to give you time to fix a totally preventable goof.

As for the rest of us, we’ll avoid designing and operating in jackass mode.

Thursday data center tidbits: arguing against virtualization, NASA data goes free range

Vern Burke, SwiftWater Telecom
Biddeford, ME

First up today is a question I ran across asking if there are any circumstances that argue against server virtualization in the data center. The only thing I can think of is the requirement for specialized hardware. If your applications require specialized hardware, keep them on their own server, otherwise, get virtualizing!

The next piece is about NASA discovering that they’ve been selling excess PCs without properly cleaning sensitive data off them. And everyone is in a panic over cloud computing being a tremendous security threat? The biggest security threat is leaving your data in the care of people who have no focus on the security of it.

Data center doof of the week goes to Tumblr for managing to kill a database cluster AND kill their network for a day for the sake of scheduled maintenance. Squeaky red noses are enroute!

6 “real” ways to know it’s time to renovate your data center.

I was just reading this piece about 10 ways to tell that your data center is overdue for renovation. Great idea but, unfortunately, that piece was WAY off track, so I’m going to list my 6 ways here.

1. Cooling

You don’t need a fancy expensive air flow study to get an idea that your data center has cooling issues. A simple walk through will make significant hot or cold spots very obvious. Significant hot or cold spots means it’s time to rework things.

2. Space

If you wait until you can’t cram one more piece of gear in, as the article suggests, you’re going to be in a heap of trouble. Make sure all idle equipment is removed and set a reasonable action limit (such as 75% full) to address the space issue BEFORE you run up against the limit.

3. Power

Contrary to the article, reading equipment load information is NOT a sign that your data center needs to be renovated, it’s just good practice. Nuisance trips of breakers and the need to reroute power circuits from other areas of the data center are a dead giveaway that the original power plan for the data center needs a serious overhaul.

4. Strategy

You can’t create an IT strategy without considering technologies as the article would have you believe. First, inventory and evaluate the existing data center, identify where it’s falling short of meeting business requirements and goals, and then consider the technology to get it back on track. Every step in it’s proper order.

5. Performance

When it becomes apparent that the existing data center infrastructure is going to fail on any of the first four points with anticipated changes coming up, it’s time to retire it. Don’t let the problems occur and THEN fix them.

6. Organization and documentation

If touching any part of the data center is a major crisis because of over complication of the systems and/or inaccurate, incomplete, or just plain missing documentation, it’s a clear signal to get it revamped and under control before it causes a complete disaster.

Thursday data center tidbits: gold plated SSD, more crazy data center metrics

Vern Burke, SwiftWater Telecom
Biddeford, Maine

First up today is the piece about LSI releasing a 300GB SSD solid state drive for almost $12,000. The speed is certainly impressive but is it really worth the price premium between a sub $100 traditional hard drive vs a $12K SSD of the same capacity?

The next piece is about the new data center metrics from The Green Grid, CUE (carbon usage effectiveness) and WUE (water usage effectiveness). Instead of taking the opportunity to create some truly useful and accurate metrics, they’ve created two that suffer from exactly the same faults as PUE (power usage effectiveness), namely that they do not take into account actual work being produced and are ridiculously easy to game. I’ll give The Green Grid credit for swinging, but this is the second and third whiffs in a row.

Finally, we come to more what if cloud computing security hysteria.
From the article comes the quote “What if the service you’re using merges with another company or goes bankrupt?”. Um, exactly the same thing if your co-location provider merges or goes bankrupt. This is NOT a cloud computing specific issue. The “users are uneasy about it” line is NOT valid evidence that there’s anything wrong at all with cloud computing security.

Email or call me or visit the SwiftWater Telecom web site for green data center services today.

Datacenters as corporate welfare: maximum incentives for minimum jobs.

Vern Burke, SwiftWater Telecom
Biddeford. Maine

I’ve just been reading the piece about the lawsuit over the proposed Verizon data center in New York. This story highlights exactly what is wrong with the mega-datacenter feeding frenzy that’s been going on and just how out of control it’s gotten.

The main complaint of the lawsuit is massive cutting of corners in the approval process for this datacenter. Can anyone seriously argue building a million square feet of sprawling mostly single story building has no environmental impact? How about water consumption and release for cooling all those servers? How about air quality requires for all the diesel backup generators for all those servers? I notice that even Microsoft has to get air quality permits for generators in their West Coast datacenters, I wonder what makes Verizon so special that they don’t make any environmental impact doing the same thing.

The characterization of a datacenter as just “a giant warehouse for computers” is a bit astonishing to me. If Verizon managed to convince everyone there was nothing more to a datacenter than that, I’m certainly impressed. Either that or these officials do know better but they think this explanation will suffice for everyone else who doesn’t know any better (move along, nothing to see here!).

Bad as this sort of thing looks, the ugliest part of this are the public subsidies. North Carolina has been a great state for ridiculous giveaways to attract huge datacenters with flashy corporate names and a tiny amount of jobs but this deal makes them look like amateurs. We now know that the going rate for buying jobs with public money is $3.1M each ($614M), surpassing the Yahoo datacenter deal at $2.1M a job. Do the math, how long would it take a modestly paying IT job to pay back $3.1M? And to do this on a 20 year basis? Given the usual life span of datacenters, that’s well more than the likely lifespan of any datacenter being built today, just due to the rapid pace of technology change. Divide it out, for workers to even make $3.1M in wages over 20 years means every job would have to pay in excess of $150,000 a year.

I’ve often remarked that I could do more good for the local community with 1/100th the subsidies being lavished on these projects. Unfortunately, I’m a small operator and the huge flashy photo op projects with famous names attached lend themselves to the publicity (over)hype much better, just as long as you don’t look at the math behind them.

It’s time to stop and do a sanity check on this whole process.

Email or call me or visit the SwiftWater Telecom web site for green data center services today.

Failure to transfer: Bumbling data center power reliability, the iWeb outage.

Vern Burke, SwiftWater Telecom
Biddeford, ME

I’ve just been reading about the recent iWeb data center power failure. Lousy power design and botched operations strikes again.

Even though specifics of iWeb’s data center power configuration weren’t specifically revealed, we can tell a lot from what actually happened. Due to a nearby fire, the data center operators made the decision to shift the facility to emergency power (an entirely reasonable move). The transfer switch serving one of 3 generators failed to transfer, leaving one third of the data center dark when UPS batteries ran out. Where do I start on the boneheaded tricks on this one.

First, we know that the 3 generators were allocated 1 to each third of the facility. This means no generator redundancy. It sounds good to say “we have 3 generators!” until you find out that they’re not being operated in parallel with at least 1 spare (n+1). Right idea, a total swing and whiff on the execution.

Second, it’s apparent that there was no manual bypass for the failed transfer switch. Were they expecting to have to shut down the whole 1/3 of the facility if they ever needed to work on that transfer switch? Dealing with a failed transfer switch shouldn’t be any more difficult than sending someone down to the power room to transfer the power manually.

Third, if they actually did have a manual bypass, were the data center operators informed by the monitoring systems that that section of the data center was still running from UPS and there was enough run time from battery to get someone to the power room to pull the manual bypass? This is a the big problem I have with super short run time backup power such as flywheel UPS. If things don’t go absolutely perfectly in the 15 seconds of runtime you get, you don’t get a chance for a manual fix, you’re going down, period.Of course, splitting the generators into separate “zones” makes the short runtime problem far worse, since it’s much more likely that you’re going to have a total failure with a single generator.

It’s apparent from the article a number of large name providers are doing a similarly lousy job at their backup power redundancy, judging by four transfer switch failures this year with major loss of data center services each time. It’s really a rather pathetic performance.

So, what’s the takeaway from all of this?

1. If you’re going to run multiple generators, run them in parallel and at least n+1. I don’t care how many generators you have, if you’re allocating single generators to single zones, you’re vulnerable.

2. If you’re not going to run the generators in parallel, at least give enough run time from the batteries to deal with the problems you know are going to come up. I don’t care how often you test, if you’re running single generators, failure is going to happen (with this configuration, they could have easily have had this happen during a test!).

3. Make sure there’s a manual bypass for automatic transfer switches and that your operations people have the monitoring and the procedure to know when to pull it.

In a substantially sized data center, the consequences of failing to transfer are a lot worse than doing things right the first time.

iWeb, data center bozos of the week (squeaky red noses are on the way!).

Email or call me or visit the SwiftWater Telecom web site for green data center services today.

Lipstick on a pig: Facebook’s data center refit.

Vern Burke, SwiftWater Telecom
Biddeford, ME

I’ve just been reading an article today about Facebook retrofitting a data center and all the great energy efficiency gains. Unfortunately, sometimes the best retrofit method for a data center is dynamite.

Most of the modifications mentioned have to do with airflow. Now, I’ll be the first one to cheer for improving and controlling airflow for improving data center efficiency. The problem is, how BAD does your airflow situation have to be to have to run the cold air temperature at 51 degrees F?! I though data centers running in the 60s were out of date, 51 is just pathetic. It’s obvious that there was certainly room for improvement here, but the improvement effort only got them to 67 and that’s still lousy.

The big problem here comes from continued reliance on the obsolete raised floor as a plenum design. There are certainly far more reasons not to use raised flooring in a data center, including unproductive floor loading, expense, fire detection and suppression requirements, under floor housekeeping, metal whisker contamination, and a whole host of airflow issues. Since the Facebook retrofit is all about the airflow, I’m going to just address the raised floor airflow issues.

If you’re really serious about balancing your data center airflow, using a raised floor as a plenum is the last thing to do. First, under floor obstructions make smooth airflow next to impossible, even if you’re totally conscientious about housekeeping. Second, there’s zip for fine control of where the air is going. Need to add just a small amount of air here? Sorry, you take multiples of full tiles or nothing. Third, pull a tile to work on underfloor facilities and you immediately unbalance the entire system. Pull a dozen tiles to put in a cable run and you now have complete chaos across the whole floor. Finally, make any changes to your equipment and you have to rebalance the whole thing.

These things are so inefficient that it isn’t any wonder that a lousy design would need ridiculously cold air to make it work. 67 is certainly an improvement, now they’ve gotten things up to being only 5-10 years out of date.

When Facebook actually retrofits a data center all the way up to modern standards, I’ll be impressed. This operation is still a pig underneath, no matter how much lipstick you put on it.

—

Tuesday data center tidbits: PayPal doesn’t pay, scary virtual servers

Vern Burke, SwiftWater Telecom
Biddeford, ME

First up today is the piece about the recent 3+ hour data center failure at PayPal. Yet another self inflicted injury by a major company who should know better. Just in case the message isn’t clear, it might be a good idea to actually TEST your failover procedures BEFORE things actually fail. (Hmmmm, I wonder where you’d put the parking brake in a data center 🙂 )

Next is the piece about the “scary side” of virtual servers. Everyone is having major angst over theoretical security “problems” when all of the issues I see are failure to follow the same well established best practices that work just fine for physical machines. It’s far easier for an attacker to compromise a virtual server with bad sysadmin practices than to invent a totally new hypervisor attack. Oh, and if you can’t trust your data center admin to do his job without having to be physically roadblocked by other departments, you have worse problems than virtualization.

Email or call me or visit the SwiftWater Telecom web site for green data center services today.

Monday data center tidbits: solar powering the data center, cleaning up after yourself virtually

Vern Burke, SwiftWater Telecom
Biddeford, Maine

First up today is a piece about solar power efficiency and whether it is practical to run a data center from 100% solar (it’s worth noting that all of the “100% solar” data centers that I’ve seen are very tiny). It’s going to take a long time before photovoltaic solar power is efficient enough to run a large load like a data center but any load you can shed is a benefit to the bottom line, not to mention shading the data center roof to cut down cooling costs.

Next up is the piece about virtual machine ghosts in the data center. There’s no substitute for paying attention and keeping control of what you’re doing, this isn’t rocket science. These people would have to do exactly the same thing with physical servers, why is it such an awful strain to do it with virtual ones?

Email or call me or visit the SwiftWater Telecom web site for green data center services today.