Tag Archives: bozo

Finally, proof positive that PUE is garbage.

Vern Burke, SwiftWater Telecom
Biddeford, ME

I’ve just been reading a piece about Microsoft removing fans from their data center servers and that having a negative effect on their PUE numbers. I’ve written on this blog before about the problems with PUE, now we have proof that it needs to be put out of it’s misery.

In a nutshell, PUE is the ratio of power consumed by the IT equipment of the data center, versus the entire power consumed by the data center. A PUE of 1.0 would indicate a data center where all the power is being consumed by the IT equipment. A PUE greater that 1.0 indicates a data center where a certain amount of power is being consumed by other than IT equipment, the biggest chunk of which is cooling.

The problem I’ve written about before with PUE is the failure to take into account the actual work being accomplished by the IT equipment in the data center. Throw in a pile of extra servers just simply turned on and idling, not doing anything useful, and you’ve just gamed your PUE into looking better.

The problem shown here is even more damning. Microsoft determined that data center energy consumption could be reduced by removing the individual cooling fans from its servers and increasing the size of the data center cooling system. Since the increase in power for the data center cooling systems is less than the power required for the individual server fans, the data center accomplishes the same amount of work for less total energy consumption, an efficiency win in anyone’s book.

The side effect of this is that, even though the total energy consumption for the data center is reduced, transferring the energy usage from the fans (part of the IT equipment number) to the cooling (part of the non-IT equipment number) makes the PUE for the data center look WORSE.

Gaming the metric simply made it inaccurate, which was bad enough. Any efficiency metric that shows a net gain in data center efficiency (same amount of work accomplished for less energy consumed) as a NEGATIVE is hopelessly broken. This also has the side effect of making a mockery of the EPA’s Energy Star for data centers, since that award is based directly on the data center’s PUE.

Misleading, inaccurate, and now totally wrong, this sucker needs to go where all the other bad ideas go to die.

SwiftWater Telecom Green Eco Cabinet Filler Panels, insulated, lightweight, inexpensive

Jackass: data center edition

Vern Burke, SwiftWater Telecom
Biddeford, Maine

I was just reading a piece about data center temperature levels, cooling, and the ASHRAE recommendations. It’s not the ASHRAE recommendations that are the problem here.

First of all, is anyone really still running servers that can’t handle the ASHRAE recommended maximum inlet temperature of 80 degrees for 20 minutes out failing the hardware? My oldest machines in service are older Rackable Systems 2x dual core Opteron 1U boxes with 2004 bios dates on the Tyan motherboards. These servers are not only perfectly happy at the ASHRAE maximum recommendation, they will run rock solid for months at high 90 degree inlet temperatures in free air cooling configurations.

The next thing is, a 30 degree inlet to outlet temperature differential is “good”? Really? My old Rackables with their highly congested 1U cases produce 13 degrees differential between inlet and outlet. If you have a server gaining 30 degrees at the outlet, you have a server that has a severe problem with its internal airflow. Of course, restricting airflow will make the server fans work harder, driving up the amount of power they require.

So, the original poster pulled a truly jackass stunt:

1. He “accepted” and commissioned a new data center without properly testing the systems.

2. He ran live servers in a data center with no power backup for the cooling systems, only the servers themselves.

3. He allowed servers to run to failure in a completely defective environment. Of course, we’re never told what the actual inlet temperature was on the servers when they failed, it could have been far higher than the ASHRAE recommended maximum.

The problem here isn’t the inlet temperature recommendation, it’s a defective data center design combined with defective operations (did anyone do a maintenance plan before running that fire alarm test?).

I guess if you can’t figure out that backup power for the servers isn’t adequate to be running anything without backup power for the cooling, then you should unload lots of money running your data center temperature low enough to give you time to fix a totally preventable goof.

As for the rest of us, we’ll avoid designing and operating in jackass mode.

Failure to transfer: Bumbling data center power reliability, the iWeb outage.

Vern Burke, SwiftWater Telecom
Biddeford, ME

I’ve just been reading about the recent iWeb data center power failure. Lousy power design and botched operations strikes again.

Even though specifics of iWeb’s data center power configuration weren’t specifically revealed, we can tell a lot from what actually happened. Due to a nearby fire, the data center operators made the decision to shift the facility to emergency power (an entirely reasonable move). The transfer switch serving one of 3 generators failed to transfer, leaving one third of the data center dark when UPS batteries ran out. Where do I start on the boneheaded tricks on this one.

First, we know that the 3 generators were allocated 1 to each third of the facility. This means no generator redundancy. It sounds good to say “we have 3 generators!” until you find out that they’re not being operated in parallel with at least 1 spare (n+1). Right idea, a total swing and whiff on the execution.

Second, it’s apparent that there was no manual bypass for the failed transfer switch. Were they expecting to have to shut down the whole 1/3 of the facility if they ever needed to work on that transfer switch? Dealing with a failed transfer switch shouldn’t be any more difficult than sending someone down to the power room to transfer the power manually.

Third, if they actually did have a manual bypass, were the data center operators informed by the monitoring systems that that section of the data center was still running from UPS and there was enough run time from battery to get someone to the power room to pull the manual bypass? This is a the big problem I have with super short run time backup power such as flywheel UPS. If things don’t go absolutely perfectly in the 15 seconds of runtime you get, you don’t get a chance for a manual fix, you’re going down, period.Of course, splitting the generators into separate “zones” makes the short runtime problem far worse, since it’s much more likely that you’re going to have a total failure with a single generator.

It’s apparent from the article a number of large name providers are doing a similarly lousy job at their backup power redundancy, judging by four transfer switch failures this year with major loss of data center services each time. It’s really a rather pathetic performance.

So, what’s the takeaway from all of this?

1. If you’re going to run multiple generators, run them in parallel and at least n+1. I don’t care how many generators you have, if you’re allocating single generators to single zones, you’re vulnerable.

2. If you’re not going to run the generators in parallel, at least give enough run time from the batteries to deal with the problems you know are going to come up. I don’t care how often you test, if you’re running single generators, failure is going to happen (with this configuration, they could have easily have had this happen during a test!).

3. Make sure there’s a manual bypass for automatic transfer switches and that your operations people have the monitoring and the procedure to know when to pull it.

In a substantially sized data center, the consequences of failing to transfer are a lot worse than doing things right the first time.

iWeb, data center bozos of the week (squeaky red noses are on the way!).

Email or call me or visit the SwiftWater Telecom web site for green data center services today.

swiftwater telecom rcs cloud computing logo

Friday data center tidbits: data centers gone wild, Facebook behind the times

First up today is the piece about the federal government “finding” 1000 more data centers than they thought they had. First of all, how does your data center inventory get so bad that you lose track of 1000 data centers? Second, how in the world do you allow the data center sprawl to get so far out of control? That’s a total of 2100 federal data centers, an average of 42 data centers for every single state. Last but not least, who in the world would think it’s a bad idea to consolidate these?

The federal goverment, data center bozos of the week, the truckloads of red squeaky noses are on their way.

The next piece is about Facebook saving big by retooling its data center cooling. Really, is it big news that not mixing cold intake air with hot exhaust air is a good idea? If Facebook is pushing this as a “look how great we are” point, they’re about 5 years too late.

Finally, here’s a survey about the causes of data center downtime. Not maintaining the data center backup batteries and overloading the backup power are just plain silly, but the telling one for me is 51% accidental human error, including false activation of the EPO (emergency power off). I’ve said it before, the gains that the National Electrical Code allows the data center in exchange for the EPO are NOT worth this kind of failure. EPO=EVIL, period.

Email or call me or visit the SwiftWater Telecom web site for green data center services today.


swiftwater telecom rcs cloud computing logo

Thursday data center tidbits: Chaos at Facebook (again)

For more than 5 hours now, developers have been unable to make changes to the Facebook Integration settings for their apps. Attempts to change these settings returns:

Sorry, something went wrong.

We’re working on getting this fixed as soon as we can.

This failure doesn’t seem to be affecting apps that are currently running but it has dragged a fair amount of app development right down to a total stop.

This failure comes close behind the recent major Facebook outage caused by a runaway software integrity checker.


SwiftWater Telecom

Monday data center tidbits: Unhyping the cloud, Facebook INsanity checks

First up today is a piece about cutting out the cloud computing hype. The problem isn’t so much the hype over cloud computing as it is the rampant outright misuse of the term cloud, attaching it to things aren’t even remotely cloudy in an attempt to ride cloud computing’s coattails without actually making the effort to DO cloud computing.

Next up is the recent Facebook service fiasco. Getting your system to mark it’s own code as invalid and not hand the problem to a human to validate before taking radical action is especially brain dead. On top of that, now you have an error compounded on top of the original error and you have a cascading failure, all because of no sanity checking and no break for human input. Facebook gets our data center bozos of the day award for trusting too much in automation and then blowing it.

Email or call me or visit the SwiftWater Telecom web site for green data center services today.


swiftwater telecom rcs cloud computing logo