Tag Archives: data center facility

6 “real” ways to know it’s time to renovate your data center.

I was just reading this piece about 10 ways to tell that your data center is overdue for renovation. Great idea but, unfortunately, that piece was WAY off track, so I’m going to list my 6 ways here.

1. Cooling

You don’t need a fancy expensive air flow study to get an idea that your data center has cooling issues. A simple walk through will make significant hot or cold spots very obvious. Significant hot or cold spots means it’s time to rework things.

2. Space

If you wait until you can’t cram one more piece of gear in, as the article suggests, you’re going to be in a heap of trouble. Make sure all idle equipment is removed and set a reasonable action limit (such as 75% full) to address the space issue BEFORE you run up against the limit.

3. Power

Contrary to the article, reading equipment load information is NOT a sign that your data center needs to be renovated, it’s just good practice. Nuisance trips of breakers and the need to reroute power circuits from other areas of the data center are a dead giveaway that the original power plan for the data center needs a serious overhaul.

4. Strategy

You can’t create an IT strategy without considering technologies as the article would have you believe. First, inventory and evaluate the existing data center, identify where it’s falling short of meeting business requirements and goals, and then consider the technology to get it back on track. Every step in it’s proper order.

5. Performance

When it becomes apparent that the existing data center infrastructure is going to fail on any of the first four points with anticipated changes coming up, it’s time to retire it. Don’t let the problems occur and THEN fix them.

6. Organization and documentation

If touching any part of the data center is a major crisis because of over complication of the systems and/or inaccurate, incomplete, or just plain missing documentation, it’s a clear signal to get it revamped and under control before it causes a complete disaster.

Datacenters as corporate welfare: maximum incentives for minimum jobs.

Vern Burke, SwiftWater Telecom
Biddeford. Maine

I’ve just been reading the piece about the lawsuit over the proposed Verizon data center in New York. This story highlights exactly what is wrong with the mega-datacenter feeding frenzy that’s been going on and just how out of control it’s gotten.

The main complaint of the lawsuit is massive cutting of corners in the approval process for this datacenter. Can anyone seriously argue building a million square feet of sprawling mostly single story building has no environmental impact? How about water consumption and release for cooling all those servers? How about air quality requires for all the diesel backup generators for all those servers? I notice that even Microsoft has to get air quality permits for generators in their West Coast datacenters, I wonder what makes Verizon so special that they don’t make any environmental impact doing the same thing.

The characterization of a datacenter as just “a giant warehouse for computers” is a bit astonishing to me. If Verizon managed to convince everyone there was nothing more to a datacenter than that, I’m certainly impressed. Either that or these officials do know better but they think this explanation will suffice for everyone else who doesn’t know any better (move along, nothing to see here!).

Bad as this sort of thing looks, the ugliest part of this are the public subsidies. North Carolina has been a great state for ridiculous giveaways to attract huge datacenters with flashy corporate names and a tiny amount of jobs but this deal makes them look like amateurs. We now know that the going rate for buying jobs with public money is $3.1M each ($614M), surpassing the Yahoo datacenter deal at $2.1M a job. Do the math, how long would it take a modestly paying IT job to pay back $3.1M? And to do this on a 20 year basis? Given the usual life span of datacenters, that’s well more than the likely lifespan of any datacenter being built today, just due to the rapid pace of technology change. Divide it out, for workers to even make $3.1M in wages over 20 years means every job would have to pay in excess of $150,000 a year.

I’ve often remarked that I could do more good for the local community with 1/100th the subsidies being lavished on these projects. Unfortunately, I’m a small operator and the huge flashy photo op projects with famous names attached lend themselves to the publicity (over)hype much better, just as long as you don’t look at the math behind them.

It’s time to stop and do a sanity check on this whole process.

Email or call me or visit the SwiftWater Telecom web site for green data center services today.

swiftwater telecom rcs cloud computing logo

Failure to transfer: Bumbling data center power reliability, the iWeb outage.

Vern Burke, SwiftWater Telecom
Biddeford, ME

I’ve just been reading about the recent iWeb data center power failure. Lousy power design and botched operations strikes again.

Even though specifics of iWeb’s data center power configuration weren’t specifically revealed, we can tell a lot from what actually happened. Due to a nearby fire, the data center operators made the decision to shift the facility to emergency power (an entirely reasonable move). The transfer switch serving one of 3 generators failed to transfer, leaving one third of the data center dark when UPS batteries ran out. Where do I start on the boneheaded tricks on this one.

First, we know that the 3 generators were allocated 1 to each third of the facility. This means no generator redundancy. It sounds good to say “we have 3 generators!” until you find out that they’re not being operated in parallel with at least 1 spare (n+1). Right idea, a total swing and whiff on the execution.

Second, it’s apparent that there was no manual bypass for the failed transfer switch. Were they expecting to have to shut down the whole 1/3 of the facility if they ever needed to work on that transfer switch? Dealing with a failed transfer switch shouldn’t be any more difficult than sending someone down to the power room to transfer the power manually.

Third, if they actually did have a manual bypass, were the data center operators informed by the monitoring systems that that section of the data center was still running from UPS and there was enough run time from battery to get someone to the power room to pull the manual bypass? This is a the big problem I have with super short run time backup power such as flywheel UPS. If things don’t go absolutely perfectly in the 15 seconds of runtime you get, you don’t get a chance for a manual fix, you’re going down, period.Of course, splitting the generators into separate “zones” makes the short runtime problem far worse, since it’s much more likely that you’re going to have a total failure with a single generator.

It’s apparent from the article a number of large name providers are doing a similarly lousy job at their backup power redundancy, judging by four transfer switch failures this year with major loss of data center services each time. It’s really a rather pathetic performance.

So, what’s the takeaway from all of this?

1. If you’re going to run multiple generators, run them in parallel and at least n+1. I don’t care how many generators you have, if you’re allocating single generators to single zones, you’re vulnerable.

2. If you’re not going to run the generators in parallel, at least give enough run time from the batteries to deal with the problems you know are going to come up. I don’t care how often you test, if you’re running single generators, failure is going to happen (with this configuration, they could have easily have had this happen during a test!).

3. Make sure there’s a manual bypass for automatic transfer switches and that your operations people have the monitoring and the procedure to know when to pull it.

In a substantially sized data center, the consequences of failing to transfer are a lot worse than doing things right the first time.

iWeb, data center bozos of the week (squeaky red noses are on the way!).

Email or call me or visit the SwiftWater Telecom web site for green data center services today.

swiftwater telecom rcs cloud computing logo

Lipstick on a pig: Facebook’s data center refit.

Vern Burke, SwiftWater Telecom
Biddeford, ME

I’ve just been reading an article today about Facebook retrofitting a data center and all the great energy efficiency gains. Unfortunately, sometimes the best retrofit method for a data center is dynamite.

Most of the modifications mentioned have to do with airflow. Now, I’ll be the first one to cheer for improving and controlling airflow for improving data center efficiency. The problem is, how BAD does your airflow situation have to be to have to run the cold air temperature at 51 degrees F?! I though data centers running in the 60s were out of date, 51 is just pathetic. It’s obvious that there was certainly room for improvement here, but the improvement effort only got them to 67 and that’s still lousy.

The big problem here comes from continued reliance on the obsolete raised floor as a plenum design. There are certainly far more reasons not to use raised flooring in a data center, including unproductive floor loading, expense, fire detection and suppression requirements, under floor housekeeping, metal whisker contamination, and a whole host of airflow issues. Since the Facebook retrofit is all about the airflow, I’m going to just address the raised floor airflow issues.

If you’re really serious about balancing your data center airflow, using a raised floor as a plenum is the last thing to do. First, under floor obstructions make smooth airflow next to impossible, even if you’re totally conscientious about housekeeping. Second, there’s zip for fine control of where the air is going. Need to add just a small amount of air here? Sorry, you take multiples of full tiles or nothing. Third, pull a tile to work on underfloor facilities and you immediately unbalance the entire system. Pull a dozen tiles to put in a cable run and you now have complete chaos across the whole floor. Finally, make any changes to your equipment and you have to rebalance the whole thing.

These things are so inefficient that it isn’t any wonder that a lousy design would need ridiculously cold air to make it work. 67 is certainly an improvement, now they’ve gotten things up to being only 5-10 years out of date.

When Facebook actually retrofits a data center all the way up to modern standards, I’ll be impressed. This operation is still a pig underneath, no matter how much lipstick you put on it.

Building out the data center the right way.

Tonight I’ve been reading an article about data center building trends. There’s some very good points to this and also some things that I think are very wrong. These also explain some things that have mystified me for some time.

Looking ahead 5 years for capacity planning isn’t a bad idea (except that the data center needs to be flexible enough to accommodate the changes that can happen in 5 years), but the whole decision on build out or not for data center infrastructure in advance hinges on the penalty for doing so. In short, there’s no penalty for building out passive infrastructure and a big penalty for building out active infrastructure.

I’ve been mystified by the idea that data center PUE (power usage effectiveness) only gets good when a data center is full. Now I understand, this is based on the idea of a data center building out (and operating) 100% of it’s cooling infrastructure in advance. If you’re only running 20% of your 5 year forecasted server capacity but you have to run 100% of your 5 year forecasted cooling capacity because it’s a monolithic system that’s either on or off, of course your efficiency is going to stink!

The PUE numbers for that kind of arrangement would be pathetic. Of course, as you add servers with the same amount of cooling power usage, the PUE would gradually get better, but who in the world would really WANT to run something that way? (Reference last year’s story about UPS cutting their data center power by switching off 30 out of 65 HVAC units!)

Leaving plenty of extra room for the transformer yard and the generator yard is a great idea (you can’t expand them once you get things built in around them). On the other hand, it would be silly to build out and power up a yard full of transformers that were sitting there doing nothing except chewing up power.

So, what sort of data center infrastructure things can safely be built out far ahead of the actual need? The physical building is a bit of a no-brainer, as long as the active sections can be isolated from the idle (you don’t need to be cooling an entire hall that’s only 10% full).

Passive electrical is another good one. This means entrances, disconnects, breaker panels, distribution cabling, transfer switches. No UPS, no DC power plants unless you’re going to be able to shut them off and leave them off until you really need them.

Passive cooling infrastructure such as ducts. Take a lesson from UPS, do NOT build out double the HVAC you need and run it all!

Finally, build out the support structures for the transformer and generator yards. Mounting pads, conduit, cabling, so the equipment is all set to drop, hook up, and go.

Don’t kill your data center’s efficiency in return for capacity 5 years from now.

Email or call me or visit the SwiftWater Telecom web site for green data center services today.


swiftwater telecom rcs cloud computing logo

Mucking out the data center.

I’ve just been reading this piece about data center contamination. There’s certainly a few good points in there but I think it misses the point by a mile on a lot of it.

There are certainly a number of contaminants that can play havoc with a data center. Copier and laser printer toner is definitely a bad one, very abrasive. Airborne sand and volcanic ash are also terribly abrasive as well. It’s also true that gaseous contaminates can cause oxidation, but I’m not sure what the average data center could reasonably do to counteract that.

At this point, I diverge. First of all, cleaning a data center to “clean room” standards is ridiculous. Most data centers are NOT built as clean rooms and the first person to open a door or the first puff of cooling air under a raised floor used as a plenum is going to blow that in a big hurry, not to mention that the article recommends doing this BEFORE raised floors, servers, and cable racks are installed. What about the dirt involved in this?

Second, what data center equipment really needs or would even measurably benefit from this level of clean? Data center equipment will usually go FAR beyond its useful lifespan with a lot less of a cleaning level than this. Cleaning the data center as if the same IT equipment was going to be run for 20 years is an awful waste of money.

Sticking with hard surfaces and avoiding fabrics in the data center is certainly a good idea, as are mats to trap as much incoming dirt as possible. Proper filters for air handling equipment is also a smart idea.

Zinc whiskers are probably the only contaminant that stand a significant chance in tiny quantities of causing an IT equipment failure in the data center. Zinc whiskers are formed on the surface of zinc plated metal, such as raised floor tiles and frames. The little whiskers break off on air flow (such as using a raised floor for a cold air plenum) and the head right for vulnerable parts of IT equipment, such as power supplies.

There are a bunch of good reasons NOT to used raised floor in the modern data center (added floor weight for no benefit, expense, under floor fire suppression, etc) but zinc whiskers and the inability to decently clean under them are the biggest. Dump the raised floor and you dump nearly everything that could really cause your equipment problems, as well as a major source of dirt.

Cleanliness in the data center is certainly an important issue, but keeping known offenders like raised flooring out and doing smart, cost effective cleaning instead of trying to achieve clean room perfection will be a lot more sensible.

Email or call me or visit the SwiftWater Telecom web site for green data center services today.


swiftwater telecom rcs cloud computing logo