Data center design for bozos: Social Security’s National Computing Center

Vern Burke, SwiftWater Telecom
Biddeford, ME

I’ve just been reading about the wear and tear problems with the Social Security Administration’s National Computing Center. The interesting part of this isn’t the race against the clock to get it replaced before it falls apart, but just how badly it was designed in the first place.

Granted, 30 years of un-upgraded wear and tear on a data center is bad news. The interesting thing to me is that the SSA Deputy Commissioner for Systems claims that the data center was built to the industry best practices of the time. Let’s take a look at those “best practices”.

1. “Let’s build it with only one HVAC system!”

Redundancy shmundancy, these things never fail, right?

2. “Let’s build it with only one UPS system!”

See #1

3. “We’re special, let’s custom build one single huge UPS system!”

Sure, take your number one most critical system in the whole data center (just ahead of the HVAC), provide no redundancy for it, and build a custom one-off that nobody may remember how to maintain in 10 years, much less 30. Brilliant.

4. “Let’s use a raised floor!”

I’ve written before on this blog about the evils of computer room raised floor, but the picture with the original article pretty much says it all. Pack 30 years of cable higgledy piggledy under your raised floor (probably with tons of defunct cable left in place, since I doubt you could pull anything out of that mess) and see how well your cooling air gets to the server racks. Brilliant again.

5. “Let’s run infrastructure pipes through the electrical spaces!”

It’s common knowledge that water and electricity don’t mix, so running any water pipe where a leak can effect anything electrical is fairly dumb. Even worse, think about having plumbers working directly over a battery string or another electrical system. Picture that plumber dropping a water pipe into the battery string bus bars.

6. “Let’s make sure we provide a non-redundant water feed for our critical but also non-redundant HVAC!”

See #1

I’m sure these are “best practices” on a planet in some alternate universe. In this universe, they just make you look like a bozo.

6 “real” ways to know it’s time to renovate your data center.

I was just reading this piece about 10 ways to tell that your data center is overdue for renovation. Great idea but, unfortunately, that piece was WAY off track, so I’m going to list my 6 ways here.

1. Cooling

You don’t need a fancy expensive air flow study to get an idea that your data center has cooling issues. A simple walk through will make significant hot or cold spots very obvious. Significant hot or cold spots means it’s time to rework things.

2. Space

If you wait until you can’t cram one more piece of gear in, as the article suggests, you’re going to be in a heap of trouble. Make sure all idle equipment is removed and set a reasonable action limit (such as 75% full) to address the space issue BEFORE you run up against the limit.

3. Power

Contrary to the article, reading equipment load information is NOT a sign that your data center needs to be renovated, it’s just good practice. Nuisance trips of breakers and the need to reroute power circuits from other areas of the data center are a dead giveaway that the original power plan for the data center needs a serious overhaul.

4. Strategy

You can’t create an IT strategy without considering technologies as the article would have you believe. First, inventory and evaluate the existing data center, identify where it’s falling short of meeting business requirements and goals, and then consider the technology to get it back on track. Every step in it’s proper order.

5. Performance

When it becomes apparent that the existing data center infrastructure is going to fail on any of the first four points with anticipated changes coming up, it’s time to retire it. Don’t let the problems occur and THEN fix them.

6. Organization and documentation

If touching any part of the data center is a major crisis because of over complication of the systems and/or inaccurate, incomplete, or just plain missing documentation, it’s a clear signal to get it revamped and under control before it causes a complete disaster.

Datacenters as corporate welfare: maximum incentives for minimum jobs.

Vern Burke, SwiftWater Telecom
Biddeford. Maine

I’ve just been reading the piece about the lawsuit over the proposed Verizon data center in New York. This story highlights exactly what is wrong with the mega-datacenter feeding frenzy that’s been going on and just how out of control it’s gotten.

The main complaint of the lawsuit is massive cutting of corners in the approval process for this datacenter. Can anyone seriously argue building a million square feet of sprawling mostly single story building has no environmental impact? How about water consumption and release for cooling all those servers? How about air quality requires for all the diesel backup generators for all those servers? I notice that even Microsoft has to get air quality permits for generators in their West Coast datacenters, I wonder what makes Verizon so special that they don’t make any environmental impact doing the same thing.

The characterization of a datacenter as just “a giant warehouse for computers” is a bit astonishing to me. If Verizon managed to convince everyone there was nothing more to a datacenter than that, I’m certainly impressed. Either that or these officials do know better but they think this explanation will suffice for everyone else who doesn’t know any better (move along, nothing to see here!).

Bad as this sort of thing looks, the ugliest part of this are the public subsidies. North Carolina has been a great state for ridiculous giveaways to attract huge datacenters with flashy corporate names and a tiny amount of jobs but this deal makes them look like amateurs. We now know that the going rate for buying jobs with public money is $3.1M each ($614M), surpassing the Yahoo datacenter deal at $2.1M a job. Do the math, how long would it take a modestly paying IT job to pay back $3.1M? And to do this on a 20 year basis? Given the usual life span of datacenters, that’s well more than the likely lifespan of any datacenter being built today, just due to the rapid pace of technology change. Divide it out, for workers to even make $3.1M in wages over 20 years means every job would have to pay in excess of $150,000 a year.

I’ve often remarked that I could do more good for the local community with 1/100th the subsidies being lavished on these projects. Unfortunately, I’m a small operator and the huge flashy photo op projects with famous names attached lend themselves to the publicity (over)hype much better, just as long as you don’t look at the math behind them.

It’s time to stop and do a sanity check on this whole process.

Email or call me or visit the SwiftWater Telecom web site for green data center services today.

Failure to transfer: Bumbling data center power reliability, the iWeb outage.

Vern Burke, SwiftWater Telecom
Biddeford, ME

I’ve just been reading about the recent iWeb data center power failure. Lousy power design and botched operations strikes again.

Even though specifics of iWeb’s data center power configuration weren’t specifically revealed, we can tell a lot from what actually happened. Due to a nearby fire, the data center operators made the decision to shift the facility to emergency power (an entirely reasonable move). The transfer switch serving one of 3 generators failed to transfer, leaving one third of the data center dark when UPS batteries ran out. Where do I start on the boneheaded tricks on this one.

First, we know that the 3 generators were allocated 1 to each third of the facility. This means no generator redundancy. It sounds good to say “we have 3 generators!” until you find out that they’re not being operated in parallel with at least 1 spare (n+1). Right idea, a total swing and whiff on the execution.

Second, it’s apparent that there was no manual bypass for the failed transfer switch. Were they expecting to have to shut down the whole 1/3 of the facility if they ever needed to work on that transfer switch? Dealing with a failed transfer switch shouldn’t be any more difficult than sending someone down to the power room to transfer the power manually.

Third, if they actually did have a manual bypass, were the data center operators informed by the monitoring systems that that section of the data center was still running from UPS and there was enough run time from battery to get someone to the power room to pull the manual bypass? This is a the big problem I have with super short run time backup power such as flywheel UPS. If things don’t go absolutely perfectly in the 15 seconds of runtime you get, you don’t get a chance for a manual fix, you’re going down, period.Of course, splitting the generators into separate “zones” makes the short runtime problem far worse, since it’s much more likely that you’re going to have a total failure with a single generator.

It’s apparent from the article a number of large name providers are doing a similarly lousy job at their backup power redundancy, judging by four transfer switch failures this year with major loss of data center services each time. It’s really a rather pathetic performance.

So, what’s the takeaway from all of this?

1. If you’re going to run multiple generators, run them in parallel and at least n+1. I don’t care how many generators you have, if you’re allocating single generators to single zones, you’re vulnerable.

2. If you’re not going to run the generators in parallel, at least give enough run time from the batteries to deal with the problems you know are going to come up. I don’t care how often you test, if you’re running single generators, failure is going to happen (with this configuration, they could have easily have had this happen during a test!).

3. Make sure there’s a manual bypass for automatic transfer switches and that your operations people have the monitoring and the procedure to know when to pull it.

In a substantially sized data center, the consequences of failing to transfer are a lot worse than doing things right the first time.

iWeb, data center bozos of the week (squeaky red noses are on the way!).

Email or call me or visit the SwiftWater Telecom web site for green data center services today.

Lipstick on a pig: Facebook’s data center refit.

Vern Burke, SwiftWater Telecom
Biddeford, ME

I’ve just been reading an article today about Facebook retrofitting a data center and all the great energy efficiency gains. Unfortunately, sometimes the best retrofit method for a data center is dynamite.

Most of the modifications mentioned have to do with airflow. Now, I’ll be the first one to cheer for improving and controlling airflow for improving data center efficiency. The problem is, how BAD does your airflow situation have to be to have to run the cold air temperature at 51 degrees F?! I though data centers running in the 60s were out of date, 51 is just pathetic. It’s obvious that there was certainly room for improvement here, but the improvement effort only got them to 67 and that’s still lousy.

The big problem here comes from continued reliance on the obsolete raised floor as a plenum design. There are certainly far more reasons not to use raised flooring in a data center, including unproductive floor loading, expense, fire detection and suppression requirements, under floor housekeeping, metal whisker contamination, and a whole host of airflow issues. Since the Facebook retrofit is all about the airflow, I’m going to just address the raised floor airflow issues.

If you’re really serious about balancing your data center airflow, using a raised floor as a plenum is the last thing to do. First, under floor obstructions make smooth airflow next to impossible, even if you’re totally conscientious about housekeeping. Second, there’s zip for fine control of where the air is going. Need to add just a small amount of air here? Sorry, you take multiples of full tiles or nothing. Third, pull a tile to work on underfloor facilities and you immediately unbalance the entire system. Pull a dozen tiles to put in a cable run and you now have complete chaos across the whole floor. Finally, make any changes to your equipment and you have to rebalance the whole thing.

These things are so inefficient that it isn’t any wonder that a lousy design would need ridiculously cold air to make it work. 67 is certainly an improvement, now they’ve gotten things up to being only 5-10 years out of date.

When Facebook actually retrofits a data center all the way up to modern standards, I’ll be impressed. This operation is still a pig underneath, no matter how much lipstick you put on it.

—

Building out the data center the right way.

Tonight I’ve been reading an article about data center building trends. There’s some very good points to this and also some things that I think are very wrong. These also explain some things that have mystified me for some time.

Looking ahead 5 years for capacity planning isn’t a bad idea (except that the data center needs to be flexible enough to accommodate the changes that can happen in 5 years), but the whole decision on build out or not for data center infrastructure in advance hinges on the penalty for doing so. In short, there’s no penalty for building out passive infrastructure and a big penalty for building out active infrastructure.

I’ve been mystified by the idea that data center PUE (power usage effectiveness) only gets good when a data center is full. Now I understand, this is based on the idea of a data center building out (and operating) 100% of it’s cooling infrastructure in advance. If you’re only running 20% of your 5 year forecasted server capacity but you have to run 100% of your 5 year forecasted cooling capacity because it’s a monolithic system that’s either on or off, of course your efficiency is going to stink!

The PUE numbers for that kind of arrangement would be pathetic. Of course, as you add servers with the same amount of cooling power usage, the PUE would gradually get better, but who in the world would really WANT to run something that way? (Reference last year’s story about UPS cutting their data center power by switching off 30 out of 65 HVAC units!)

Leaving plenty of extra room for the transformer yard and the generator yard is a great idea (you can’t expand them once you get things built in around them). On the other hand, it would be silly to build out and power up a yard full of transformers that were sitting there doing nothing except chewing up power.

So, what sort of data center infrastructure things can safely be built out far ahead of the actual need? The physical building is a bit of a no-brainer, as long as the active sections can be isolated from the idle (you don’t need to be cooling an entire hall that’s only 10% full).

Passive electrical is another good one. This means entrances, disconnects, breaker panels, distribution cabling, transfer switches. No UPS, no DC power plants unless you’re going to be able to shut them off and leave them off until you really need them.

Passive cooling infrastructure such as ducts. Take a lesson from UPS, do NOT build out double the HVAC you need and run it all!

Finally, build out the support structures for the transformer and generator yards. Mounting pads, conduit, cabling, so the equipment is all set to drop, hook up, and go.

Don’t kill your data center’s efficiency in return for capacity 5 years from now.

Email or call me or visit the SwiftWater Telecom web site for green data center services today.

Vern

Mucking out the data center.

I’ve just been reading this piece about data center contamination. There’s certainly a few good points in there but I think it misses the point by a mile on a lot of it.

There are certainly a number of contaminants that can play havoc with a data center. Copier and laser printer toner is definitely a bad one, very abrasive. Airborne sand and volcanic ash are also terribly abrasive as well. It’s also true that gaseous contaminates can cause oxidation, but I’m not sure what the average data center could reasonably do to counteract that.

At this point, I diverge. First of all, cleaning a data center to “clean room” standards is ridiculous. Most data centers are NOT built as clean rooms and the first person to open a door or the first puff of cooling air under a raised floor used as a plenum is going to blow that in a big hurry, not to mention that the article recommends doing this BEFORE raised floors, servers, and cable racks are installed. What about the dirt involved in this?

Second, what data center equipment really needs or would even measurably benefit from this level of clean? Data center equipment will usually go FAR beyond its useful lifespan with a lot less of a cleaning level than this. Cleaning the data center as if the same IT equipment was going to be run for 20 years is an awful waste of money.

Sticking with hard surfaces and avoiding fabrics in the data center is certainly a good idea, as are mats to trap as much incoming dirt as possible. Proper filters for air handling equipment is also a smart idea.

Zinc whiskers are probably the only contaminant that stand a significant chance in tiny quantities of causing an IT equipment failure in the data center. Zinc whiskers are formed on the surface of zinc plated metal, such as raised floor tiles and frames. The little whiskers break off on air flow (such as using a raised floor for a cold air plenum) and the head right for vulnerable parts of IT equipment, such as power supplies.

There are a bunch of good reasons NOT to used raised floor in the modern data center (added floor weight for no benefit, expense, under floor fire suppression, etc) but zinc whiskers and the inability to decently clean under them are the biggest. Dump the raised floor and you dump nearly everything that could really cause your equipment problems, as well as a major source of dirt.

Cleanliness in the data center is certainly an important issue, but keeping known offenders like raised flooring out and doing smart, cost effective cleaning instead of trying to achieve clean room perfection will be a lot more sensible.

Email or call me or visit the SwiftWater Telecom web site for green data center services today.

Vern

The top data center operations mistake.

I was just reading about the top 10 data center operations mistakes and I noticed they completely missed by far the most important one of all.

It’s certainly true that lack of proper procedures, training to properly follow the procedures, and training to make sound decisions when the procedure isn’t up to the situation are all important to smooth data center operations. The military wisdom that no plan of battle ever survives contact with the enemy frequently comes into play. The most beautifully crafted and approved procedures don’t mean a thing when it comes to an unanticipated situation and nobody involved can make a smart decision on their own.

The biggest mistake by far, in my opinion, and the one we’ve seen the most examples of in the last several years of data center failures, is failure to analyze risks sensibly. A large percentage of these outages have been the direct result of high risk, zero gain actions.

One good example of this is the irresistible urge to mess around inside of live electrical equipment. There is little to nothing you can do in a data center that is more dangerous and higher risk than working on exposed high voltage power, both to the health and safety of the person doing the working and the operation of the data center itself. The result of screwing this up can be spectacularly catastrophic (witness the transformer explosion at The Planet’s data center in 2009 due to a miswired added high voltage circuit).

Given the hazard, you would think that data centers would reserve that for only the most extreme need, however, many of those failures have been for the most trivial of purposes. One of the bad ones from this year involved a “maintenance” operation to manually check phase rotation in a panel (the correct A-B-C ordering of 3 phase power). Since this phase rotation is pertinent only to 3 phase electric motors, this is ZERO issue with data center IT equipment and far from justifiable, given the risk.

It comes down to a healthy dose of common sense. If you’re going to take all of your data center generator capacity offline to do something trivial and you only have 15 second run time flywheel UPS systems, that’s probably a BAD choice. If you know you restore generator capacity in far less time than the run time of your UPS, that makes a lot more sense.

Don’t do risky things for no gain and you’ll avoid the nightmares of the most preventable data center operations mistakes. It’s just that easy.

Email or call me or visit the SwiftWater Telecom web site for green data center services today.

Vern

Annoucement: our #datacenter is growing!

I’m pleased to announce a new 1000 sq ft expansion to our data center facilities at 22 Pearl St, Biddeford Maine! This new data center space will support our cloud computing services, as well as customer co-location services with energy saving green features.

Our 22 Pearl St locations include:

Full loading dock and freight elevator

3 phase power

Green 48VDC and 130VDC power

Free air cooling

600 lb/sq ft floor load capacity

Extensive data center security

Extensive building security

Close proximity to train and airport

Extremely reliable power, close proximity to 2 major generating stations

Only 1 1/2 hours from Boston

Partial and full cabinets to 42U available or co-locate your own.

Email or call me at 207-399-7108 for more information or visit our site at http://www.swiftwatertel.com to order!

Vern

#cloudcomputing gone ridiculous. #datacenter

I just got through reading a claim that cloud computing providers are hypocritical. This has to be pretty close to the silliest thing I’ve ever read.

The point being made here, as far as I can tell, is that, since cloud computing resembles a public utility, and the big cloud providers ( as well as every other data center) have generators because they don’t trust a mature service like commercial power, this proves they’re hypocritical when they ask businesses to trust them because cloud computing is a lot less mature. Where the heck do I start with this one.

First up, you’d have to consider every decent data center out there hypocritical, since no one in their right mind who farms computers “trusts” the commercial power grid, no matter how mature it is. Would you bet your business computers on the commercial power being perfect? Untrimmed trees, maintenance sloppiness, transformer failures, I don’t care how reliable it’s been, it’s eventually going to let you down.

So the claim of hypocrisy comes because cloud computing providers ask businesses to trust their data to the Internet and the Internet fails sometimes. So does the commercial power grid. So do businesses own in house computers. By that measure, every computer manufacturer is hypocritical for asking their customers to put their trust in computers that can and do fail.

Oh, wait a minute, I’m supposed to be comparing this to public utilities. Ever notice that every telephone office (a public utility) has a backup generator too? By that measure, they’re hypocritical too!

Not that I’m defending the outages that major cloud providers have had (mostly attributable to human stupidity) but this is just silly. Anything based on technology can fail at any time. Power grids, telephones, cloud computing, even your precious in house servers you love to hug. Some places in the country have well documented and incredibly bad problems with commercial power reliability.

The smart way to handle things that are fallible is not to put your eggs all in one basket (backup generators, redundant service with 2 or more cloud providers). Of course, you could always go back to paper and oil lamps to make sure nothing was ever going to fail on you.

Email or call me or visit the SwiftWater Telecom web site for cloud computing services and see how reliable they can really be.

Vern