Probability of Blade Chassis Failure

Question

In my organisation we are thinking about buying blade servers - instead of rack servers. Of course technology vendors also make them sound very nice. A concern, that I read very often in different forums, is, that there is a theoretical possibility of the server chassis going down - which would in consequence take all the blades down. That is due to shared infrastructure.

My reaction on this probability would be to have redundancy and by two chassis instead of one (very costly of course).

Some people (including e.g. HP Vendors) try to convince us, that the chassis is very very unlikely to fail, due to many redundancies (redundant power supply, etc.).

Another concern on my side is, that if something goes down, spare parts might be required - which is difficult in our location (Ethiopia).

So I would ask to experienced administrators, that have managed blade server: What is your experience? Do they go down as a whole - and what is the sensible shared infrastructure, that might fail?

That question could be extended to shared storage. Again I would say, that we need two storage units instead of only one - and again the vendors say, that this things are so rock solid, that no failure is expected.

Well - I can hardly believe, that such a critical infrastructure can be very reliable without redundancy - but maybe you can tell me, whether you have successfull blade-based projects, that work without redundancy in its core parts (chassis, storage...)

At the moment, we look at HP - as IBM looks much too expensive.

Answer 1

There's a low probability of complete chassis failure...

You'll likely encounter issues in your facility before sustaining a full failure of a blade enclosure.

My experience is primarily with HP C7000 and HP C3000 blade enclosures. I've also managed Dell and Supermicro blade solutions. Vendor matters a bit. But in summary, the HP gear has been stellar, Dell has been fine, and Supermicro was lacking in quality, resiliency and was just poorly-designed. I've never experienced failures on the HP and Dell side. The Supermicro did have serious outages, forcing us to abandon the platform. On the HP's and Dells, I've never encountered a full chassis failure.

• I've had thermal events. The air-conditioning failed at a co-location facility sending temperatures to 115°F/46°C for 10 hours.
• Power surges and line failures: Losing one side of an A/B feed. Individual power supply failures. There are usually six power supplies in my blade setups, so there's ample warning and redundancy.
• Individual blade server failures. One server's issues do not affect the others in the enclosure.
• An in-chassis fire...

I've seen a variety of environments and have had the benefit of installing in ideal data center conditions, as well as some rougher locations. On the HP C7000 and C3000 side, the main thing to consider is that the chassis is entirely modular. The components are designed minimize the impact of a component failure affecting the entire unit.

Think of it like this... The main C7000 chassis is comprised of front, (passive) midplane and backplane assemblies. The structural enclosure simply holds the front and rear components together and supports the systems' weight. Nearly every part can be replaced... believe me, I've disassembled many. The main redundancies are in fan/cooling, power and networking an management. The management processors (HP's Onboard Administrator) can be paired for redundancy, however the servers can run without them.

Fully-populated enclosure - front view. The six power supplies at the bottom run the full depth of the chassis and connect to a modular power backplane assembly at the rear of the enclosure. Power supply modes are configurable: e.g. 3+3 or n+1. So the enclosure definitely has power redundancy.

Fully-populated enclosure - rear view. The Virtual Connect networking modules in the rear have an internal cross-connect, so I can lose one side or the other and still maintain network connectivity to the servers. There are six hot-swappable power supplies and ten hot-swappable fans.

Empty enclosure - front view. Note that there's really nothing to this part of the enclosure. All connections are passed-through to the modular midplane.

Midplane assembly removed. Note the six power feeds for the midplane assembly at the bottom.

Midplane assembly. This is where the magic happens. Note the 16 separate downplane connections: one for each of the blade servers. I've had individual server sockets/bays fail without killing the entire enclosure or affecting the other servers.

Power supply backplane(s). 3ø unit below standard single-phase module. I changed power distribution at my data center and simply swapped the power supply backplane to deal with the new method of power delivery

Chassis connector damage. This particular enclosure was dropped during assembly, breaking the pins off of a ribbon connector. This went unnoticed for days, resulting in the running blade chassis catching FIRE...

Here are the charred remains of the midplane ribbon cable. This controlled some of the chassis temperature and environment monitoring. The blade servers within continued to run without incident. The affected parts were replaced at my leisure during scheduled downtime, and all was well.

Answer 2

I've been managing small numbers of blade servers for eight years now, and I've yet to have a system-wide failure that took a number of blades offline. I've come real close due to power-related problems, but haven't yet had a chassis-wide failure that wasn't attributable to outside sources.

Your observation that the chassis does represent a single-point-of-failure is correct, though they do build in a large amount of redundancies in them these days. All of the blade systems I've used have had parallel power feeds to the blades, and multiple network jacks going through separate paths, and in the case of Fibre-channel multiple paths from the blade to the back-of-rack optical ports. Even the chassis information system had multiple paths.

With appropriate network engineering (redundant NIC usage, MPIO for storage) single-problem events are entirely survivable. In my time with these systems I've had the following problems, none of which affected more than one blade if any:

• Two power-supplies fail in the blade-rack. There was enough redundancy in the other 4 to support the load.
• Loosing a phase for a 3-phase power-supply. These supplies are rare these days, but the other two phases had enough capacity to support the load.
• Losing an inter-chassis management loop. It was like this for years before a vendor tech on another call noticed it.
• Losing the inter-chassis management loops entirely. We lost management-console access, but the servers kept running as if nothing was wrong.
• Someone accidentally rebooted the back-of-rack network backplane. Everything in that chassis was using redundant NICs so experienced no interruption of service; all the traffic moved to the other backplane.

TomTom's point about cost is very true though. To get to full cost-parity your blade chassis will have to be fully loaded and likely not using special things like back-of-rack switches. Blade-racks make sense in areas where you really need the density because you're space-constrained

Answer 3

That question could be extended to shared storage. Again I would say, that we need two storage units instead of only one - and again the vendors say, that this things are so rock solid, that no failure is expected.

Actually no. You concerns so far made sense, this sentence puts them into "read the stuff in front of your eyes". HA with full replication is a known enterprise feature for storage units. Point is that a SAN (Storage unit is a lot more complex than a blade chassis that at the end is just "stupid metal". Everything in a blade chassis except some backplanes is replaceable - all modules etc. are replaceable, and individual blades ARE allowed to fail. Noone says blade center in itself gives the blades high availability.

This is a lot different from a SAN which is supposed to be up 100% of the time - in consistent state - so there you have stuff like replication etc.

THAT SAID: watch your numbers. I have considered buying blades for some time now and they NEVER MADE FINANCIAL SENSE. The chassis are just too expensive and the blades not really cheaper compared to normal computers. I would suggest looking at the SuperMicro Twin architecture as alternative.

Answer 4

The blade servers I've had experience with are those from IBM. Those particular ones are totally modular and there is much redundancy built in. So, if something is going to fail it's going to be one of the components like a PSU or a modular switch etc. But again, there is redundancy even in those.

Since being involved with the IBM blades I haven't seen a complete failure before.

With the other brands I suspect they would be constructed in a similar fashion.
It would be a good idea to speak to a vendor as well and do a lot of reading.
It is a big investment.

Answer 5

Failures leading to multiple blade server outages in the same enclosure are comparable (in likelihood and cause) to failures leading to multiple server outages in the same rack.

Initial set up to minimize single points of failure (two separate AC sources of power, each of which can handle the entire load, running to separate DC power supplies, such that either half can handle the entire load; two separate network attachments, either of which can handle the entire expected load, etc) and the difference between something that takes out all the blades in a chassis or all the 2U servers in a rack is very small.

Answer 6

A concern, that I read very often in different forums, is, that there is a theoretical possibility of the server chassis going down - which would in consequence take all the blades down. That is due to shared infrastructure.

Indeed! About 5 years ago, while managing two HP Proliant p-Class blade enclosures, I encountered chassis wide issues several times.

I've had blade servers not being able to turn on, if they had been turned off (servers are not turned off often, but became a very real problem to us just the same). I've had servers suddenly switching off and not being able to being switched on again. Finally, I had all the servers switching off and not being able to come on again.

As I recall it, pretty much all the issues were attributed to bad power backplanes or controller backplanes. We had those replaced a number of times and the non-specific, off-the-record message I got from the techs were, that they had their share of problems with this generation of blade enclosures.

I decided back then, that the benefit of blade servers were simply not worth the risk, if I had anything to say in future purchases.

Fast forward to my next employer, and my current one, for that matter. They already had HP Proliant c-Class enclosures running, so my luke warm feeling for blades didn't really matter. In the 5 years I've dealt with the c-Class enclosures, I've never experienced anything like I did with the p-Class, where an entire enclosure failed on me. They've been running without major problems.

(Except for the time a rain storm sent rain through the roof, 4 stories, a small hole in the seal of the computer room, down a cable and into the chassis)

Answer 7

Both DELL & HP Blade chassis lack a redundant mid-plane. This is where IBM Bladecenter proves to be a winner. To my knowledge its the only blade chassis that provides a redundant mid-plane. Though HP offers a fantastic suite of management software for the blades, we bought a Bladecenter E for our company just to avoid a single point of failure of the entire chassis.