17

I've always assumed that using Y cables from the PDU to redundant PSUs in the same server was totally acceptable practice, however I was recently speaking to someone that said that their colo wouldn't allow it.

I'm no electrician, but it would seem to me that since the PSUs are redundant, they aren't both drawing at the same time. If this is the case, it would seem that using a Y cable on two PSUs doesn't risk overdrawing the rated ampacity of the cable or individual outlet.

Have I missed something here? Why would a colo choose to ban Y-power cables in server racks?

MDMarra
  • 100,734
  • 32
  • 197
  • 329
  • 1
    Can you link to an example cable? – ewwhite Jan 07 '13 at 15:22
  • Something [like this, maybe](http://i.imgur.com/4QxuW.jpg) – MDMarra Jan 07 '13 at 15:26
  • 9
    Hm, seems like we really need a start-to-finish "Power for Sysadmins" primer here. Some of these answers have opened a whole new can of worms – MDMarra Jan 07 '13 at 15:53
  • 3
    I'd strongly suspect they had some idiot in the past take a single C13 port from the PDU and "distribute" it to 5 servers or something that blew out the PDU... And now they have a rule against Wye splitters. – Chris S Jan 07 '13 at 16:01
  • 1
    This Q should be added to the "Canonical" list. YMMV... – Deer Hunter Jan 07 '13 at 16:15
  • The colo might be worried about electrical shorts tripping circuit breakers to BOTH sides of a redundant PSU arrangement at once. – Warren P Jan 07 '13 at 16:23
  • 4
    @DeerHunter I don't think this question should be canoncial in and of itself, but a "Power for Sysadmins" primer sure seems like a good idea for a new canonical. – HopelessN00b Jan 07 '13 at 16:36
  • @HopelessN00b I completely agree. See also my comment below. – Tonny Jan 07 '13 at 18:59

8 Answers8

19

I am no electrician either, but I think you will at least lose the possibility of keeping your server up and running when doing so. On the contrary if you connect each PSU to a different power source, your server will still have an availble power source (hopefully).

Khaled
  • 36,533
  • 8
  • 72
  • 99
  • 12
    This. Y-cables don't pose an electrical hazard, they pose a redundancy hazard since. – Lee Harrison Jan 07 '13 at 15:27
  • So you're talking about having multiple PDUs in a rack? That's not really feasible in a lot of cases, especially if you're using vertical PDUs. – MDMarra Jan 07 '13 at 15:27
  • Depends on the rack I suppose. We are a smallish company with 2 tall server cabinets and both accomodate redundant power supplies by having independant power rails on each side of the cabinet. Of course your server room will need to be wired properly in order to take advantage of this as well. – Lee Harrison Jan 07 '13 at 15:30
  • @MDMarra: Maybe, you will have a risk if the PSUs are working in load balancing mode and a failure occurs. In this case, the power will increase on the other still active PSU. Look at [this post](http://serverfault.com/questions/260335/r610-redundant-power-supply). – Khaled Jan 07 '13 at 15:33
  • @Khaled but that still won't result in unsafe power consumption as far as I am aware. – MDMarra Jan 07 '13 at 15:37
  • mdmarra i went to a datacenter in london which had pairs of longish (12 port i think) pdu's, one up each side of the rack zip tied to the poles. Without some major fiddling you lost like a quarter of the usable rack space. fun times. – Sirex Jan 07 '13 at 20:33
  • @MDMarra there are PDUs with more than one circuit, too. 2-phase or 3-phase PDUs are not that uncommon, even [Dell sells them](http://accessories.euro.dell.com/sna/productdetail.aspx?c=ie&l=en&s=dhs&cs=iedhs1&sku=450-15691). Non-vertical rackmount variants are also available . – the-wabbit Apr 15 '13 at 09:34
  • Yeah, but if you lose that PDU, you lose all three phases in that rack – MDMarra Apr 15 '13 at 10:53
16

I've never used them because they're a single point of failure, at best.

Every server I deploy into a real datacenter has each PSU plugged into a different PDU in the rack, each of which are attached to a different independent UPS, on different circuits, ideally even fed from different power feeds.

If the UPS, PDU or circuit your Y-cable's attached to goes down, the redundant PSUs are going to be useless, so it seems like a waste at best, and a false sense of redundancy at worst.

EDIT:

I'll just mention that I'm talking about lower capacity 1U or 2U UPSes mounted inside the server rack, rather than the much larger, much more expensive UPS units that take a up rack all to themselves. Those are definitely built to be highly redundant in and of themselves, without the need for a secondary unit.

HopelessN00b
  • 53,795
  • 33
  • 135
  • 209
  • Most of the environments that I have worked in have a Symmetra or similar UPS that consumes multiple racks. I don't really see too many places having redundant UPSes of this scale, but I also don't have a lot of exposure to the "power" side of things - thus this question. I also really have never seen a failed PDU (not that it doesn't happen, I'm sure). When you hit that scale, do people really have multiple Symmetras to feed multiple PDUs? – MDMarra Jan 07 '13 at 16:06
  • @MDMarra Yeah, see my edit, I got tunnel vision - I've yet to see anyone set up a large UPS system like the type you mention with a second unit for redundancy. If/when additional units of that size are added, it's about extra capacity, not extra redundancy. – HopelessN00b Jan 07 '13 at 16:12
  • 4
    @MDMarra I do a lot of datacenter design and building. It rarely is multiple UPS systems for redundancy. The big UPS systems themselves can operate with various levels of failures (batteries, power-controllers) and will fall back gracefully to just rerouting mains power if needed. Mains power will be taken over by generator if needed. So normally the UPS is just a rectifier/surge-protector and covers the take-over from mains to generator. The UPS will have multiple output circuits which feed the PDU's in the rack(s). Each server connects to 2 different curcuits. – Tonny Jan 07 '13 at 16:37
  • @Tonny Hmmm, very good info. I wish I'd phrased the question in a way that this could be an answer and not a comment so you could get some rep out of it :) – MDMarra Jan 07 '13 at 16:41
  • @MDMarra :-) I'm doing OK withotu the points :-) But a canonical question/answer on best practices for serverroom power seems to be in order. (And let's not forget earthing issues on that one.) – Tonny Jan 07 '13 at 18:59
8

The primary reason such items are frowned upon is due to redundancy, lack-thereof. Using such a cable means all of your server's power-inputs are being fed by the same circuit so when that circuit dies (or the PDU it's connected to, I've had that happen) so does the server. Colos strongly recommend Primary and Secondary circuits for just this reason and want to see multi-PSU servers plugged into two circuits.

Way back in the day I had a group of machines that shipped with a single 3-way Y cable and 3 normal power cables for a large (7U if I remember right) 3 PSU system. The data-center I was working in at the time (this was about 1999) didn't have enough power-outlets for that kind of thing, so we ended up using the Y cable; 2 legs of the Y on one UPS, and a straight up power-cable for the 3rd PSU to the second UPS. 3-PSU systems are thankfully much less common now.

PSU Load-balancing, or is it switching?

There are differences to how power-supplies handle loading. As various power-supply benchmarks have shown, peak efficiencies are reached once you get over 50% loading. There are gains to be had for running things all on one PSU, it's more likely to be efficient. It is for this reason that some server manufacturers draw all of a server's current through a single PSU and switch to the other one when a failure happens or a whim strikes; a 230 Watt system will get best efficiency from its dual 400 Watt PSUs by running all the load through only one PSU.

Such switching systems only draw from one PSU, and therefore one circuit if using fully separated power circuits, at a time.

The downside to switching systems is that load can move unpredictably in the community of PSUs connected to a certain circuit. If enough of them throw their weight to a single circuit it can overload it. This is bad power-design, since you want to design things so that you can lose a full circuit and have things stay up, but it's still something that trips up systems engineers.

Load-balancing servers draw equal amounts of current from both PSUs. This gives predictability in circuit loading, though can still cause blown breakers if the Systems Engineers load their circuits over 50% and a circuit dies forcing the PSUs to draw 100% from one circuit, which now exceeds its rating. Again, bad power design but it's a common mistake.

Startup loads

There are two kinds of startup-loading:

  1. Everything runs flat out until BIOS (or OS Boot, or app-load) catches up and things calm down.
  2. Inrush current loading right as things get turned on.

The first is something we're all familiar with. That 120 disk SAS array may draw only 4000 Watts when running normally, but if all the disk-shelves restart at the same time it may draw 6500 Watts.

The same holds true for servers. Fans run at full speed, yes. CPUs run at full speed for a bit, yes. RAM is run at full voltage during post, yes. It's likely to draw as much as it can draw during those first stages of POST but rapidly drops off as the BIOS hands things off to the OS and power regimes take over. A server that normally draws 110 Watts during normal usage may temporarily draw as much as 200W for those first few phases.

It's this temp loading that most people think of when they say things like, "it runs the power-supplies at full on startup". Those 400 Watt power-supplies plugged into a server that draws 230W on a busy day aren't going to draw 400 Watts, they'll draw 230W... combined.

The second isn't well known, but when people run across it they get worried. This is inrush current, and takes a few milliseconds during which draw can be quite a lot higher than it normally is. The inrush current for IT devices with AC to DC converters in them (which is all of them) almost always happens twice:

  • One time when the cable is plugged in, as the pre-power stage gets power. It's this stage that allows the power-button on the front to power on the device.
  • A second time when the main distribution stage powers up and starts the device.

Because of the timings, this only becomes a factor when restoring power to a dead circuit. All those devices powering on at exactly the same time can do weird things to the power on that circuit, and that can cause damage all by itself. Doing a staged startup alleviates this.

This is the other area people think of when they say things like "power-supplies run full-tilt on startup", since each PSU has its own inrush current. But as I said, this lasts for a few milliseconds and comes in two stages.

sysadmin1138
  • 133,124
  • 18
  • 176
  • 300
  • Are server PSUs optimized for a different point on the load curve than consumer ones? For the latter peak efficiency is normally at ~50% and falls off on either side of that point. Ex test #3 here is the 50% point. http://www.jonnyguru.com/modules.php?name=NDReviews&op=Story3&reid=327 Similar results were shown for both of the other reviews at the top of the main page; and on most if not all of the reviews I've seen of quality PSUs. – Dan Is Fiddling By Firelight Jan 08 '13 at 19:46
  • @DanNeely You're correct, I was remembering the curves wrong. Will update. – sysadmin1138 Jan 08 '13 at 20:15
4

Our local Colo did not like them either. We had a shared cabinet and were only allowed to utilize a single PDU port for our dual PSU server. The colo didn't like it for redundancy reasons, but for a non-critical machine it was perfectly acceptable to us. There aren't any major power issues from an electrical perspective.

This is why our colo didn't like it:

  1. If the PDU dies the server is dead
  2. If the cable gets fried your server is dead

Here is why I like it:

  1. Non-mission critical box
  2. Supplied power to both PSUs giving me redundancy on the box
  3. I wasn't all that concerned with potential down-time based on the failure of the PDU, nor was I all that concerned about the cable going batshit crazy. (see #1).

The colo didn't have any issues with them electrically speaking. The draw was identical, the box only used what it needed, regardless of the number of PSUs consuming power. However, shortly after buying this cable with my Dell box Dell did stop publicly offering the cables for sale.

Brent Pabst
  • 6,069
  • 2
  • 24
  • 36
  • 4
    `If the PSU dies the server is dead` - Did you mean PDU there? – MDMarra Jan 07 '13 at 15:41
  • IMO, the colo/datacenter shouldn't mandate what is in essence an operational issue. If I choose to use a Y-cable then that's my choice, regardless of the redundancy or lack thereof. – joeqwerty Jan 07 '13 at 15:52
  • @joeqwerty Fully agree, if the data center is held responsible for uptime to the box however, I would fully accept their need to mandate the use of 2 straight cables instead. That wasn't the case for me, and in a shared cabinet scenario the Y-cable was a much better choice. – Brent Pabst Jan 07 '13 at 16:13
  • I think the "PSU" in point 3 should also be "PDU", resulting in the points meaning: you consider the risk of cable or PDU failure to be much smaller than the risk of PSU failure which sounds about right to me. I've seen many more PSU failures than cable failures. I've not done much rack-based work, so I can't really comment on PDU failure rates. – Kevin Cathcart Jan 07 '13 at 22:21
  • @joeqwerty To me, and from my experience, it sounds like sometime in the past someone had a Y cable hooked up to a dual-PSU server (maybe even the datacenter tech did it), and then a circuit died, and a pointy-haired-boss let loose on the DC that the "redundant" power they paid for (in both the server and dual circuits in their rack) didn't work, they were in violation of their SLA, etc etc. Easier for the DC to just ban the cables than to worry about whether people understood the outcome of how they were being used, especially if it was done by their own in-house techs. – gregmac Jan 08 '13 at 00:02
3

When a server boots up, both PSU start up at full power to do a system check. This draws double the normal operating power. Now that power is more of an issue, the circuits coming into your cabinet are very specific. They don't want you to trip a breaker should you have to reboot your server.

Also, some office building don't let you do that because of fire code. You are doubling up the power that could be consumed from one outlet.

NeerPatel
  • 329
  • 2
  • 6
  • 18
  • 1
    Do you have any documentation that shows that powering on a server with two 750W PSUs will cause 1500W of draw? – MDMarra Jan 07 '13 at 15:52
  • 2
    Your effect is right (or close to it) but I think your cause is wrong: Servers draw more power at startup because they have "startup stuff" to do: Spinning up disks, power on fan check, charging up capacitors, etc. - I don't know any PSU that does alias test. Doing so would require a load to test with (big honkin resistor) & switching hardware which I've never seen in a PSU. If you have seen such a beast I'd like to hear about it. – voretaq7 Jan 07 '13 at 16:05
  • 1
    @voretaq7 But even so, this additional load should fall well below the rating of a single PSU in the system. Sure there's increased consumption, but it's not an unsafe increase. – MDMarra Jan 07 '13 at 16:14
  • @MDMarra That depends on how many disks you have to spin and how intelligent the controller is -- I saw a box that produced a 2-3 Amp spike (on 208V power - so ~450-500W plus the normal 150ish it ran at) because its disk spin instructions were "all 16 disks ***RIGHT FREAKIN NOW***. (That configuration didn't last long.) Generally though you're right - we're not talking "Full rated draw on both power supplies" (at least I've never seen it be THAT bad) – voretaq7 Jan 07 '13 at 16:43
  • @MDMarra I know from personal experience that most HP Proliant servers will briefly (3 to 8 seconds) draw full load on both PSU's on startup. Some models Dell Poweredge do it too. – Tonny Jan 07 '13 at 16:44
  • @Tonny I know some Dells *switch* their load to each PSU in the chassis briefly (running on PSU #1, then PSU #2, then both), but I've never seen the max draw for the system exceed the rated wattage of one power supply - have you actually seen a case where the system draws the full rated wattage on *both* power supplies at once? AFAIK that breaks the idea of redundancy - no single PSU could supply that load if the other failed. – voretaq7 Jan 07 '13 at 16:49
  • @voretaq7 I'm not 100% sure on the Dell's. It could be what you describe and I didn't look carefully enough. I am 100% sure about HP Proliant servers. But then: HP couldn't get power right if their lives depended on it. I have never seen any HP equipment with a good PSU. Servers, switches, laptops, desktops, they just don't seem to be able to produce 1 model with a good PSU. Whatever model: At least 10% of equipment seems to have some sort of power issue. – Tonny Jan 07 '13 at 18:22
  • Its not only about disks. The PSU needs to do self-test, fans go 100% RPM. Its called [POST](http://en.wikipedia.org/wiki/Power-on_self-test) – Ahmed Al Hafoudh Jan 07 '13 at 23:00
3

A colo which is offering two redundant power circuits into your cabinet may be interested in having the two circuits balanced.

This may be why they don't like you connecting servers with dual PSUs to a single circuit, regardless if this is done with a Y-cable or with two cables plugged into the same PDU.

Oliver
  • 5,973
  • 24
  • 33
  • 4
    Maybe, but then again if I'm paying for the circuits then it should be my choice how I choose to use them, load balanced or not. As long as I don't exceed the allowed capacity of either circuit the colo/datacenter shouldn't have anything to say about how I use them. – joeqwerty Jan 07 '13 at 16:09
  • If they feel the need to keep those two circuits balanced, I'd be worried that they are not as redundant as they were supposed to. After all, at any given point if one goes down, the other is supposed to be ready to take the full load instantly. – kasperd Mar 20 '15 at 07:41
2

Being in the colo and hosting business for quite a while, I can give you the three very best reasons why a hosting company would not allow it.

Money, money and money

Think about how much a single 15 or 20 Amp circuit cost you, the hosting companies would much rather you purchase a second one for failover and charge you the big bucks twice a month instead.

Alex
  • 3,129
  • 21
  • 28
  • Well, I'm not sure that covers it, honestly - you can plug two cables into the same circuit just as well as you can plug in one Y-cable. – HopelessN00b Jan 07 '13 at 16:13
  • Sure you can, but you will eventually run out of space for your plugs. That means a second PDU. Believe me, that business is all about two things 1) Space 2) Power. Whatever will make you consume more of one or the other they will try it. – Alex Jan 07 '13 at 16:17
  • Alright, but running your PDU out of plugs doesn't necessitate the lease of a second circuit, so I'm not sure how this is related to profit-taking by the colo/host. – HopelessN00b Jan 07 '13 at 16:23
  • But it does make you use 1 more U if not 2. For customers renting a 1/4 or even 1/2 cabinet, that could be enough to have them upgraded to 1/2 or a full cabinet. – Alex Jan 07 '13 at 16:26
-2

I'm really surprised nobody on here did not realize you can have two 20amp PDUs running to different power then take a y cable plug into two different servers and plug into the first PDU then take another y cable plug into the second power supply of the two servers and run it to the second PDU. You now have redundancy and less power cord mess.

d3vmono
  • 1
  • 1