12

Here we have some servers and almost each of them has a dedicated UPS. There are dependencies between them so they must be switched on in the correct sequence. Ultimately we are experiencing serious problems with the power supply, so the servers are shutdown and then restarted in a random order when power is restored. It is not a problem if the servers were switched off during a blackout, it is important they work correctly without any human intervention once power is restored.

Our UPS are quite cheap and the only configuration parameter useful for my goal is power the load xx seconds after power is restored. In theory putting the right delays on each UPS I can fix the order of server restart but I don't trust the UPS will behave as expected.

Is it the right way to go ?
Do high level UPS give other options to fix the restart sequence ?
One final note: my Ups are in the range of 1000 - 2200 VA

Reaces
  • 5,597
  • 4
  • 38
  • 46
Filippo
  • 377
  • 5
  • 18
  • 1
    This is one of the nice things offered by `systemd` - the ability to define proper dependencies in the startup process. Wait until service X is available before trying to start service Y. – MSalters Jun 08 '16 at 09:55
  • 1
    @MSalters AFAIK systemd dependency management only works when the units are handled by the same `systemd` instance and not for services running on completely different servers... – HBruijn Jun 08 '16 at 10:11
  • 1
    @HBruijn: Sort of, network mounts for instance work across servers. That is to say, if server1 mounts a filesystem hosted by server2, then serve1's services which depend on the mount will pause until server2 has started those services. And IIRC you can also have servers wait for DHCP (don't ask me why a server uses DHCP, but it was mentioned in an answer) – MSalters Jun 08 '16 at 10:44

4 Answers4

25

The standard answer for this is "not at all". Fix the software to handle restarts in random order. If you really need SOME servers to start first (example: Active Directory) put them on USV's that are possibly surviving a LOT longer. A low power atom based server is good enough as Active Directory controller and will survive a day on a small USV.

Do high level UPS give other options to fix the restart sequence ?

No. I would say it is generally assumed programmers are competent enough to work around the issue properly.

What you COULD do is:

  • Have servers start "randomly". Except for DHCP / Active Directory there is nothing really demanding an order that can not be fixed.
  • Have a control server after some time (5 minutes) start the services on the various machines in the correct order.

I would say that this type of setup is a lot more common. I would call any software that REQUIRES server starts in a particular order (outside of pure infrastructure) as broken and unfit for business.

Just as note: our own setup is a low cost 20kva USV (low cost because we got one used) for the servers, with a slaved 2000VA USV for a machine serving as "root" of the network (and backup machine). Slaved means that the USV is behind the big one - so it only switches to battery when the large one (that lasts between half an hour and 8 hours depending on how much of our computing grid is online) is going into terminal shutdown.

Reaces
  • 5,597
  • 4
  • 38
  • 46
TomTom
  • 51,649
  • 7
  • 54
  • 136
  • 2
    I think this is sometimes easier said than done (AD, as you say, is an obvious example) but I do agree. The correct solution is to work on eliminating dependencies for things like start order of servers or services. If nothing else, it should be possible on a web app, for example, to write code that says "If I can't connect to my back-end, 'sleep' and try again later rather than crash horribly". – Rob Moir Jun 08 '16 at 08:26
  • he problem with AD is not even AD - it is mostly IPv4 DHCP which is not prepared for computers being online before the dhcp server. Ipv6 handles this ;) – TomTom Jun 08 '16 at 09:17
  • That's true. IPv4 is a pain... and I still get people here asking me why we need to bother with "this newfangled IPv6 rubbish". – Rob Moir Jun 08 '16 at 09:53
  • But also that can be avoided - hardcode IP and DNS info into the servers. Then AD can start after the server and no problem either. – TomTom Jun 08 '16 at 10:10
  • 1
    "It is generally assumed programmers are competent enough to work around the issue" - you must not do much programming! No, in all seriousness, there's a huge amount of reasons a system may need to be brought up in a specific order. Yes, software should "fail gracefully" and re-try connections that are broken, but it's not always possible. From what I recall, some of the nice PDU's have the ability to start/stop individual ports, so perhaps something could be done there. – SnakeDoc Jun 08 '16 at 15:14
  • 1
    I had to search for "USV" and found "Unmanned surface vehicle". I know this is wrong, but I want it to be right. – Braiam Jun 08 '16 at 21:02
  • As a programmer, this is the right answer. There is almost nothing that you can't just write code around to fix if your really need a specific start order. The best way is to reduce or totally remove the dependency on start order. There is no reason that "the code" can't handle the "wait for dependent service" in one way or another. Even if you have to cheese it and use sleep(60000) or method_call_that_waits_for_port_to_open(80) (there are usually much better ways) it's still better then assuming that a server start order will be followed when no one is around. – coteyr Jun 09 '16 at 09:50
  • @Braiam Yeah, sorry- that is actualyl the german abbreviation.... – TomTom Jun 09 '16 at 09:55
  • Exactly. SnakeDoc obviously is either no or a realyl bad programmer. There ARE very few tings you can not do as programmer, which MOSTLY go down to really bad specs (IPv4 DHCP being a prime example - the server MUST run when a client interface becomes active as it only queries for the IP address once) but even there there is a workaround (client reusing last address if no DHCP server is found). – TomTom Jun 09 '16 at 09:56
  • I can't imagine a case where exact startup order should reasonably be required. Network resources can come and go, and worst case your dependent services should crash and be restarted after some time by a watchdog. In the case of DHCP, for example, if the lease is still valid the client should (and usually will) continue to use its old address. In any event, the client will continue to attempt to get an address until it does. – codatory Jun 10 '16 at 00:34
  • But to answer the actual question, the Eaton 5P and 5PX units we typically install have outlet groups that can be configured with a startup delay which we very occasionally use to stagger startups when say a VM Host requires access to a SAN and network gear to function. Also, pretty much any managed PDU will have sequencing functionality (like the TrippLite units we install). – codatory Jun 10 '16 at 00:36
  • Which is not an answer as the question states that each server has his own power supply. – TomTom Jun 10 '16 at 05:55
14

Managed Power Distributions Units (rather than the UPS) often do support customised delays in enabling individual outlets after power is resumed.

Typically that is to prevent circuit breakers from tripping when a cabinet full of systems powers up at the same time immediately after power is restored, but that can also be used to preserve the boot order of your system dependancies.

HBruijn
  • 77,029
  • 24
  • 135
  • 201
  • Yeah, correct. This is advanced functionality and it is not assumed the USV is actually connected to the servers - but it is powering racks that then use PDU's to handle the details. – TomTom Jun 08 '16 at 09:19
6

I had this exact issue. The only difference being we invested in sturdy rack mounted APC power units (for example APC SmartUPS 3000 ). With the APC PowerChute network shutdown software (PowerChute Network Shutdown software) , I'm able to shut down and bring up servers in a specific order. Another handy feature of the software was setting the servers to shut down at the very last minute, i.e. calculating how much battery power the APC units had left and shutting down the servers with just enough time for them properly shut down instead of just powering off.

The software is...not user friendly but it's nothing difficult if you take some time to figure it out. If you're interested in investing more in your infrastructure, this is definitely the route to go.

Winski Tech
  • 169
  • 3
  • 13
  • 1
    We have Apc Smart Ups too, some of them are relatively old and perhaps have weak batteries. It is difficult to make some tests on them because they are in production. Besides we lack any freak load, I mean a load which can bear a sudden power down without problems. That said every time I simulated a blackout the UPS behaved differently that expected, it could be due to a misconfiguration but my feeling is that those UPS are not very reliable. – Filippo Jun 09 '16 at 08:51
  • @Filippo certainly YMMV but I have a mix of SmartUPS 3000 and 3000XLMs across multiple sites using the PowerChute software for going on 3 years and after figuring out the software, there is certainly a learning curve and some testing is needed, it's been pretty solid. – Winski Tech Jun 09 '16 at 17:53
2

It sounds like the UPS units are low-cost and not capable of being configured for a specific output-on wait time after power is restored (some higher end units are). To get the same functionality, you need to pick a specific host to always power on right away (maybe whichever system is allowed to boot at any time) and leave all the other servers in a powered off state (configured in bios to return to power off when AC is applied, and to honor the Wake On Lan magic packet to power on when told to do so). Then, on the main host that does boot, run a script/utility to time the transmission of the WOL magic packet to each host.

Jeff Meden
  • 698
  • 6
  • 5