Clustered machines management

Question

Our startup firm runs multiple services hosted on JBoss AS. Those services are distinguished by their business and are clustered for the sake of high availability and scalability.

What I'm looking for is something that can:

get as an input a set of Linux machines that will host to my services,
get as an input the number of instances for each service,
automatically deploy the services onto the Linux machines,
detect and respond to failures according to some predefined rules (restart, install the service on a backup machine etc.).

Any suggestions?

This would be better suited for Serverfault. Perhaps a mod can move it? — Dan McGrath, Jan 11 '12 at 16:29

score 2 · Accepted Answer · edited Nov 24 '22 at 22:42

2

Without more information on your requirements I would suggest looking into Puppet and Chef to handle items 1, 2 and 3.

Item 4 is a little harder -- You need a monitoring system (look around on here, or ask google. You'll get lots of suggestions), and if you want automatic responses your monitoring system will have to feed some other system to take action.
This needn't be too complicated - I've implemented something similar using InterMapper: When a failure is detected InterMapper runs a "command-line notifier" specifying the failed system. The command line notifier is just a shell script that takes the system name as an argument and restarts it, sending out an email if the restart fails..

edited Nov 24 '22 at 22:42

Faith Kostow

33
2

answered Jan 11 '12 at 22:22

voretaq7

79,879
17
130
214

My first instinct for #4 is to look through results from http://www.google.com/search?q=load+balancing+heartbeat – Jeff Ferland Jan 12 '12 at 02:18
@JeffFerland also something good to look at, but it doesn't help you if there's a misconfiguration that gets replicated in #3 -- You need some kind of monitoring/alerting so you find out before your users call :) – voretaq7 Jan 12 '12 at 02:24
Agreed... I guess I just take something like Nagios as a given. As pointed out by others, it can also work to drive events besides notifications, but I've always felt it best suits a monitoring role. – Jeff Ferland Jan 12 '12 at 02:26

score 1 · Answer 2 · answered Jan 11 '12 at 22:39

1

From 1 to 3 Agree with @voretaq7, Chef, Puppet or any automated configuration management software. Note tht all of these arent suitable for your needs, Configuration Management software is a too vague term sometimes.

For the last one, perhaps a good set of scripts can be used from Nagios event handlers which will be basically scripts that will try to repair things when a monitor goes warning or critical.

answered Jan 11 '12 at 22:39

theist

1,229
2
10
24

If I didn't hate Nagios with the fury of a nova I might have suggested it. I might not quite be over my PTSD from all the bad Nagios setups I've seen, but my therapist says I'm making progress! :-) – voretaq7 Jan 12 '12 at 02:26

score 1 · Answer 3 · answered Jan 12 '12 at 00:58

Since you are using JBoss, you should consider JON, Jboss Operations Network. You can combine it with RHN Satellite to have a full end to end deployment capability. Red Hat Cloudforms is a newer product of a similar offering. Cloud Forms is targeted more for managing internal cloud deployments.

At a minimum JON will likely handle many of the options you are looking for and many Open Source tools such as Puppet, Chef, PXE, and so forth can be used to handle some of the lower level needs.

Clustered machines management

3 Answers3