Restart all Jenkins Nodes

Question

I manage a Jenkins server (running 2.89.4) with about 40 nodes or so attached, each running either Centos7, OSX, or Windows (7, 8, 10). Recently, we suffered some power outages and many slaves lost their connection to the master. This required a lot of manual remoting and ssh'ing to reconnect them all. Since, I have made sure each slave process is running a server, however I'd like an additional safety net of being able to automatically restart every vm and/or box these slaves are running on. The problem is, since there are several different types of operating systems, there isn't a simple way to do it (that I know of yet).

I've tried using a Groovy script, but that can mostly just be used for restarting the slave service, not the machine itself. This was my reference: https://wiki.jenkins.io/display/JENKINS/Monitor+and+Restart+Offline+Slaves

I've also looked into an Ansible approach, but many machines are unreachable, and Ansible requires different types of scripts for Windows vs. Unix.

The best approach I've seen so far is here: https://stackoverflow.com/questions/29165080/restart-jenkins-slave-from-master

But it doesn't explain how to run this on every single slave, it just looks like it executes on one slave. Has anyone else done anything like this before? I'd love to create a single job, or single script than can be executed and it restarts every single node (using a command like shutdown -r -f) connected to Jenkins regardless of platform.

Answer 1

I solved this by creating a Multi-Configuration job and selecting all the slaves by name. Then, I added two Conditional Step based on "Execution Node." I selected only the windows slaves for the first one, and only the Mac/Centos slaves for the second one. For each conditional step, I added a single step: Execute Windows Batch Command for the Windows condition, and Execute Shell for the unix section. Here are the contents of my Batch and Shell boxes respectively:

Batch:

echo "Restarting Windows Machine...."
hostname
shutdown -r -f

Shell:

echo "Restarting Unix Machine...."
hostname
sudo shutdown -r now

I commented out the shutdown steps for now in my job so I wouldn't accidentally restart everything before testing. Be sure to uncomment those before attempting to run this.