-1

What is the recommended methodology to upgrade a HashiCorp Nomad server or client on CentOS Linux 7.5 without downtime?

I'm trying to migrate from v0.10.4 to the just-released v0.11.

Is there a way to perform a lazy-upgrade that will defer/wait for existing tasks to end before swapping binaries to ensure zero downtime?

Elle Fie
  • 681
  • 6
  • 21

1 Answers1

1

The official Nomad upgrade guide covers everything you need.

Basically the process consists of the following steps

  • Replace an old Nomad binary with a new one
  • Restart Nomad process

I've just tested it on one of my staging servers and it worked like a charm. Docker containers have not been restarted during the Nomad update process.

vkozyrev
  • 1,770
  • 13
  • 13
  • 1
    The Nomad binary cannot be replaced while it is in use: "/usr/bin/nomad: Text file busy". Bringing down the Nomad client's systemctl service, replacing the binary, and bringing it back up shuts down all running services, both raw_exec and dockerized. – Elle Fie Apr 23 '20 at 17:13
  • 1
    Interesting! I've never seen this and have performed in place replacements many times. How do you perform the replacement? Simple `mv /path/to/new/nomad /usr/bin/nomad` works well for me while Nomad is still running. – vkozyrev Apr 24 '20 at 08:56
  • My bad, mv works while scp/cp does not, to get around the "Text file busy" error. The "sudo systemctl restart nomad-client.service", however, does restart currently-running service tasks. This is especially problematic for services like Redis that have not been configured with persistence on disk. – Elle Fie Apr 25 '20 at 14:49
  • Could you provide the output of `sudo systemctl cat nomad-client.service`? I am interested to see what happens in your systemctl unit that may cause the restart of containers because when I restart my Nomad service it does not affect the containers. – vkozyrev Apr 26 '20 at 09:18
  • seconded that restarting nomad doesn't restart my docker services – maxm May 28 '20 at 17:25