8

I am running MongoDB on an Ubuntu server. It's using an upstart script to start mongod when the machine boots. I noticed that if the process crashes, it doesn't get restarted.

How can I make sure that if it crashes, the mongod process is restarted?

Flavien
  • 367
  • 1
  • 2
  • 8

7 Answers7

13

I found the answer myself: the simplest way to achieve that is to add the following two lines at the end of the upstart script installed by MongoDB (/etc/init/mongodb.conf):

respawn
respawn limit 10 90

This will try to restart the process if it terminates, and stop if it crashes more than 10 times in 90 seconds.

Flavien
  • 367
  • 1
  • 2
  • 8
4

I don't think newer versions of MongoDB/Ubuntu use upstart. Here's how I achieved this on Ubuntu 18.04, MongoDB 4.0.2:

Open /lib/systemd/system/mongod.service in a text editor:

sudo nano /lib/systemd/system/mongod.service

Add Restart=on-failure and RestartSec=10 to restart the server after 10 seconds whenever it exits with a non-zero exit code:

[Service]
User=mongodb
Group=mongodb
EnvironmentFile=-/etc/default/mongod
Restart=on-failure
RestartSec=10
...

Use Restart=always if you'd like to restart no matter the exit code.

Then you can systemctl daemon-reload and then restart the service, or just reboot the server.

joe
  • 153
  • 4
  • Are services in `/lib/systemd/system/` managed by packages themselves? I.e. do I need to make the change again when I update mongod? – Jespertheend Aug 18 '23 at 09:26
3

Building on the accepted answer (and since I do not have enough rep to comment on it)

Respawn is NOT a mongo configuration parameter, it is for the upstart service configuration file.

respawn
respawn limit 10 90

Digital Ocean has a nice tutorial for using upstart and systemd (newer). https://www.digitalocean.com/community/tutorials/how-to-configure-a-linux-service-to-start-automatically-after-a-crash-or-reboot-part-1-practical-examples

Justin Thyme
  • 31
  • 1
1

Open mongod.service by this systemctl command:

sudo systemctl edit --full mongod.service

edit the [service] block by just add these two commands:

Restart=on-failure
RestartSec=10

and restart the service

sudo systemctl restart mongod.service

OR, check the status, again

sudo systemctl status mongod.service

YOU CAN USE THIS TECHNIQUE IN ANY SERVICE TO RESTART ON FAILED

@syedasadrazadevops

Community
  • 1
SyedAsadRazaDevops
  • 111
  • 4
1

I think you want something like Monit or God here to automatically restart services when they stop/crash. Your description of the crash itself would suggest the OOM Killer, which can be avoided by configuring some swap space on the host.

Adam C
  • 5,222
  • 2
  • 30
  • 52
0

First of all, you should understand why the system service crashes. Lack of memory? Bugged libs? Wrong or not updated package? Your software using MongoDB is not working properly?

In the unlucky case you can't reach an answer, you could brute force your system: I'd check regularly via a script if the mongod PID file exists, it should be stored somewhere in /var/run, if not, service mongod start.

fsoppelsa
  • 457
  • 1
  • 6
  • 12
  • Yes, I know why it crashed, the process ran out of memory. I just had to restart it manually. I'd like the process to restart regardless of why it crashed. A server with no mongod process running is no good to me. On Windows I can configure services to restart automatically if they exit. My question was about what that script should be - I'm sure someone wrote one before. – Flavien Jun 13 '13 at 17:14
  • Then use Windows. – fsoppelsa Jun 14 '13 at 08:04
0

You could use Supervisord, this tool can restart anything you need and it is very simple to configure.

curratore
  • 113
  • 1
  • 5