23

I have deployed my application onto AWS EC2 and I want to implement automation where if I restart my instance or when the Nginx web server is down, it will restart by itself. I do not really know where to start with this.

I heard I can use crontab to schedule automatic monitoring and if it is down, it can send email alerts and restart the webserver.

Dave M
  • 4,514
  • 22
  • 31
  • 30
Newbee Coding
  • 341
  • 1
  • 2
  • 5

8 Answers8

87

It is a feature of SystemD. Override existing unit file for NGINX by running systemctl edit nginx then paste in:

[Service]
Restart=always

Save.

If NGINX is down due to, e.g. the OOM killer, it will be restarted after dying. If you have a configuration error in NGINX, it will not be restarted, of course.

To verify this configuration. start NGINX service with systemctl start nginx, and verify it is running with systemctl status nginx.

Kill it via pkill -f nginx. Confirm that NGINX is running anyway with systemctl status nginx.

Danila Vershinin
  • 5,286
  • 5
  • 17
  • 21
  • Hi there, when I stop the service. It did not start again automatically, it says [Service] : command not found – Newbee Coding Feb 18 '20 at 02:05
  • 1
    @NewbeeCoding That seems like you did not have a nginx.service file to start with. Is this nginx something you built yourself from source code? – Zan Lynx Feb 18 '20 at 05:40
  • Hi, @ZanLynx, this was from a source code that had built it already – Newbee Coding Feb 18 '20 at 05:51
  • 13
    @NewbeeCoding compiling/installing from source is not a good thing on a production system as it involves recompiling for upgrades and holding compilation software in prod is a security risk. Why not go for a packaged install and save a lot of trouble? If you use compiled install you need to set up at the minimum the startup script plus log rotation. For an example of a SystemD script look [here](https://www.nginx.com/resources/wiki/start/topics/examples/systemd/). I recommend copying to `/etc/systemd/system/nginx.service` (`/lib` is where packaged SystemD unit files go). – Danila Vershinin Feb 18 '20 at 12:43
  • @NewbeeCoding I agree 100% with Danila that you should avoid compiling your own software for production systems, if at all possible. I wrote [an elaborate answer on that very topic](https://unix.stackexchange.com/a/404117/109651) that you might find informative (the answer is about Apache on Debian, but the principle is exactly the same). – marcelm Feb 20 '20 at 15:17
  • 1
    This should be the accepted answer – Climax Jan 18 '22 at 05:56
  • this works as advertised, tested it myself – King Friday Jul 06 '22 at 15:35
  • This is the best solution to go with when using systemd – Eric Aug 04 '23 at 17:44
11

Use monit which purpose is to take care of situations like this.

apt install monit

nano /etc/monit/conf.d/nginx.conf

Put content below inside this file and restart monit

check process nginx with pidfile /var/run/nginx.pid
start program = "/usr/sbin/service nginx start"
stop program = "/usr/sbin/service nginx stop"
JoshMc
  • 184
  • 7
iWizard
  • 418
  • 3
  • 12
  • 27
  • Hi @iWizard, when I tried to save the file, it threw back an error saying no such file or directory. Do I have to mkdir the directory? – Newbee Coding Feb 17 '20 at 12:32
  • @NewbeeCoding is your OS Ubuntu? If yes then monit should be under /etc/monit/. If iy there the just try to mkdir /etc/monit/conf.d – iWizard Feb 17 '20 at 12:36
  • I am running, Windows 10 using Ubunto 18.04 LTS to connect to my application – Newbee Coding Feb 17 '20 at 12:42
  • @NewbeeCoding have you checked folder? – iWizard Feb 17 '20 at 12:47
  • when I started the monit service and check the status it says monit: error connection to monit daemon – Newbee Coding Feb 17 '20 at 12:50
  • @NewbeeCoding what about this /etc/init.d/monit status – iWizard Feb 17 '20 at 13:25
  • Some how I got the service to start but it is not working when I stop Nginx and I can't access the 2812 to view the status page – Newbee Coding Feb 17 '20 at 13:31
  • you will have to investigate why is not working but with monit you can also send email when monit successfully restart nginx and when it fails and so on... – iWizard Feb 17 '20 at 13:34
  • 1
    thanks for your solution. Got it working. Now when my Nginx is down, it starts again automatically. But I still can't access the UI to monitor it. but thanks for the help! – Newbee Coding Feb 17 '20 at 14:27
  • 14
    This all feels like solving the wrong problem. Something else is wrong if this is even something you have to worry about, and even failing that, there's no reason to add complexity when systemd is already capable of restarting the service. – Michael - sqlbot Feb 18 '20 at 21:16
  • @Michael-sqlbot question is "How to automatically restart Nginx when it goes down". My anwser solves the question – iWizard Jun 29 '20 at 13:27
5

You already have lots of answers how to do it, but I'd investigate what is happening to make it shut down in the first place, and fix that.

When nginx crashes, all currently running requests will be terminated in an unknown state -- files half-transferred, API calls unreplied. In principle the application on top should deal with this, in practice they seldom do, but at that layer it will manifest as weird and unreproducible behaviour, giving the people using the service an feeling of instability (and rightfully so).

Simon Richter
  • 3,317
  • 19
  • 19
3

Its actually fairly simple

Go to /lib/systemd/system Backup your ngnix systemd unit (in case) with sudo cp ngnix.service ngnix.service.old

Add the following 2 lines to the end of the service block in ngnix.service

Restart=on-failure
RestartSec=5s

Load the new config with sudo systemctl daemon-reload

To test - Lets try killing ngnix

cat /var/run/nginx.pid will give you the PID

sudo kill -9 PID will kill nginix

You'll find that if you check a the PID, there will be a different PID. If you hadn't run those lines, killing ngnix would result in a server that's down. This will only trigger a restart on a non graceful shutdown

Journeyman Geek
  • 6,977
  • 3
  • 32
  • 50
  • 1
    `/lib/systemd/system` is reserved for unit files installed from packages. For custom units/overrides by admin there is a dedicated location, `/etc/systemd/system`. – Danila Vershinin Feb 18 '20 at 12:45
  • To expand on @DanilaVershinin note; it's possibly best to use `sudo systemctl edit ngnix.service` (or manually create `/etc/systemd/system/nginx.service.d/override.conf`) containing the desired extra config. Note that if you wish to override an existing value (that isn't boolean), you often need to set it to an empty value first, then set it to the desired value. If you wish to replace the default `nginx.service` completely, drop it in `/etc/systemd/system/nginx.service`. – Jeremy Davis Feb 18 '20 at 21:14
0

If you want to restart the process like you said in the question title, all the other answers seem great. If you want to restart the instance like you said in the question body, you can use an Auto Scaling Group.

Set up the Auto Scaling Group with minimum and maximum 1 instance, and point the healthcheck to the content served by your Nginx web server (ideally, something that doesn't consume a lot of CPU like robots.txt). The Auto Scaling Group will detect when your instance is unhealthy (might take 3 or 5 healthcheck fails) and will kill the instance and start a new one.

Keep in mind, you will lose all state stored in the instance (as the instance is actually destroyed). Keeping state inside a server is a cloud anti-pattern anyway. You'll also lose caches.

Blueriver
  • 130
  • 4
0

Tried to use systemctl edit nginx and pasted those lines:

RestartSec=5s

But getting this error and when nginx crashes its not restarted /etc/systemd/system/nginx.service.d/override.conf:1: Assignment outside of section. Ignoring.

whoo
  • 11
  • This does not provide an answer to the question. Once you have sufficient [reputation](https://serverfault.com/help/whats-reputation) you will be able to [comment on any post](https://serverfault.com/help/privileges/comment); instead, [provide answers that don't require clarification from the asker](https://meta.stackexchange.com/questions/214173/why-do-i-need-50-reputation-to-comment-what-can-i-do-instead). - [From Review](/review/late-answers/500184) – Ginnungagap Oct 18 '21 at 09:55
  • Add `[Service]` above this line – hack4mer Mar 16 '23 at 04:02
0

auto restart on failures for Nginx on both the prod servers, staging server, and dev server. Server name as:

dev staging prod

I modified "/lib/systemd/system/nginx.service" and added the following line to [Service] section at the end -> Restart=on-failure

after changing the configuration we did, we have to execute the command ' systemctl daemon-reload ' for the auto-restart configuration to take effect.

This should restart the Nginx when it crashes

CHAVDA MEET
  • 101
  • 1
-3

If your distribution is based on systemd you can use sample script like this to start the daemon if it is not started/died

#!/bin/bash
systemctl is-active nginx
if [ "$?" -ne 0 ]
then systemctl start nginx
mail....... #send some mail
fi

the record in cron should be something like

*/5 * * * * /path/to/the/script

And this must be run as root (root cron)

Romeo Ninov
  • 5,263
  • 4
  • 20
  • 26
  • 1
    This is definitely overcomplicating things. Why use an extra cron job to manage a systemd service? You're already using systemd, which is a service manager itself. Simply set the systemd unit file to restart the nginx service when it dies. – Martijn Heemels Feb 20 '20 at 10:21
  • @MartijnHeemels, I create it based on my knowledge of systemd. If systemd can do it via configuration of the server OK, but I was unaware of this fact. Should I repeat what other answer? Are negative votes indicator of wrong or harm solution? – Romeo Ninov Feb 20 '20 at 10:25