213

I use nginx as a reverse proxy. Whenever I update the config for it using

sudo "cp -r #{nginx_config_path}* /etc/nginx/sites-enabled/"
sudo "kill -s HUP `cat /var/run/nginx.pid`"

I face a brief downtime. How can I avoid that?

Saurav Shah
  • 2,245
  • 2
  • 14
  • 4
  • 1
    Are those meant to be command line commands? I've never seen anyone wrap an entire sudo command in quotes like that, it may not be necessary. – brianmearns Apr 12 '16 at 12:59
  • 5
    Just a general comment: I think the standard/recommended practice is create a soft/symbolic link for your site configuration under `sites-enabled`, not copy it. Not related to your particular issue, but you may want to look into that. – brianmearns Apr 12 '16 at 13:01
  • 2
    You should not be facing a downtime. `kill HUP` is the way to do a graceful reload in nginx. – Jonathan Vanasco May 06 '16 at 16:40

6 Answers6

284

Run service nginx reload or /etc/init.d/nginx reload

It will do a hot reload of the configuration without downtime. If you have pending requests, then there will be lingering nginx processes that will handle those connections before it dies, so it's an extremely graceful way to reload configs.

Sometimes you may want to prepend with sudo

Community
  • 1
Hengjie
  • 3,123
  • 1
  • 16
  • 10
  • 15
    Both of those should do exactly what the question states: send `SIGHUP` to the nginx master process. There should not be a difference. http://nginx.org/en/docs/control.html – Gnarfoz Jul 27 '12 at 12:36
  • When I issue the command on CentOS it keeps saying "Usage /etc/init.d/nginx (start..stop...restart..reload)" .. and that's exactly how I used it. Within the file /init.d/nginx I found kill -HUP `cat $PIDFILE` || echo -n " can't reload" – mashup Apr 13 '15 at 18:21
  • 1
    do you know what the difference is between `service nginx reload` and `nginx -s reload`? If I run the former, I get this output: `Reloading nginx configuration: nginx.`, but my changes aren't updated. If I run the latter, I get no output, but my changes are reflected. – Ryan Quinn Apr 21 '17 at 03:46
  • I just tried this after adding a `log_not_found` directive but found that I had to actually do a restart to get it to work. I guess reloading does not work for all directives? – mydoghasworms Jun 22 '17 at 12:24
  • I get downtime when I do this. For example I get blips like 502 bad gateway which is killing some requests. I'm running an old version of nginx - not sure if that's the reason – Mark Aug 05 '20 at 14:48
  • sorry not bad gateway but Error socket hang up – Mark Aug 05 '20 at 17:56
  • This is not an option when nginx is running in a container. – reinierpost May 12 '21 at 14:54
  • This doesn't really answer op's question though, does it? Reload still causes downtime, and kills the site processes even if they're not effected by the site config. It does it more gracefully than a full restart, but it still causes downtime, which op doesn't want. – Cerin Jan 18 '22 at 16:55
  • Worth pointing out that on RedHat 7 and CentOS 7 the command changed to `systemctl reload nginx` – Peter Thoeny Jun 16 '23 at 01:22
141

Run /usr/sbin/nginx -s reload

See http://wiki.nginx.org/CommandLine for more command line options.

Shiv Kumar Sah
  • 1,511
  • 1
  • 9
  • 4
12

No, you are incorrect, you aren't supposed to be facing any downtime with the procedure you describe. (Nginx can do not only configuration reload on the fly without any downtime, but even the upgrade of the executable on the fly, still without any downtime.)

As per http://nginx.org/docs/control.html#reconfiguration, sending the HUP signal to nginx makes sure that it performs a graceful restart, and, if the configuration files are incorrect, the whole procedure is abandoned, and you're left with the nginx as before sending the HUP signal. At no point should any downtime be possible.

In order for nginx to re-read the configuration file, a HUP signal should be sent to the master process. The master process first checks the syntax validity, then tries to apply new configuration, that is, to open log files and new listen sockets. If this fails, it rolls back changes and continues to work with old configuration.

cnst
  • 13,848
  • 9
  • 54
  • 76
11

Nginx and Signals

The kill approach you used (kill -s HUP $(cat /var/run/nginx.pid) is correct. Init scripts for RH or Debian distributions are in the end also implemented using kill command. You can check Init example from nginx website or contents of Ubuntu Nginx package.

There are multiple signals, that nginx can listen to (mentioned in wiki):

  • TERM, INT - Quick shutdown.
  • QUIT - Graceful shutdown.
  • KILL - Halts a stubborn process.
  • HUP - Configuration reload. Start the new worker processes with a new configuration. Gracefully shutdown the old worker processes.
  • USR1 - Reopen the log files.
  • USR2 - Upgrade Executable on the fly.
  • WINCH - Gracefully shutdown the worker processes.

Nginx Reload

Nginx reload (HUP signal) is more specifically implemented as several steps [1,2]:

  • The master process checks the syntax validity.
  • Applies new configuration, that is, to open log files and new listen sockets.
  • If this fails, it rolls back changes and continues to work with old configuration.
  • If this succeeds, it starts new worker processes, and sends messages to old worker processes requesting them to shut down gracefully.
  • Old worker processes close listen sockets and continue to service old clients.
  • After all clients are serviced, old worker processes are shut down.

Only one issue I can think of why you had downtime (based on the reload process) is that you were using only one worker process (worker_processes directive), which by design was serving old clients, but had closed listen socket, therefore you couldn't open new connection.

I can also recommend you to always use /usr/sbin/nginx -t to validate configuration files before applying new config.

Nginx Reload in Depth

Reconfigure signal is handled in file ngx_process_cycle.c and we can see it starts new worker processes in function ngx_start_worker_processes(...) and at the end it stops old worker processes in function ngx_signal_worker_processes(...), which iterates over them with NGX_SHUTDOWN_SIGNAL signal.

Resources:

Lirt
  • 800
  • 6
  • 8
  • So that means that "nginx -s reload" does not restart the nginx service (i.e. resulting in a new pid) but just instructs the nginx service to reload its configuration without a restart? If I have a docker container in which I am attached to the nginx process with "nginx -g 'daemon off;' " making a reload would not kill my container then? – Crine Jul 10 '20 at 18:51
  • 1
    @Crine No, it will not kill your container. – Lirt Jul 15 '20 at 07:15
  • Interestingly, I'm doing this and it doesn't seem to be picking up changes made to the `nginx.conf` file. I've added a few `location` blocks to test and indeed hitting localhost/test didn't exist even after sending the nanny process a HUP (and verifying new worker process pids). Is there any insight available into the steps you outlined here (ie, applies new config, rolls back, etc)? Interested in debugging this. – a p Sep 20 '21 at 21:42
  • @ap can you confirm that the configuration file you use is actually working and the issue is only in reload process? – Lirt Sep 21 '21 at 09:17
10

For completeness, the systemd way of doing it:

systemctl reload nginx
Selcuk
  • 203
  • 2
  • 8
2

Usually, reloading configuration file of a service should not affect the running service. However, this depends on how the SIGHUP signal is processed.

If a specific service is experiencing a downtime during reload, this can be circumvented by running the same service on multiple servers preferably using a load balancer. In this case, you can take out one server at a time and reload/restart it. Then, it can be re-added after confirming it is OK.

Khaled
  • 36,533
  • 8
  • 72
  • 99
  • While this does not directly answer the question, this is definitely a best-practice scenario that the OP would be smart to follow for avoiding downtime in general. – Andrew M. Apr 11 '12 at 17:29
  • 1
    Details on how nginx handles different signals: http://nginx.org/en/docs/control.html – Gnarfoz Jul 27 '12 at 12:37