2

We have an nginx instance sitting in front of apache 2 (nginx handles static resources, dynamic content requests get proxied to apache) which is primarily using mod_wsgi to server various python/django apps.

We have suddenly seeing a high server load (steadily climbing to more than 50, normally it's less than 1) and are looking for the best ways to track down the cause.

Is there a way to see which virtual hosts are responsible for the apache processes?

Some general tips in diagnosing the cause of high server load would be welcome too.

Edit: Thanks to everyone for their help, it turned out to be caused by a code error and no server reconfiguration was necessary. However, the tips provided will be useful in the future.

Andrew Ingram
  • 121
  • 4

2 Answers2

4

The mod_status module in Apache will show you which virtualhost / request a particular process is handling, but this will only be useful to you if the requests are long-lived. Adding execution time (%D) to your Apache LogFormat is also useful.

More important is tracking down the bottleneck. For this you need to investigate iostat and vmstat (usually installed by default) and a lovely newer tool that combines the two and adds even more stuff, dstat.

Insyte
  • 9,394
  • 3
  • 28
  • 45
  • We have just had dstat installed, but we're not really sure how to use it in this context. – Andrew Ingram Aug 24 '09 at 16:19
  • Run `dstat` and take a look at the output: What is the CPU spending its time on? If the "usr" column is where the high numbers are, your process is CPU bound. If it's the "wai" column, your CPU is blocking on IO. If that's the case, take a look at your disk and paging columns. If you see blocks continually being swapped in and out, you're running out of memory. Those are just the two most common / likely scenarios. – Insyte Aug 24 '09 at 22:55
  • If it's still not making sense, feel free to edit your question and add 15 or 20 lines of dstat output. – Insyte Aug 24 '09 at 22:56
  • it's calmed down at the moment, but when it happens again we'll try Jeff Waugh's suggestion. If that doesn't work I'll add in the dstat output. – Andrew Ingram Aug 25 '09 at 08:31
  • however, the machine has 20gb of RAM, so I would be suprised if it is a memory problem – Andrew Ingram Aug 25 '09 at 08:33
1

If you're seeing a lot of action in the 'wa' column under 'cpu' when you run vmstat (ie. a lot of time spent waiting on input/output), then I suspect the problem is actually nginx, not Apache. It may be doing too much I/O because its buffers aren't large enough to handle the proxied content.

Try adding this to nginx.conf:

proxy_buffers  32 4k;

Increase as necessary.

Jeff Waugh
  • 11
  • 2
  • This seems like a plausible explanation, the primary function of the server is serving dynamically generated resized images, so almost all files served are going to be large compared to html. Do you have any advice on optimising nginx for this kind of proxying? – Andrew Ingram Aug 25 '09 at 09:06
  • It's all about getting the proxy buffer settings right... You want the proxy_buffer* settings big enough to avoid hitting the disk as much as possible, and have proxy_temp_path somewhere sensible just in case nginx does have to hit the disk. – Jeff Waugh Aug 25 '09 at 09:55