1

I've got nginx running as a load balancer in front of several web servers. I'd like to be able to see live statistics with response time (maybe a histogram of the last 10 seconds?)

I think Varnish has something similar

Tail isn't very useful as there are about 1k requests per second

Do you know of any tool that can do this? (just so that I don't reinvent the wheel)

edit

I think I wasn't clear enough in the original post. We're already using collectd to grather various kinds of stats from all the servers, and we've got a nice interface to see those graphs live.

I'm looking for something similar to htop

Adrian Mester
  • 173
  • 9

3 Answers3

0

I don't have a specific solution here, so hopefully someone else will chime in, but this looks like a job for RRDTool. You'd need to look for (or write) something which will pull the data you need from the log into the round-robin database, but the tool is set up to handle just what you're talking about, just the most-recent chunk of time-series data.

pjmorse
  • 1,550
  • 1
  • 17
  • 34
0

You know about the Nginx stats stub, right? There are also some links at the bottom of that page related to Nginx monitoring techniques. I'm not sure about getting the response time out of that, though.

cjc
  • 24,916
  • 3
  • 51
  • 70
0

Unfortunately, I don't know of any existing way to get live response time data. Given that I'd expect it to be a rare occasion when you need to check such data, it seems likely that the performance cost it requires (i.e. additional processing/logging per request) does not seem justified. I'd suggest simply logging the response time data to your logs (e.g. like this) and run a shell script to parse a portion of your logs (and output a summary) when you are interested in the data.

The live statistics are not too difficult to come by - but getting response time with them is harder. The approaches below only provide raw data - they do not graph it for you.

If you can live without response time (e.g. just req/s - and a breakdown of # reading, writing, active, total connections, etc), use the HTTP Stub Status Module. It provides a text output of information that can be easily be queried and parsed. (To use this module, Nginx needs to be compiled with --with-http_stub_status_module)

Sample Output:

Active connections: 291
server accepts handled requests
  16630948 16630948 31070465
Reading: 6 Writing: 179 Waiting: 106

(The 3rd line is accepted connections, handled connections, accepted requests)

There is an interesting article on using this data to generate graphs with rrdtool, here.

If the response time data is particularly essential to you, it should be possible to modify the Stub Status Module. The code (only about 5kB actually) is in src/http/modules/ngx_http_stub_status_module.c. For instance, there is a modification here that has the module output aggregated counts of HTTP status codes returned.

Lastly, if you just want response time. You could save the data to Memcached using the HTTP Memc Module. For the purposes of your histogram, you would essentially create 2 keys per second - base the key on the time (e.g. $time_local) - one key to store a counter of the number of requests in that second, and one to store the average response time (or cumulative response time) - working with the $request_time and/or $upstream_response_time variables.

I am sure you can come up with an easy way of graphing the data, once you have it, but here are some possibly starting points:

  • Have a script pull the data (from stub, or memcached, etc) and generate ascii bars (e.g. like this), use watch to monitor the output of the script.
  • If your server has PHP (and you willing to use it (not the ideal scripting language)), consider modifying this script.
  • Finally, perhaps consider gnuplot (output as_ascii), although it seems excessive for the purpose.
cyberx86
  • 20,805
  • 1
  • 62
  • 81