Nginx and passenger request timeout (nginx status code 499)

Question

DISCLAIMER: I have read numerous posts here, on stackoverflow and other pages and I could not come up with a solution, so hopefully this helps.

We are running a Rails App on Passenger together with nginx and HAProxy.

Passenger: 4.0.7 Nginx: 1.4.1

The app is running fine, but from time to time, requests take too long to finish:

# nginx access.log x.x.x.x - - [19/Mar/2015:01:07:19 +0100] "GET / HTTP/1.0" 499 0 "-" "runscope-radar/2.0"

The request takes too long to finish, nginx returns a 499 and HAProxy converts this into a 504. After about 30 minutes, the server goes back to normal, but the error keeps reappearing.

I'm pretty sure it has nothing to do with HAProxy as a curl to localhost on the machine also returns a timeout.

There is nothing special going on the server (according to htop the CPU usage of 4 cores is never above 10% and from the 8GB about only 1 GB is used, the loadavg is under 0.1 all the time).

I set the ulimit to 65535.

The config file for nginx is:

    user  www-data www-data;
    worker_processes  1;
    worker_rlimit_nofile 20000;

    events {
        worker_connections  768;
    }

    http {
        passenger_root /opt/passenger/passenger-4.0.7;
        passenger_ruby /usr/bin/ruby1.8;

        include       mime.types;
        default_type  application/octet-stream;

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;

        # Virtual Hosts
        include /opt/nginx/conf/sites-enabled/*;
    }

The phusion location in nginx is:

server {
  charset utf-8;
  listen 3101;

  root /srv/ruby/code/current/public;

  passenger_enabled on;
  passenger_base_uri /;


  passenger_ruby /home/code/.rbenv/versions/1.9.3-p448/bin/ruby;

  passenger_min_instances 3;

  passenger_show_version_in_header off;
  passenger_friendly_error_pages off;
}

I played around with various parameters (worker_processes, worker_connections, worker_rlimit_nofile) but nothing helped.

Any more suggestions?