We have a staging machine that has recently been returning a 403 status. It is a centOS machine running nginx. Nginx passes off some traffic to a scala Play app and Apache which runs a PHP app (Drupal). The server is protected by basic htaccess auth.
When I navigate to the server, I get the basic auth and log in ok.
If I request the robots.txt file, it comes up right away and the nginx access.log shows the request was a 200 OK and records my htaccess username.
XX.XXX.XXX.XXX - MYUSER [31/Dec/2015:16:16:02 +0000] "GET /robots.txt HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36" "-"
But then when I request the site home page (the drupal home page), I get the 403 error in the browser. When I tail the nginx error logs I see this message is logged moments after my request:
XX.XXX.XXX.XXX - MYUSER [31/Dec/2015:16:18:02 +0000] "GET /index.php HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36" "-"
Apparently the 499 is an nginx specific code for a connection closed by the client while the request was being processed. However my browser is still waiting for a response. More than a minute later I get the 503 error in the browser and this is logged in the nginx access.log:
XX.XXX.XXX.XXX - MYUSER [31/Dec/2015:16:20:32 +0000] "GET /index.php HTTP/1.1" 504 584 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36" "-"
When I take a look at the nginx error.log I see this:
2015/12/31 16:20:32 [error] 11645#0: *78933 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XXX.XXX.XXX, server: staging.mysite.com, request: "GET /index.php HTTP/1.1", upstream: "http://127.0.0.1:8080/index.php", host: "staging.mysite.com"
So I was wondering if the problem was Apache or Drupal. The apache access_log does not record my user's request for index.php and the error_log doesn't show anything for this date. I restarted apache - no change.
To check drupal, I made a simple php test page that bootstraps drupal. I was able to load that page, despite a long delay. This resulted in a record in the nginx access.log, but not the apache access_log.
I'm not very familiar with troubleshooting these issues, or working with nginx, so I've kind of dead-ended here. Does anyone see a reasonable next step?
