nginx suddenly redirecting everything to 403 page, can this be debugged or traced?

Question

I'm not sure what happened to my nginx install. Suddenly all page requests are being redirected to the 403 page.

Yesterday I tried to add a user agent to block, restarted service from that point everything was being sent to 403. I backed out that change, restart nginx and everything is still being directed to the 403 page. Even if I remove the $http_user_agent and $http_referer if statements everything is still sent to 403.

I have even restored the entire nginx folder from a backup and all my page requests are continued to be directed to the 403 page....

Not sure how to troubleshoot this, the conf files come back clean. Is there a trace I can do for nginx when requests come in?

[root@soupcan nginx]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Here is the website conf:

server {
    listen       80;
    server_name  localhost;

    #charset koi8-r;

    access_log  /var/log/nginx/website1/access.log  main;
    error_log /var/log/nginx/website1/error.log;

    root /srv/www/website1;

    ## Block http user agent - morpheus fucking scanner ##
    if ($http_user_agent ~* "morfeus fucking scanner|ZmEu|Morfeus strikes again.|OpenWebSpider v0.1.4 (http://www.openwebspider.org/)") {
        return 403;
     }

    if ($http_referer ~* (semalt.com|WeSEE)) {
        return 403;
    }

    ## Only allow GET and HEAD request methods. By default Nginx blocks
    ## all requests type other then GET and HEAD for static content.
    if ($request_method !~ ^(GET|HEAD)$ ) {
      return 405;
    }


    location / {
        index  index.html index.htm index.php;
        ssi on;
    }

    location ~ \.php {
        try_files $uri =404;
        include /etc/nginx/fastcgi_params;
        fastcgi_pass 127.0.0.1:9000;
        #fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME /srv/www/website1/$fastcgi_script_name;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }


    # Redirect server error pages to the static page
    error_page 403 404 /error403.html;
    location = /error403.html {
        root /usr/share/nginx/html;
    }
}

nginx.conf

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    gzip  on;
    gzip_disable "msie6";
    gzip_min_length 1100;
    gzip_vary on;
    gzip_proxied any;
    gzip_buffers 16 8k;
    gzip_types text/plain text/css application/json application/x-javascript
        text/xml application/xml application/rss+xml text/javascript
        image/svg+xml application/x-font-ttf font/opentype
        application/vnd.ms-fontobject;

    server_tokens off;

    include /etc/nginx/conf.d/*.conf;
    # Load virtual host configuration files.
    include /etc/nginx/sites-enabled/*;

    # BLOCK SPAMMERS IP ADDRESSES
    include /etc/nginx/conf.d/blockips.conf;
}

Permissions for webroot dir:

[root@soupcan nginx]# namei -om /srv/www/website1/
f: /srv/www/website1/
 dr-xr-xr-x root  root   /
 drwxr-xr-x root  root   srv
 drwxrwxr-x brian nobody www
 drwxr-x--x brian nobody website1

EDIT

Found out that CentOS 6.6 and SELinux breaks nginx. Im still looking for a solution, but here is the cause.

EDIT 2

Solution posted below.

Answer 1

The issue was caused by upgrading CentOS from 6.5 to 6.6 and how SElinux allows content type through. With this upgrade SElinux by default only allows httpd_t content through (similar to how they treat apache), and because I store all my webcontent in /srv/www/ these user created folders did not have content label set automatically by the system.

To check this run the following command against your webroot and your /etc/nginx directories and compare the content types:

ls -Z /srv/www/

I've run these commands and restarted nginx and everything is now working normally.

grep nginx /var/log/audit/audit.log | audit2allow -m nginx > nginx.te
grep nginx /var/log/audit/audit.log | audit2allow -M nginx
semodule -i nginx.pp

Im not sure what this SElinux module does, but I found it reading this post about the same issue. I may try backing it out today, because I think the second thing I did to fix this actually worked.

[09:15 AM] robotoverlord ~>chcon -Rv --type=httpd_sys_content_t /srv/www/
[09:15 AM] robotoverlord ~> ls -Z /srv/www/
drwxr-xr-x. www-data nobody unconfined_u:object_r:httpd_sys_content_t:s0 website1
[09:15 AM] robotoverlord ~>service nginx restart

Additional info on content labeling for SElinix

Problem solved!

Answer 2

chmod ogw file 

sets permission on file for owner, group and world, each being the sum of read(4), write(2), execute(1), if wanted

no proper access to read write

drwxr-x--x brian nobody website1

nginx is reading only, so you have to let him into!

cd /srv/
find . -type d -exec chmod 755 {} \;