4

Useless GET requests for favicon.ico and assets in my /static/ folder are littering my nginx access log. And it's a busy server - meaning I/O is being wasted too. Not to mention my fail2ban installation sometimes takes warped decisions!

How do I exclude these nuisance log lines? I'm on nginx 1.4.x.

I found a way to do it in apache, but not in nginx.

Here's my nginx virtual host file:

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=my_cache:10m max_size=100m inactive=6m;

upstream my_server {

  server unix:/home/myuser/myfolder/myapp/myapp.sock  fail_timeout=0;
}

server {
    server_name example.com www.example.com;

    listen 80;

    return 301 https://example.com$request_uri;
}

server {
    server_name www.example.com;

    listen 443 ssl;
    listen [::]:443 ssl;

    . . . SSL related stuff . . .   

    . . . adding security related headers . . .

    return 301 https://example.com$request_uri;
}



server {

    server_name example.com;
    listen 443 ssl;
    listen [::]:443 ssl;

    . . . SSL related stuff . . .

    . . . adding security related headers . . .

    charset utf-8;
    underscores_in_headers on;

    limit_conn conn_limit_per_ip 10;
    limit_req zone=req_limit_per_ip burst=20;

    # write error log file for https errors:
    error_log /var/log/nginx/https-error_log warn;


    location ~* \.(?:ico|css|js|gif|jpg|jpeg|png|svg|woff|ttf|eot)$ { 

    root /home/myuser/myfolder/myapp; 
    access_log off;
    expires 120d;
    add_header Pragma public;
    add_header Cache-Control public;

    }

    location = /favicon.ico { access_log off; log_not_found off; }

    location /static/ {

    access_log off;
        alias /home/myuser/myfolder/myapp;
    }

    location /status {
        stub_status on;
        allow 127.0.0.1;
        allow 40.114.247.165;
        deny all;
    }




    location / {

        #proxy_pass_request_headers on;
        proxy_buffering on;
        proxy_buffers 24 4k;
        proxy_buffer_size 2k;
        proxy_busy_buffers_size 8k;

        try_files $uri @https_proxy_to_app;
    }

    location @https_proxy_to_app {
        proxy_set_header X-Forwarded-Proto https;
        # additional proxy parameters
        include proxy_params;

        proxy_redirect off;
        proxy_pass http://my_server;
    }


    # Error pages
    error_page 500 502 503 504 /500.html;
    location = /500.html {
        root /home/myuser/myfolder/myapp/templates/;
    }

    location = /backflip.png {
        root /home/myuser/myfolder/myapp/static/img/;
    }

}

In nginx.conf, I have defined the following in the http block:

log_format compression '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" "$gzip_ratio"';

access_log /var/log/nginx/access.log compression;

And here's an example of stuff I'm seeing in the access logs:

192.16.53.127 - - [24/Sep/2017:01:12:24 +0000] "GET /static/emoticons/puke1.png HTTP/1.1" 301 178 "https://example.com/page2/" "service-1460643764266048;Mozilla/5.0 (Linux; Android 6.0; QMobile X32 Power Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/55.0.2883.91 Mobile Safari/537.36" "-"

152.195.94.170 - - [24/Sep/2017:01:12:24 +0000] "GET /static/img/favicon-192.png HTTP/1.1" 301 178 "https://example.com/comment/3" "Opera/9.80 (Android; Opera /24.0.2254/69.162; U; en) Presto/2.12.423 Version/12.16" "-"
Hassan Baig
  • 2,325
  • 12
  • 29
  • 48
  • 1
    access_log off in the correct block does what you need. I have it working just fine on my Nginx server. You need to supply your full Nginx server block configuration and a sample of the access logs show the problem. – Tim Sep 24 '17 at 00:57
  • Are you looking for something like this: https://serverfault.com/questions/414027/how-to-block-nginx-access-log-statements-from-specific-user-agents – James Shewey Sep 24 '17 at 01:31
  • Possible duplicate of [How to block Nginx access log statements from specific user agents](https://serverfault.com/questions/414027/how-to-block-nginx-access-log-statements-from-specific-user-agents) – James Shewey Sep 24 '17 at 01:31
  • @JamesShewey: I'm trying to turn off access logging for static assets. `access_log off;` ought to have worked. My question's more about how I've misconfigured this directive - so I think taking the approach pointed out in this other question is excessive in my case. – Hassan Baig Sep 24 '17 at 01:44
  • Use headers to work out which location block is matching - static or the one with the extensions. That will give you another clue. See [debugging Nginx location issues](https://www.photographerstechsupport.com/tutorials/hosting-wordpress-on-aws-tutorial-part-4-wordpress-website-optimization/#debuggingnginxlocations). – Tim Sep 24 '17 at 02:58
  • Okay, but tell me something. If I have `access_log off` in both the static and extensions related block, should I be able to solve the problem anyway? – Hassan Baig Sep 24 '17 at 03:01
  • The examples you give are 301 responses, possibly from the first or second server block. Perhaps you should turn logging off in those blocks. – Richard Smith Sep 24 '17 at 08:27
  • @RichardSmith: but that would turn logging off for a lot of legitimate requests as well, so lots of collateral damage, no? – Hassan Baig Sep 24 '17 at 10:43
  • 1
    You can still use `location` blocks to fine tune it. Alternatively, upgrade your `nginx` version and use the conditional `access_log` directive. – Richard Smith Sep 24 '17 at 11:13

2 Answers2

9

Each server block generates logs and each location block generates logs. You cannot just add a location block to an existing configuration and expect it to work without considering how it interacts with all of the other location blocks in the configuration. See how nginx processes a request for more.

A simple solution to your problem might be to use a global rule to determine if a request should be logged or not.

For example, in your nginx.conf file:

log_format compression '$remote_addr - $remote_user [$time_local] '
                '"$request" $status $body_bytes_sent '
                '"$http_referer" "$http_user_agent" "$gzip_ratio"';

map $request_uri $loggable {
    default                                             1;
    ~*\.(ico|css|js|gif|jpg|jpeg|png|svg|woff|ttf|eot)$ 0;
}

access_log /var/log/nginx/access.log compression if=$loggable;

See this document for more.

Richard Smith
  • 12,834
  • 2
  • 21
  • 29
  • Upgrading `nginx` is not really relevant to the answer. – Richard Smith Sep 25 '17 at 13:45
  • Well it's added info in that I'm unable to get `map` to work with vanilla `1.4.x`. So don't you think a one-stop-shop answer should include upgrading details as well? – Hassan Baig Sep 25 '17 at 13:47
  • Firstly, 1.4.x is quite old. Secondly, the update procedure is dependent on what platform you are using. `nginx` is available across a wide range of OS, and the update procedure is different for each. If you wish to document the update procedure you could write a [question & answer](https://stackoverflow.blog/2011/07/01/its-ok-to-ask-and-answer-your-own-questions/) and make it clear in the title which operating system it applies to. – Richard Smith Sep 25 '17 at 13:54
  • 1
    Using nginx/1.10.2, this solution has a typo: The docs indicate that "A regular expression should either start from the “~” symbol for a case-sensitive matching, or from the “~*” symbols (1.0.4) for case-insensitive matching." (ref: [Module ngx_http_log_module](http://nginx.org/en/docs/http/ngx_http_map_module.html)) For this version, **line 7 should be written with a leading tilda**: `~\.(ico|css|js|gif|jpg|jpeg|png|svg|woff|ttf|eot)$ 0;` – Chris Jun 14 '18 at 20:02
  • Sometimes people suffix a request with GET/query parameters. This will cause the regexp to fail due to trailing content. **Best solution:** Use `$uri` instead of `$request_uri`. Alternatives: (1) do not anchor to EOL, e.g. remove the "$" (2) use `$request_filename` instead of `$request_uri` (similar to using `$uri`) – Chris Jun 14 '18 at 20:50
  • @Chris You are right, but this is just an example, and admins will need to choose a regular expression to match the URIs they wish to log. The disadvantage of using `$uri` is that it can be rewritten internally, whereas `$request_uri` is guaranteed to be the original request. But removing `$` or using `($|\?)` instead are certainly options. – Richard Smith Jun 14 '18 at 21:39
0

Upgrading from nginx 1.4.x to nginx 1.13.x (Ubuntu 14.04 LTS):

1) Go to /etc/apt/sources.list.d/ and create nginx.list via sudo nano nginx.list.

2) Paste these two in separate lines: deb http://nginx.org/packages/mainline/ubuntu/ trusty nginx and deb-src http://nginx.org/packages/mainline/ubuntu/ trusty nginx

3) Save and exit the file. On the terminal, enter sudo wget -q -O- http://nginx.org/keys/nginx_signing.key | sudo apt-key add -

4) Next sudo apt-get update

5) Finally sudo apt-get install nginx. The latest version would be installed.

Selective logging: Richard's answer is better than mine, so deprected mine.

Hassan Baig
  • 2,325
  • 12
  • 29
  • 48