Logging the client IP with Nginx/Varnish/Apache

Question

I have Nginx listening on port 443 as an SSL terminator, and proxying unencrypted traffic to Varnish on the same server. Varnish 3 is handling this traffic, and traffic coming in directly on port 80. All traffic is passed, unencrypted, to Apache instances on other servers in the cluster. The Apache instances use mod_rpaf to replace the logged client IP with the contents of the X-Forwarded-For header.

My problem is that if the traffic is coming via Nginx, while the 'correct' client IP is getting logged in the VarnishNCSA logs, it looks as if Varnish is (understandably) replacing Nginx's X-Forwarded-For header with 127.0.0.1 downstream, and this is getting logged with Apache. Is there a nice simple way to stop Varnish rewriting X-Forwarded-For if it's already populated?

Varnish is used for in-memory caching of static content and load balancing for the back-end servers. I'm not saying that some or all of this couldn't be done in Nginx alone, but other than the logging, it's working very well as is. — jetboy, Oct 24 '12 at 22:51

Shane Madden · Accepted Answer · 2012-10-25T15:49:55.187

Absolutely; the Varnish handling of X-Forwarded-For is actually just defined in the default vcl_recv function.

if (req.restarts == 0) {
    if (req.http.x-forwarded-for) {
        set req.http.X-Forwarded-For =
    req.http.X-Forwarded-For + ", " + client.ip;
    } else {
        set req.http.X-Forwarded-For = client.ip;
    }
}

The default definition of a function is always appended to one you've defined in your active VCL file, but if your defined function always handles a request, then the default logic will never execute.

Set a vcl_recv along these lines:

sub vcl_recv {
    /* Your existing logic goes here */
    /* After that, we'll insert the default logic, with the X-Forwarded-For handling removed */
    /* The return (lookup); at the end ensures that the default append behavior won't have an impact */

    if (req.request != "GET" &&
      req.request != "HEAD" &&
      req.request != "PUT" &&
      req.request != "POST" &&
      req.request != "TRACE" &&
      req.request != "OPTIONS" &&
      req.request != "DELETE") {
        /* Non-RFC2616 or CONNECT which is weird. */
        return (pipe);
    }
    if (req.request != "GET" && req.request != "HEAD") {
        /* We only deal with GET and HEAD by default */
        return (pass);
    }
    if (req.http.Authorization || req.http.Cookie) {
        /* Not cacheable by default */
        return (pass);
    }
    return (lookup);
}

Edit:

Since Varnish handles some connections directly as well, a better approach might be to have it selectively set the header. You'll still want include the full vcl_recv so that the default doesn't apply its own header, but include this at the top:

if (req.restarts == 0) {
    if (!req.http.x-forwarded-for) {
        set req.http.X-Forwarded-For = client.ip;
    }
}

So with the default logic, for non-Nginx traffic, X-Forwarded-For is set to the client IP, and is used by mod_rpaf correctly? For Nginx traffic, X-Forwarded-For would look like '127.0.0.1, 1.2.3.4'? The mod_rpaf documentation states: "...if there is an incoming X-Forwarded-For header and the proxy is in it's list of known proxies it takes the *last* IP from the incoming X-Forwarded-For header and changes the remote address of the client in the request structure", so now I'm confused why the default VCL logic isn't working in my scenario. — jetboy, Oct 25 '12 at 11:06
@jetboy For traffic through nginx, it's the other way around. Assuming that you have nginx configured to set the header (`proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;` - make sure this is in place in nginx), then Varnish will receive a header of `X-Forwarded-For: 1.2.3.4`, and append the client IP from its perspective to the end of that; `X-Forwarded-For: 1.2.3.4, 127.0.0.1`. That `127.0.0.1` is what `mod_rpaf` is grabbing. I've edited my answer with some varnish logic that'll selectively set the header when you need it, and leave it alone when nginx set the header. — Shane Madden, Oct 25 '12 at 15:51
Note to self: Don't skim-read code! Thanks Shane. It makes perfect sense now. — jetboy, Oct 26 '12 at 08:17

Logging the client IP with Nginx/Varnish/Apache

1 Answers1