Wrong client IP passed to Apache from Varnish and HAProxy

Question

I have a setup where I am using Varnish/Hitch >> HAProxy >> Apache. It works except for a problem where the client IP address isn't being passed correctly to the backend Apache server. The Apache log shows the IP address of the machine HAProxy is running on.

My Varnish command line contains:

varnishd -b 127.0.0.1:8080 -a 127.0.0.1:8000,PROXY

Hitch has this:

backend = "[127.0.0.1]:8000"
write-proxy-v2 = on

HAProxy is configured with:

defaults
    option forwardfor
    mode http

frontend CacheFrontend
    bind    *:8080 

backend apache
    server          apache web01:80

In Apache I am using the remoteip_module and have this in httpd.conf

RemoteIPHeader X-Forwarded-For

From what I've read, there is no reason for me to change the log format in Apache when using this module.

I'm unsure where the misconfiguration is.

EDIT:

Here's a short PHP script showing what is being passed to Apache:

<?php

echo $_SERVER['HTTP_X_FORWARDED_FOR'] . PHP_EOL;
echo $_SERVER['REMOTE_ADDR'] . PHP_EOL;

<redacted_client_ip>, 127.0.0.1
10.7.7.107

10.7.7.107 is the IP of the HAProxy machine.

What is being sent in `X-Forwarded-For`? – tater Dec 23 '20 at 07:17 — tater, Dec 23 '20 at 07:17
I edited my question to include that information. – Tuaris Dec 23 '20 at 07:36 — Tuaris, Dec 23 '20 at 07:36

score 0 · Accepted Answer · answered Dec 23 '20 at 08:28

PROXY protocol is definitely the way to go, as it transports the original client IP address regardless of the number of hops.

However, the make this work, all hops should understand the PROXY protocol.

Varnish

I didn't see this in your config overview, but you should definitely set the .proxy_header property in the backend definition of your Varnish server.

Please remove the -b Varnish command line option and replace it with the typical -f /etc/varnish/default.vcl. In your default.vcl file you should declare your backend and make sure it speaks PROXY.

Here's an example of the default.vcl file:

vcl 4.0;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .proxy = 2;
}

As mentioned, the -b should be replaced with a -f. This is what this could look like:

varnishd -f /etc/varnish/default.vcl -a 127.0.0.1:8000,PROXY

HaProxy & Apache

As far as HaProxy is concerned: please make sure incoming HaProxy connections also use PROXY.

And for Apache, you can choose to just read the X-Forwarded-For header to get the client IP address, or also accept PROXY protocol connections.

That was the missing piece! I switched to using a VCL and added `.proxy_header = 2;` to the VCL and changed HAProxy's bind line to: `bind *:8080 accept-proxy`. For Apache I see there are two modules that might enable PROXY protocol connections. With `mod_remoteip` there is the directive `RemoteIPProxyProtocol On`. Then there is another module `mod_proxy_protocol` which uses `ProxyProtocol On`. Am I right in thinking that both of those would do the same thing? — Tuaris, Dec 23 '20 at 08:50
@Tuaris both httpd modules allow receiving "proxy" protocol. mod_proxy_protocol code was rolled into mod_remoteip though — Rob Olmos, Jan 26 '22 at 01:50

Wrong client IP passed to Apache from Varnish and HAProxy

1 Answers1

Varnish

HaProxy & Apache