AWS ELB X-Forwarded-For occasionally missing

Question

My classic load balancer doesn't supply the X-Forwarded-For header occasionally, I can't say for sure whether the same request is going to have it or not.

I noticed that a request to the URL pointing to the load balancer gets proxied via either of the two IP addresses: IP A or IP B. As you can see in the log below, IP A always contain the X-Forwarded-For header and the user agent, the IP B has neither of them. Don't pay attention to the HTTP code, it's intentionally 403 when there is no X-Forwarded-For:

<IP A> - - [20/Apr/2018:15:04:11 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 403 428
<IP B> - - [20/Apr/2018:15:04:12 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 401 611 "-" "curl/7.47.0" <X-Forwarded-For IP>
<IP A> - - [20/Apr/2018:15:04:14 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 403 428
<IP B> - - [20/Apr/2018:15:04:15 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 401 611 "-" "curl/7.47.0" <X-Forwarded-For IP>
<IP A> - - [20/Apr/2018:15:04:16 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 403 428
<IP B> - - [20/Apr/2018:15:04:17 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 401 611 "-" "curl/7.47.0" <X-Forwarded-For IP>
<IP A> - - [20/Apr/2018:15:04:17 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 403 428
<IP B> - - [20/Apr/2018:15:04:18 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 401 611 "-" "curl/7.47.0" <X-Forwarded-For IP>
<IP A> - - [20/Apr/2018:15:04:19 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 403 428
<IP B> - - [20/Apr/2018:15:04:20 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 403 428
<IP A> - - [20/Apr/2018:15:04:21 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 403 428
<IP A> - - [20/Apr/2018:15:04:34 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 403 428
<IP B> - - [20/Apr/2018:15:04:34 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 403 428
<IP A> - - [20/Apr/2018:15:04:35 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 403 428
<IP B> - - [20/Apr/2018:15:04:36 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 403 428
<IP A> - - [20/Apr/2018:15:04:36 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 401 612 "-" "curl/7.47.0" <X-Forwarded-For IP>
<IP B> - - [20/Apr/2018:15:04:37 +0100] "GET /wc-api/v3/orders/?ttt HTTP/1.1" 403 428

What could possibly cause such a behavior?

UPDATE

Both IP A and B wipe the headers randomly, not only IP B.

UPDATE 2

The 403 HTTP errors are caused by something else, not my apache config, so they are not intentional as it's said above

Review your listener configurations on the balancer. Specifically, find one that is in TCP mode. — Michael - sqlbot, Apr 20 '18 at 16:28
According to the logs, your assertion that IP-A always has them and IP-B doesn't is incorrect. Are your health checks passing? — Michael - sqlbot, Apr 23 '18 at 16:28
@Michael-sqlbot right my bad, thanks. So it doesn't depend on the particular IP, both can wipe the headers randomly. I have a TCP healthcheck on port 80 and it's passing. — super.t, Apr 24 '18 at 13:24
What's in the ELB logs to correlate to these requests? Anything interesting or unusual? I can't think of anything that would be expected to cause this, other than the balancer being in TCP mode. It seems like something that should not happen. If you can capture some of the defective requests with wireshark, it might prove informative. Verify also that this is the balancer you think it is -- see "network interfaces" in the left hand navigation pane of the EC2 console. — Michael - sqlbot, Apr 24 '18 at 14:07
@Michael-sqlbot yes the ELB logs match to the target's apache access log. What I've noticed is that the load balancer has 2 network interfaces attached, they are in the same avail zone. I didn't do that intentionally, what's the point in having two? Also according to the logs one of the interfaces is much more likely to lose the headers than the other. — super.t, Apr 25 '18 at 10:09
@Michael-sqlbot also seems like the 403 error is caused by something else, not my apache configuration. — super.t, Apr 25 '18 at 10:36

AWS ELB X-Forwarded-For occasionally missing

0 Answers0