0

We have received complaints that some users cannot access the site during the day. Somehow they see maintanence page. When we examined the haproxy logs, we saw that all 503 error's requests' source ips are ipv6. We don't have an AAAA dns entry.

i checked waf and firewall wan rules. Also checked server's logs too. Requests are not forwarded to backend servers.

an example log: [ <131>Nov 16 14:59:33 HaProxy haproxy[54113]: 2001:4860:7:631::e0:60332 [16/Nov/2022:14:59:33.173] HTTPS_443-Balance~ HTTP_80_443_ipv4/IIS-03 0/0/-1/-1/0 503 2695 - - SC-- 153/147/5/0/0 0/0 "GET https://example.com/path HTTP/2.0" ]

tazemeta
  • 1
  • 1
  • Do you have any upstream proxies like cloudflare that are altering the DNS records being presented to the end users? – Ferrybig Nov 23 '22 at 09:47
  • Yes, we have cloudflare. – tazemeta Nov 23 '22 at 10:44
  • Cloudflare provides dual stack servers, so you see both IPv4 and IPv6 in your logs. You need to fix your application (if you have an very expansive tier account, you can turn IPv6 off in cloudflare) – Ferrybig Nov 23 '22 at 10:49
  • unfortunately i cant turn off ipv6. It's not enterprise version. what kind a setting i change on origin side(pfsense + haproxy) ? – tazemeta Nov 23 '22 at 12:33
  • 1
    See https://serverfault.com/a/784519/151073, it is a problem with your upstream backend server. A common cause is that it gets the IP address in an Forwarded-For header, but then crashes as it logging it to a database because the field is too small – Ferrybig Nov 23 '22 at 12:46

1 Answers1

0

A source IP address is in the logs, your haproxy got something via IPv6. So your CDN routing to you is working.

As mentioned in the comments, for haproxy "SC" termination state relates to a TCP session unexpectedly aborted by the server. Making your back ends IPv6 capable will make this work, without any changes to your CDN or load balancer. Or at least tolerant of IPv6 addresses in http headers, assuming haproxy is in http mode.

Regarding who and what client is suddenly IPv6 capable, researching has the prefix is sometimes enlightening. The last digit group of that IPv6 address is too long, but the prefix is managed by Google, apparently for Google Chrome Prefetch Proxy. Google and Cloudflare turn on IPv6 for their users, so yes you get IPv6 addresses.

John Mahowald
  • 32,050
  • 2
  • 19
  • 34
  • I've noticed something. When i was checking access logs, i've seen some successful requests with ipv6 ips. After that i checked server access logs then i've seen their "network forwarded ip" fields are ipv6 but "source.ip" fields are ipv4. So if my backends are not ipv6 capable, shouldn't these requests also fail? I'm still searchnig why tcp sessin is abortng. – tazemeta Nov 24 '22 at 08:35