10

On a Debian server with nginx, I get no response from a web server over HTTPS and IPv6. HTTP works fine.

  • netstat reports port 443 listening on the IPv6 address
  • firewall is open, ipv6scanner.com reports port 443 open
  • locally (over terminal) wget and curl receive a correct response, so the nginx configuration is OK
  • no sign of an error from nginx error.log
  • no record in access.log when it fails, so the communication probably is not reaching the web server
  • DNS is fine. Translation works, and the connection does not work even when the IP address is accessed directly

Every attempt to connect from "outside" (meaning outside of the network, from the internet) fails (web browser, telnet, ipv6-test.com, curl...). There is no response at all.

It can be tested on www.ekasparova.eu. I am clueless. What else can I check?

edit:

the output of traceroute6 --mtu www.google.com is as follows:

traceroute to www.google.com (2a00:1450:4014:800::2004), 30 hops max, 65000 byte packets
1  * F=1500 * *
2  * * *
~
30  * * *

So it never reaches the end...

edit2:

My ip6tables-save output (local firewall):

# Generated by ip6tables-save v1.6.0 on Wed Oct 17 06:25:40 2018
*filter
:INPUT DROP [32:9320]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:ufw6-after-forward - [0:0]
:ufw6-after-input - [0:0]
:ufw6-after-logging-forward - [0:0]
:ufw6-after-logging-input - [0:0]
:ufw6-after-logging-output - [0:0]
:ufw6-after-output - [0:0]
:ufw6-before-forward - [0:0]
:ufw6-before-input - [0:0]
:ufw6-before-logging-forward - [0:0]
:ufw6-before-logging-input - [0:0]
:ufw6-before-logging-output - [0:0]
:ufw6-before-output - [0:0]
:ufw6-logging-allow - [0:0]
:ufw6-logging-deny - [0:0]
:ufw6-reject-forward - [0:0]
:ufw6-reject-input - [0:0]
:ufw6-reject-output - [0:0]
:ufw6-skip-to-policy-forward - [0:0]
:ufw6-skip-to-policy-input - [0:0]
:ufw6-skip-to-policy-output - [0:0]
:ufw6-track-forward - [0:0]
:ufw6-track-input - [0:0]
:ufw6-track-output - [0:0]
:ufw6-user-forward - [0:0]
:ufw6-user-input - [0:0]
:ufw6-user-limit - [0:0]
:ufw6-user-limit-accept - [0:0]
:ufw6-user-logging-forward - [0:0]
:ufw6-user-logging-input - [0:0]
:ufw6-user-logging-output - [0:0]
:ufw6-user-output - [0:0]
-A INPUT -j ufw6-before-logging-input
-A INPUT -j ufw6-before-input
-A INPUT -j ufw6-after-input
-A INPUT -j ufw6-after-logging-input
-A INPUT -j ufw6-reject-input
-A INPUT -j ufw6-track-input
-A INPUT -j LOG --log-prefix "[IPTABLES] " --log-tcp-options
-A INPUT -j LOG --log-prefix "[IPTABLES] " --log-tcp-options
-A FORWARD -j ufw6-before-logging-forward
-A FORWARD -j ufw6-before-forward
-A FORWARD -j ufw6-after-forward
-A FORWARD -j ufw6-after-logging-forward
-A FORWARD -j ufw6-reject-forward
-A FORWARD -j ufw6-track-forward
-A FORWARD -j LOG --log-prefix "[IPTABLES] " --log-tcp-options
-A FORWARD -j LOG --log-prefix "[IPTABLES] " --log-tcp-options
-A OUTPUT -j ufw6-before-logging-output
-A OUTPUT -j ufw6-before-output
-A OUTPUT -j ufw6-after-output
-A OUTPUT -j ufw6-after-logging-output
-A OUTPUT -j ufw6-reject-output
-A OUTPUT -j ufw6-track-output
-A ufw6-after-input -p udp -m udp --dport 137 -j ufw6-skip-to-policy-input
-A ufw6-after-input -p udp -m udp --dport 138 -j ufw6-skip-to-policy-input
-A ufw6-after-input -p tcp -m tcp --dport 139 -j ufw6-skip-to-policy-input
-A ufw6-after-input -p tcp -m tcp --dport 445 -j ufw6-skip-to-policy-input
-A ufw6-after-input -p udp -m udp --dport 546 -j ufw6-skip-to-policy-input
-A ufw6-after-input -p udp -m udp --dport 547 -j ufw6-skip-to-policy-input
-A ufw6-after-logging-forward -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw6-after-logging-input -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw6-before-forward -m rt --rt-type 0 -j DROP
-A ufw6-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw6-before-forward -p ipv6-icmp -m icmp6 --icmpv6-type 1 -j ACCEPT
-A ufw6-before-forward -p ipv6-icmp -m icmp6 --icmpv6-type 2 -j ACCEPT
-A ufw6-before-forward -p ipv6-icmp -m icmp6 --icmpv6-type 3 -j ACCEPT
-A ufw6-before-forward -p ipv6-icmp -m icmp6 --icmpv6-type 4 -j ACCEPT
-A ufw6-before-forward -p ipv6-icmp -m icmp6 --icmpv6-type 128 -j ACCEPT
-A ufw6-before-forward -p ipv6-icmp -m icmp6 --icmpv6-type 129 -j ACCEPT
-A ufw6-before-forward -j ufw6-user-forward
-A ufw6-before-input -i lo -j ACCEPT
-A ufw6-before-input -m rt --rt-type 0 -j DROP
-A ufw6-before-input -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw6-before-input -m conntrack --ctstate INVALID -j ufw6-logging-deny
-A ufw6-before-input -m conntrack --ctstate INVALID -j DROP
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 1 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 2 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 3 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 4 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 128 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 129 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 133 -m hl --hl-eq 255 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 134 -m hl --hl-eq 255 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 135 -m hl --hl-eq 255 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 136 -m hl --hl-eq 255 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 141 -m hl --hl-eq 255 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 142 -m hl --hl-eq 255 -j ACCEPT
-A ufw6-before-input -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 130 -j ACCEPT
-A ufw6-before-input -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 131 -j ACCEPT
-A ufw6-before-input -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 132 -j ACCEPT
-A ufw6-before-input -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 143 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 148 -m hl --hl-eq 255 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 149 -m hl --hl-eq 255 -j ACCEPT
-A ufw6-before-input -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 151 -m hl --hl-eq 1 -j ACCEPT
-A ufw6-before-input -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 152 -m hl --hl-eq 1 -j ACCEPT
-A ufw6-before-input -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 153 -m hl --hl-eq 1 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 144 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 145 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 146 -j ACCEPT
-A ufw6-before-input -p ipv6-icmp -m icmp6 --icmpv6-type 147 -j ACCEPT
-A ufw6-before-input -s fe80::/10 -d fe80::/10 -p udp -m udp --sport 547 --dport 546 -j ACCEPT
-A ufw6-before-input -d ff02::fb/128 -p udp -m udp --dport 5353 -j ACCEPT
-A ufw6-before-input -d ff02::f/128 -p udp -m udp --dport 1900 -j ACCEPT
-A ufw6-before-input -j ufw6-user-input
-A ufw6-before-output -o lo -j ACCEPT
-A ufw6-before-output -m rt --rt-type 0 -j DROP
-A ufw6-before-output -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw6-before-output -p ipv6-icmp -m icmp6 --icmpv6-type 1 -j ACCEPT
-A ufw6-before-output -p ipv6-icmp -m icmp6 --icmpv6-type 2 -j ACCEPT
-A ufw6-before-output -p ipv6-icmp -m icmp6 --icmpv6-type 3 -j ACCEPT
-A ufw6-before-output -p ipv6-icmp -m icmp6 --icmpv6-type 4 -j ACCEPT
-A ufw6-before-output -p ipv6-icmp -m icmp6 --icmpv6-type 128 -j ACCEPT
-A ufw6-before-output -p ipv6-icmp -m icmp6 --icmpv6-type 129 -j ACCEPT
-A ufw6-before-output -p ipv6-icmp -m icmp6 --icmpv6-type 133 -m hl --hl-eq 255 -j ACCEPT
-A ufw6-before-output -p ipv6-icmp -m icmp6 --icmpv6-type 136 -m hl --hl-eq 255 -j ACCEPT
-A ufw6-before-output -p ipv6-icmp -m icmp6 --icmpv6-type 135 -m hl --hl-eq 255 -j ACCEPT
-A ufw6-before-output -p ipv6-icmp -m icmp6 --icmpv6-type 134 -m hl --hl-eq 255 -j ACCEPT
-A ufw6-before-output -p ipv6-icmp -m icmp6 --icmpv6-type 141 -m hl --hl-eq 255 -j ACCEPT
-A ufw6-before-output -p ipv6-icmp -m icmp6 --icmpv6-type 142 -m hl --hl-eq 255 -j ACCEPT
-A ufw6-before-output -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 130 -j ACCEPT
-A ufw6-before-output -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 131 -j ACCEPT
-A ufw6-before-output -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 132 -j ACCEPT
-A ufw6-before-output -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 143 -j ACCEPT
-A ufw6-before-output -p ipv6-icmp -m icmp6 --icmpv6-type 148 -m hl --hl-eq 255 -j ACCEPT
-A ufw6-before-output -p ipv6-icmp -m icmp6 --icmpv6-type 149 -m hl --hl-eq 255 -j ACCEPT
-A ufw6-before-output -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 151 -m hl --hl-eq 1 -j ACCEPT
-A ufw6-before-output -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 152 -m hl --hl-eq 1 -j ACCEPT
-A ufw6-before-output -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 153 -m hl --hl-eq 1 -j ACCEPT
-A ufw6-before-output -j ufw6-user-output
-A ufw6-logging-allow -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW ALLOW] "
-A ufw6-logging-deny -m conntrack --ctstate INVALID -m limit --limit 3/min --limit-burst 10 -j RETURN
-A ufw6-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw6-skip-to-policy-forward -j DROP
-A ufw6-skip-to-policy-input -j DROP
-A ufw6-skip-to-policy-output -j ACCEPT
-A ufw6-track-output -p tcp -m conntrack --ctstate NEW -j ACCEPT
-A ufw6-track-output -p udp -m conntrack --ctstate NEW -j ACCEPT
-A ufw6-user-input -p tcp -m tcp --dport 20 -j ACCEPT
-A ufw6-user-input -p tcp -m tcp --dport 21 -j ACCEPT
-A ufw6-user-input -p tcp -m tcp --dport 25 -j ACCEPT
-A ufw6-user-input -p tcp -m tcp --dport 53 -j ACCEPT
-A ufw6-user-input -p tcp -m tcp --dport 80 -j ACCEPT
-A ufw6-user-input -p tcp -m tcp --dport 110 -j ACCEPT
-A ufw6-user-input -p tcp -m tcp --dport 143 -j ACCEPT
-A ufw6-user-input -p tcp -m tcp --dport 587 -j ACCEPT
-A ufw6-user-input -p tcp -m tcp --dport 993 -j ACCEPT
-A ufw6-user-input -p tcp -m tcp --dport 995 -j ACCEPT
-A ufw6-user-input -p tcp -m tcp --dport 8080 -j ACCEPT
-A ufw6-user-input -p tcp -m tcp --dport 8081 -j ACCEPT
-A ufw6-user-input -p tcp -m tcp --dport 10000 -j ACCEPT
-A ufw6-user-input -p udp -m udp --dport 53 -j ACCEPT
-A ufw6-user-input -p tcp -m multiport --dports 29799:29899 -j ACCEPT
-A ufw6-user-input -p udp -m udp --dport 25 -j ACCEPT
-A ufw6-user-input -p tcp -m tcp --dport 8082 -j ACCEPT
-A ufw6-user-limit -m limit --limit 3/min -j LOG --log-prefix "[UFW LIMIT BLOCK] "
-A ufw6-user-limit -j REJECT --reject-with icmp6-port-unreachable
-A ufw6-user-limit-accept -j ACCEPT
COMMIT
# Completed on Wed Oct 17 06:25:40 2018

edit3:

Thanks to everyone's help I was able to convince the datacenter operator that the problem is in their infrastructure. The problem really was in the MTU setting on a virtual router in the path to internet.

Community
  • 1
j.kaspar
  • 211
  • 1
  • 8

2 Answers2

19

You have an MTU problem.

I tested wget -O /dev/null https://www.ekasparova.eu while observing the traffic with tcpdump. This is what I saw:

19:56:57.048361 IP6 2001:db8::1.47386 > 2a04:f310:100:3:f816:3eff:fea3:4553.443: Flags [S], seq 262121609, win 28800, options [mss 1440,sackOK,TS val 298423713 ecr 0,nop,wscale 7], length 0
19:56:57.087457 IP6 2a04:f310:100:3:f816:3eff:fea3:4553.443 > 2001:db8::1.47386: Flags [S.], seq 2396216876, ack 262121610, win 28560, options [mss 1440,sackOK,TS val 82836580 ecr 298423713,nop,wscale 7], length 0
19:56:57.087490 IP6 2001:db8::1.47386 > 2a04:f310:100:3:f816:3eff:fea3:4553.443: Flags [.], ack 1, win 225, options [nop,nop,TS val 298423723 ecr 82836580], length 0
19:56:57.087692 IP6 2001:db8::1.47386 > 2a04:f310:100:3:f816:3eff:fea3:4553.443: Flags [P.], seq 1:322, ack 1, win 225, options [nop,nop,TS val 298423723 ecr 82836580], length 321
19:56:57.126190 IP6 2a04:f310:100:3:f816:3eff:fea3:4553.443 > 2001:db8::1.47386: Flags [.], ack 322, win 232, options [nop,nop,TS val 82836590 ecr 298423723], length 0
19:56:57.141224 IP6 2a04:f310:100:3:f816:3eff:fea3:4553.443 > 2001:db8::1.47386: Flags [P.], seq 2857:3678, ack 322, win 232, options [nop,nop,TS val 82836594 ecr 298423723], length 821
19:56:57.141301 IP6 2001:db8::1.47386 > 2a04:f310:100:3:f816:3eff:fea3:4553.443: Flags [.], ack 1, win 248, options [nop,nop,TS val 298423736 ecr 82836590,nop,nop,sack 1 {2857:3678}], length 0

The first 3 packets is the handshake. Both ends announce mss 1440 which means they are capable of receiving packets with 1440 bytes of TCP payload, counting headers as well it totals to 1500 bytes of IP traffic, which is what Ethernet commonly supports.

The next 2 packets is client hello and acknowledgement it was received by the server.

The final 2 packets is where things get interesting. By default tcpdump shows relative sequence numbers, which in this case make the capture easier to read. In the packet from the server this is the interesting part seq 2857:3678. We see a jump from 1 to 2857 which means there is a gap of 2856 bytes which the client did not yet receive. 2856 bytes corresponds to two packets of 1428 bytes. The difference between 1440 and 1428 is the size of a timestamp option.

So, the server sent the server hello split across 3 packets. But the first two were too large for the network and were not delivered to the client.

In the final packet from client to server we see this sack 1 {2857:3678}. This is a selective acknowledgement sent by the client informing the server that there is a gap in the data it has received this far.

Likely the server keeps sending the two lost packets over and over again. But no matter how many times it retransmits the same two packets they remain too large for the network. And probably a router on the path sends an error message back to the server informing it the packets are too large and need to be retransmitted in smaller packets.

If the server received those error messages, it would retransmit the packets smaller as needed. And it would remember the smaller PMTU such that on subsequent requests it does not have to repeat this discovery step.

A possible explanation for all of this is that you have a misconfigured firewall which drops all of the error messages informing your server it needs to retransmit the data in smaller packets.

kasperd
  • 30,455
  • 17
  • 76
  • 124
  • 1
    Interesting. Thank you! Those error messages - I guess it is ICMP protocol... Is there a way to test it? The firewall on the server, and the firewall between the server and the internet, should be open for all ICMP communication. – j.kaspar Oct 16 '18 at 19:51
  • @j.kaspar What are the firewalls? How are they configured? How did you test them? – Michael Hampton Oct 16 '18 at 19:57
  • @MichaelHampton there is iptables firewall installed directly on the server. The second one belongs to the datacenter and I am able to manage it's rules. I don't really know how to test the ICMP. But the rules on both are set to anywhere<->anywhere allowed for ICMP – j.kaspar Oct 16 '18 at 20:02
  • 1
    @j.kaspar On a Linux server, try `traceroute6 --mtu www.google.com` and look for `F=####` inserted into the output lines, or output lines where no response comes back at all. On second thought, just run it and edit your question with the output. – Michael Hampton Oct 16 '18 at 20:12
  • @MichaelHampton Done. However I am not sure how to interpret the result. Does that mean, that the communication doesn't pass the second hop at all? – j.kaspar Oct 16 '18 at 20:35
  • @j.kaspar Pretty much. It doesn't even pass the first hop! I would say the prime suspect is your datacenter firewall. Of course, if you only have a single host, then it is probably a waste of money anyway. – Michael Hampton Oct 16 '18 at 20:36
1

I agree with @kasperd that it is MTU issue. For example, by default wget -6 -O/dev/null http://www.ekasparova.eu wouldn't work (it would get short redirect to https://www.babysoul.cz/ on same IP, but then it would hang on next bigger packet). Then I force-reduced MSS for your host:

ip -6 ro add 2a04:f310:100:3:f816:3eff:fea3:4553 advmss 1000 via $MY_GW

and after that wget works normally. So, it is MTU issue. Comparing output of mtr -6 -n --psize 1410 www.ekasparova.eu (which works) with mtr -6 -n --psize 1411 www.ekasparova.eu would indicate the problem is either at your host 2a04:f310:100:3:f816:3eff:fea3:4553 or its upstream at 2a04:f310:100::125

What you could do as workaround (aside from contacting your upstream):

Test at which packet size it breaks (ie. wget -6 -O/dev/null http://v6.testmyipv6.com/MTUtest/1500.dat probably won't work for you while it should, but wget -6 -O/dev/null http://v6.testmyipv6.com/MTUtest/1000.dat will work just fine), and then either:

  • (worse) clamp your MSS for default IPv6 route (as I did above). Note that is will work only for TCP; for example UDP DNS packets will still be broken, or
  • (better) reduce your interface MTU (for example ifconfig eth0 mtu 1200). This should work for all packets. Problem is that if something on the way has even lower MTU, you won't be able to communicate with them. And lowering MTU will result in somewhat lower performance (not that a big deal unless you are a big site usually)
  • (best) try if removing IPv6 firewall (yours, and at your upsteam) helps; and when you find out it does, try to put it back together step by step without breaking PMTU discovery, until you find problemating line. Problem is it requires more work and cooperation from your ISP (and opening firewall might make you vulnerable for the time).
Matija Nalis
  • 2,478
  • 24
  • 37
  • Reducing MSS on the default route is not a bad suggestion. It's something I have been doing in production environments to work around MTU problems in other people's networks. I don't go as low as 1000 though. 1220 is small enough to keep the full packet within the 1280 bytes which IPv6 guarantees to work end-to-end. However the problem in question wouldn't have been mitigated by reducing MSS on the default route as it only affects packet sizes in one direction. – kasperd Oct 17 '18 at 11:19
  • One can mangle the MSS in transit (should be possible to do with `ip6tables`). You are not supposed to do that, but it turns out that clamping the MSS in transit to a maximum of 1220 is a very efficient workaround for MTU problems. And it can be done at either endpoint or any inbetween router, and it will mitigate the MTU problems for TCP in both directions. – kasperd Oct 17 '18 at 11:22
  • Lowering the MTU on an endpoint will influence the outgoing MSS and it will also limit packets you send even if you received a higher MSS. Thus lower MTU at an endpoint can mitigate the MTU issues for TCP in both directions. However it only helps for UDP in one direction. And reducing the MTU on routers inbetween can mitigate the MTU problem or introduce new MTU problems depending on circumstances. So reducing MSS is the one mitigation which can help without potentially introducing new MTU problems. – kasperd Oct 17 '18 at 11:25
  • @kasperd shouldn't lowering the MSS on one side work for traffic flowing in both ways, as it is negotiated (as a lower MSS) for whole TCP session during 3way handshake? As for lowering MTU and it breaking incoming UDPs, while it is true it won't solve the problem, it shouldn't create additional problems as those too-big UDPs wouldn't work anyway (as his broken upstream would drop them anyway). – Matija Nalis Oct 17 '18 at 11:47
  • 1
    No. MSS is negotiated independently for the two directions. Each side know the maximum they are willing to send and tell the other end the maximum they are willing to receive. When you set `advmss` you only influence the size of segments you are going to be receiving not segments you are going to be sending. The maximum you are willing to send is not communicated - because there is no need for that. Both derive their default value from the MTU, one of the two can be overridden with `advmss`. I don't know a way for a routing entry to override the other, but if there is a way, I'd like to learn. – kasperd Oct 17 '18 at 12:15