At home (79.1.1.156), my ISP doesn’t provide IPv6 so I’m connecting my server to an external server (95.1.1.126) that has IPv6 access via a GRE tunnel, with this simple setup:
ip tunnel add gretun mode gre ttl 255 local 79.1.1.156 remote 95.1.1.126 pmtudisc
ip link set dev gretun up
ip -6 addr add fd12:3456:789a:1::2 dev gretun
ip -6 route add fd12:3456:789a:1::1/128 dev gretun metric 1
I assign the home server the IP6 address fd12:3456:789a:1::2 and the external server fd12:3456:789a:1::1. The external server has the same configuration swapping IPs.
Pinging from one server to the other works perfectly. The problem arises with TCP, it looks like the “2nd ACK” (3rd step of TCP 3 way handshake) is dropped and never reaches destination.
On the server (95.1.1.126) I run:
socat - tcp6-listen:3000
Then on the client (79.1.1.156) I run:
telnet fd12:3456:789a:1::1 3000
Trying fd12:3456:789a:1::1...
Connected to fd12:3456:789a:1::1.
Escape character is '^]'.
aaaaa
aaaaa
^]quit
From the client’s side I see (I've added >> on the ACK packets that are missing on the server side, notice how the server keeps sending SYN-ACK's):
02:46:42.063091 IP 79.1.1.156 > 95.1.1.126: GREv0, length 84: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [S], seq 2473235054, win 28160, options [mss 1408,sackOK,TS val 1651249830 ecr 0,nop,wscale 7], length 0
02:46:42.129852 IP 95.1.1.126 > 79.1.1.156: GREv0, length 84: IP6 fd12:3456:789a:1::1.3000 > fd12:3456:789a:1::2.43704: Flags [S.], seq 3770099713, ack 2473235055, win 28080, options [mss 1416,sackOK,TS val 2610159587 ecr 1651249830,nop,wscale 7], length 0
>> 02:46:42.129897 IP 79.1.1.156 > 95.1.1.126: GREv0, length 76: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [.], ack 1, win 220, options [nop,nop,TS val 1651249897 ecr 2610159587], length 0
02:46:43.131703 IP 95.1.1.126 > 79.1.1.156: GREv0, length 84: IP6 fd12:3456:789a:1::1.3000 > fd12:3456:789a:1::2.43704: Flags [S.], seq 3770099713, ack 2473235055, win 28080, options [mss 1416,sackOK,TS val 2610160589 ecr 1651249830,nop,wscale 7], length 0
>> 02:46:43.131835 IP 79.1.1.156 > 95.1.1.126: GREv0, length 76: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [.], ack 1, win 220, options [nop,nop,TS val 1651250899 ecr 2610159587], length 0
02:46:43.354503 IP 79.1.1.156 > 95.1.1.126: GREv0, length 83: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [P.], seq 1:8, ack 1, win 220, options [nop,nop,TS val 1651251122 ecr 2610159587], length 7
02:46:43.626815 IP 79.1.1.156 > 95.1.1.126: GREv0, length 83: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [P.], seq 1:8, ack 1, win 220, options [nop,nop,TS val 1651251394 ecr 2610159587], length 7
02:46:43.914770 IP 79.1.1.156 > 95.1.1.126: GREv0, length 83: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [P.], seq 1:8, ack 1, win 220, options [nop,nop,TS val 1651251682 ecr 2610159587], length 7
02:46:44.458751 IP 79.1.1.156 > 95.1.1.126: GREv0, length 83: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [P.], seq 1:8, ack 1, win 220, options [nop,nop,TS val 1651252226 ecr 2610159587], length 7
02:46:45.147699 IP 95.1.1.126 > 79.1.1.156: GREv0, length 84: IP6 fd12:3456:789a:1::1.3000 > fd12:3456:789a:1::2.43704: Flags [S.], seq 3770099713, ack 2473235055, win 28080, options [mss 1416,sackOK,TS val 2610162605 ecr 1651249830,nop,wscale 7], length 0
>> 02:46:45.147800 IP 79.1.1.156 > 95.1.1.126: GREv0, length 76: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [.], ack 1, win 220, options [nop,nop,TS val 1651252915 ecr 2610159587], length 0
02:46:45.546762 IP 79.1.1.156 > 95.1.1.126: GREv0, length 83: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [P.], seq 1:8, ack 1, win 220, options [nop,nop,TS val 1651253314 ecr 2610159587], length 7
02:46:47.818801 IP 79.1.1.156 > 95.1.1.126: GREv0, length 83: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [P.], seq 1:8, ack 1, win 220, options [nop,nop,TS val 1651255586 ecr 2610159587], length 7
02:46:49.211726 IP 95.1.1.126 > 79.1.1.156: GREv0, length 84: IP6 fd12:3456:789a:1::1.3000 > fd12:3456:789a:1::2.43704: Flags [S.], seq 3770099713, ack 2473235055, win 28080, options [mss 1416,sackOK,TS val 2610166669 ecr 1651249830,nop,wscale 7], length 0
>> 02:46:49.211807 IP 79.1.1.156 > 95.1.1.126: GREv0, length 76: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [.], ack 1, win 220, options [nop,nop,TS val 1651256979 ecr 2610159587], length 0
02:46:49.367297 IP 79.1.1.156 > 95.1.1.126: GREv0, length 83: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [FP.], seq 8:15, ack 1, win 220, options [nop,nop,TS val 1651257134 ecr 2610159587], length 7
02:46:52.170758 IP 79.1.1.156 > 95.1.1.126: GREv0, length 90: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [FP.], seq 1:15, ack 1, win 220, options [nop,nop,TS val 1651259938 ecr 2610159587], length 14
02:46:52.738281 IP 95.1.1.126 > 79.1.1.156: GREv0, length 76: IP6 fd12:3456:789a:1::1.3000 > fd12:3456:789a:1::2.43704: Flags [F.], seq 1, ack 16, win 228, options [nop,nop,TS val 2610170196 ecr 1651259938], length 0
02:46:52.738361 IP 79.1.1.156 > 95.1.1.126: GREv0, length 76: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [.], ack 2, win 220, options [nop,nop,TS val 1651260506 ecr 2610170196], length 0
While on the server I never get the client’s “ACK” of the TCP 3 way handshake (I've added >> where the "ACK"s should have been received):
11:46:42.095788 IP 79.1.1.156 > 95.1.1.126: GREv0, length 84: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [S], seq 2473235054, win 28160, options [mss 1408,sackOK,TS val 1651249830 ecr 0,nop,wscale 7], length 0
11:46:42.096254 IP 95.1.1.126 > 79.1.1.156: GREv0, length 84: IP6 fd12:3456:789a:1::1.3000 > fd12:3456:789a:1::2.43704: Flags [S.], seq 3770099713, ack 2473235055, win 28080, options [mss 1416,sackOK,TS val 2610159587 ecr 1651249830,nop,wscale 7], length 0
>>
11:46:43.098129 IP 95.1.1.126 > 79.1.1.156: GREv0, length 84: IP6 fd12:3456:789a:1::1.3000 > fd12:3456:789a:1::2.43704: Flags [S.], seq 3770099713, ack 2473235055, win 28080, options [mss 1416,sackOK,TS val 2610160589 ecr 1651249830,nop,wscale 7], length 0
>>
11:46:45.114065 IP 95.1.1.126 > 79.1.1.156: GREv0, length 84: IP6 fd12:3456:789a:1::1.3000 > fd12:3456:789a:1::2.43704: Flags [S.], seq 3770099713, ack 2473235055, win 28080, options [mss 1416,sackOK,TS val 2610162605 ecr 1651249830,nop,wscale 7], length 0
>>
11:46:49.178098 IP 95.1.1.126 > 79.1.1.156: GREv0, length 84: IP6 fd12:3456:789a:1::1.3000 > fd12:3456:789a:1::2.43704: Flags [S.], seq 3770099713, ack 2473235055, win 28080, options [mss 1416,sackOK,TS val 2610166669 ecr 1651249830,nop,wscale 7], length 0
>>
11:46:49.399856 IP 79.1.1.156 > 95.1.1.126: GREv0, length 83: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [FP.], seq 8:15, ack 1, win 220, options [nop,nop,TS val 1651257134 ecr 2610159587], length 7
11:46:49.400046 IP 95.1.1.126 > 79.1.1.156: GREv0, length 88: IP6 fd12:3456:789a:1::1.3000 > fd12:3456:789a:1::2.43704: Flags [.], ack 1, win 228, options [nop,nop,TS val 2610166891 ecr 1651249830,nop,nop,sack 1 {8:16}], length 0
11:46:52.203490 IP 79.1.1.156 > 95.1.1.126: GREv0, length 90: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [FP.], seq 1:15, ack 1, win 220, options [nop,nop,TS val 1651259938 ecr 2610159587], length 14
11:46:52.246072 IP 95.1.1.126 > 79.1.1.156: GREv0, length 76: IP6 fd12:3456:789a:1::1.3000 > fd12:3456:789a:1::2.43704: Flags [.], ack 16, win 228, options [nop,nop,TS val 2610169737 ecr 1651259938], length 0
11:46:52.704683 IP 95.1.1.126 > 79.1.1.156: GREv0, length 76: IP6 fd12:3456:789a:1::1.3000 > fd12:3456:789a:1::2.43704: Flags [F.], seq 1, ack 16, win 228, options [nop,nop,TS val 2610170196 ecr 1651259938], length 0
11:46:52.770338 IP 79.1.1.156 > 95.1.1.126: GREv0, length 76: IP6 fd12:3456:789a:1::2.43704 > fd12:3456:789a:1::1.3000: Flags [.], ack 2, win 220, options [nop,nop,TS val 1651260506 ecr 2610170196], length 0
After some seconds of closing the client’s connection, the server running “socat” prints the information sent from the client and closes the socket. This "some seconds" is the interval between 2:46:49 and 2:46:52 in the client (add +9h for the server timestamp). Looks like despite never getting the ACK from the client, the server still processes the rest of the packets when closing the connection, but this doesn’t work for HTTP connections.
This problem also happens if I invert the client/server roles (run telnet on the remote server and socat at the home server)
This problem doesn’t happen if I use IPv4 addresses inside the GRE tunnel.
I’ve tried both sides running Debian (kernel 4.19) and Ubuntu (kernel 5.0), same problem.
The same configuration works fine between 2 servers directly connected to Internet. The main difference here is that I’m connecting to my ISP via a PPPoE connection that I set up in my home server (79.1.1.156), so we have PPPoE > GRE > IPv6.
