Different TCP MSS values for different providers with same MTU

Question

I'm having an issue in internet connectivity with a particular DSL provider, and after lots of research it has boiled down to the following.

Let's say there are two providers A and B connected through PPPoE (on Ubuntu 14.04, identical configurations options, same standard MTU 1492)

Provider: A

ppp0      Link encap:Point-to-Point Protocol
          inet addr:192.168.100.1  P-t-P:192.168.100.2  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1
          RX packets:422406 errors:0 dropped:0 overruns:0 frame:0
          TX packets:383588 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:410702336 (410.7 MB)  TX bytes:76099873 (76.0 MB)

Provider: B

ppp0      Link encap:Point-to-Point Protocol
          inet addr:192.168.200.1  P-t-P:192.168.200.2  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1
          RX packets:3519252 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2547278 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:4506505131 (4.5 GB)  TX bytes:337199879 (337.1 MB)

Here are the tcpdump excerpts from both the providers:

Provider: A (Successful IMAP connection attempt)

10:58:33.085863 IP provider-1.example.com.56892 > imap.example.net.imaps: Flags [SEW], seq 3899194038, win 8192, options [**mss 1452**,nop,wscale 8,nop,nop,sackOK], length 0
10:58:33.133981 IP imap.example.net.imaps > provider-1.example.com.56892: Flags [S.E], seq 2576858288, ack 3899194039, win 29200, options [mss 1452,nop,nop,sackOK,nop,wscale 9], length 0
10:58:33.134472 IP provider-1.example.com.56892 > imap.example.net.imaps: Flags [.], ack 1, win 516, length 0
10:58:33.134989 IP provider-1.example.com.56892 > imap.example.net.imaps: Flags [P.], seq 1:156, ack 1, win 516, length 155
10:58:33.244366 IP imap.example.net.imaps > provider-1.example.com.56892: Flags [.], ack 156, win 60, length 0
10:58:33.252382 IP imap.example.net.imaps > provider-1.example.com.56892: Flags [.], seq 1:1453, ack 156, win 60, length 1452
10:58:33.253614 IP imap.example.net.imaps > provider-1.example.com.56892: Flags [.], seq 1453:2905, ack 156, win 60, length 1452
10:58:33.254268 IP provider-1.example.com.56892 > imap.example.net.imaps: Flags [.], ack 2905, win 516, length 0
10:58:33.255073 IP imap.example.net.imaps > provider-1.example.com.56892: Flags [.], seq 2905:4357, ack 156, win 60, length 1452
10:58:33.255392 IP imap.example.net.imaps > provider-1.example.com.56892: Flags [P.], seq 4357:4996, ack 156, win 60, length 639
10:58:33.255778 IP provider-1.example.com.56892 > imap.example.net.imaps: Flags [.], ack 4996, win 516, length 0
10:58:33.277521 IP provider-1.example.com.56892 > imap.example.net.imaps: Flags [P.], seq 156:282, ack 4996, win 516, length 126
10:58:33.344742 IP imap.example.net.imaps > provider-1.example.com.56892: Flags [P.], seq 4357:4996, ack 156, win 60, length 639
10:58:33.345176 IP provider-1.example.com.56892 > imap.example.net.imaps: Flags [.], ack 4996, win 516, options [nop,nop,sack 1 {4357:4996}], length 0
10:58:33.392306 IP imap.example.net.imaps > provider-1.example.com.56892: Flags [P.], seq 4996:5047, ack 282, win 60, length 51
10:58:33.425148 IP provider-1.example.com.56892 > imap.example.net.imaps: Flags [.], ack 5047, win 515, length 0

Provider: B (Unsuccessful IMAP connection attempt)

10:11:45.242953 IP provider-2.example.com.44741 > imap.example.net.imaps: Flags [SEW], seq 3877989337, win 8192, options [**mss 1460**,nop,wscale 8,nop,nop,sackOK], length 0
10:11:45.264231 IP imap.example.net.imaps > provider-2.example.com.44741: Flags [S.], seq 3355386868, ack 3877989338, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0
10:11:45.264962 IP provider-2.example.com.44741 > imap.example.net.imaps: Flags [.], ack 1, win 513, length 0
10:11:45.271981 IP provider-2.example.com.44741 > imap.example.net.imaps: Flags [P.], seq 1:156, ack 1, win 513, length 155
10:11:45.294039 IP imap.example.net.imaps > provider-2.example.com.44741: Flags [.], ack 156, win 60, length 0
10:11:45.300146 IP imap.example.net.imaps > provider-2.example.com.44741: Flags [P.], seq 4381:4996, ack 156, win 60, length 615
10:11:45.300727 IP provider-2.example.com.44741 > imap.example.net.imaps: Flags [.], ack 1, win 513, options [nop,nop,sack 1 {4381:4996}], length 0
10:11:55.293705 IP imap.example.net.imaps > provider-2.example.com.44741: Flags [F.], seq 4996, ack 156, win 60, length 0
10:11:55.294414 IP provider-2.example.com.44741 > imap.example.net.imaps: Flags [.], ack 1, win 513, options [nop,nop,sack 1 {4381:4996}], length 0

What I have found out is, if I set the value of TCP MSS at 1452 manually, using mss clmaping on the Linux Box for provider B (which is at 1460 can be seen at tcpdump output), then the connection is established successfully.

My questions are, why with one provider tcp is setting the mss value to 1452 (which seems also correct for a PPPoE with MTU 1492) and the other not. The machines are identical and from the same image with same Kernel and same OS versions. Everything is identical as far as I can see. Can the provider control this or where else should I look?

I would really like to know that I have tried everything before changing this manually.

it's always by lowest common, so it could be anything really. vlan, isp, nxlan, gre or any encapsulation — Jacob Evans, Feb 17 '17 at 12:45
Having managed tens of routers from Ukrain to Morocco: *never* trust default MTUs for routers/modems. Everything with copper, I used to downgrade to 1200. Being able to establish two-ways TCP connections doesn't mean you can do it in GRE, in UDP, ... and pointing this out to your provider, they'ld just tell you they're not at fault, ... — SYN, Feb 17 '17 at 13:09
Might be a path MTU black hole. Hard to be certain because it appears something is blocking parts of the _server_ traffic back to you; it would be necessary to do a tcpdump on the server to confirm it. Anyway, it's _someone's_ fault between you and the destination host. See RFC 2923 for background. — Michael Hampton, Feb 17 '17 at 14:25
@MichaelHampton, thanks for your comments. A tcpdump on the server won't be possible for me, as it is another E-Mail provider. I have alsot thought of path MTU. What is making me thinking is, if you look closely at the begenning of connection process, one is starting with tcp mss 1452 the other with 1460 (and failing). Why the tcp behaving so differently? — Diamond, Feb 17 '17 at 14:46
@MichaelHampton, I have read that RFC and couple of other relevant RFCs too, i.e. RFC6691, RFC879,. What is that repeating "Aagain, read the relevant section of RFC 2923"? Does that mean, there is something obvious hint in that RFC what I am unable to understand? What makes you think I have no background idea of PMTU discovery issue? — Diamond, Feb 18 '17 at 20:31

Different TCP MSS values for different providers with same MTU

0 Answers0