Mac OSX 10.9 listen backlog works not properly

Question

I Have been working on some server-client code recently, which I found a very confusing problem.I have a server listen on port and set backlog = 2, and my client create 5 thread to connect.

In man, I've noticed that

The backlog parameter defines the maximum length for the queue of pending
 connections.  If a connection request arrives with the queue full, the
 client may receive an error with an indication of ECONNREFUSED.  Alterna-
 tively, if the underlying protocol supports retransmission, the request
 may be ignored so that retries may succeed.

which means my client connect would fail or retry later.

But when my client running, it simply get a SIGPIPE signal and failed.

So I run sudo tcpdump -ilo0 port 10000 and get result:

summertekiMacBook-Pro:select summer$ sudo tcpdump -ilo0 port 10000

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lo0, link-type NULL (BSD loopback), capture size 65535 bytes

10:29:16.396240 IP localhost.56347 > localhost.ndmp: Flags [S], seq 3366561899, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 396158772 ecr 0,sackOK,eol], length 0

10:29:16.396241 IP localhost.56349 > localhost.ndmp: Flags [S], seq 902832276, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 396158772 ecr 0,sackOK,eol], length 0

10:29:16.396242 IP localhost.56351 > localhost.ndmp: Flags [S], seq 1956535575, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 396158772 ecr 0,sackOK,eol], length 0

10:29:16.396244 IP localhost.56348 > localhost.ndmp: Flags [S], seq 2161003109, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 396158772 ecr 0,sackOK,eol], length 0

10:29:16.396246 IP localhost.56350 > localhost.ndmp: Flags [S], seq 1318035540, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 396158772 ecr 0,sackOK,eol], length 0

10:29:16.396296 IP localhost.ndmp > localhost.56347: Flags [S.], seq 2871094527, ack 3366561900, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 396158772 ecr 396158772,sackOK,eol], length 0

10:29:16.396307 IP localhost.ndmp > localhost.56351: Flags [S.], seq 3931313020, ack 1956535576, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 396158772 ecr 396158772,sackOK,eol], length 0

10:29:16.396332 IP localhost.ndmp > localhost.56349: Flags [S.], seq 3467781056, ack 902832277, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 396158772 ecr 396158772,sackOK,eol], length 0

10:29:16.396349 IP localhost.ndmp > localhost.56348: Flags [S.], seq 2666080832, ack 2161003110, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 396158772 ecr 396158772,sackOK,eol], length 0

10:29:16.396366 IP localhost.ndmp > localhost.56350: Flags [S.], seq 2467582351, ack 1318035541, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 396158772 ecr 396158772,sackOK,eol], length 0

10:29:16.396375 IP localhost.56347 > localhost.ndmp: Flags [.], ack 1, win 9186, options [nop,nop,TS val 396158772 ecr 396158772], length 0

10:29:16.396381 IP localhost.56351 > localhost.ndmp: Flags [.], ack 1, win 9186, options [nop,nop,TS val 396158772 ecr 396158772], length 0

10:29:16.396386 IP localhost.56349 > localhost.ndmp: Flags [.], ack 1, win 9186, options [nop,nop,TS val 396158772 ecr 396158772], length 0

10:29:16.396391 IP localhost.56348 > localhost.ndmp: Flags [.], ack 1, win 9186, options [nop,nop,TS val 396158772 ecr 396158772], length 0

10:29:16.396398 IP localhost.56350 > localhost.ndmp: Flags [.], ack 1, win 9186, options [nop,nop,TS val 396158772 ecr 396158772], length 0

10:29:16.396408 IP localhost.ndmp > localhost.56347: Flags [R], seq 2871094528, win 0, length 0

10:29:16.396413 IP localhost.ndmp > localhost.56351: Flags [R], seq 3931313021, win 0, length 0

10:29:16.396419 IP localhost.56347 > localhost.ndmp: Flags [P.], seq 1:1001, ack 1, win 9186, options [nop,nop,TS val 396158772 ecr 396158772], length 1000

10:29:16.396424 IP localhost.56351 > localhost.ndmp: Flags [P.], seq 1:1001, ack 1, win 9186, options [nop,nop,TS val 396158772 ecr 396158772], length 1000

10:29:16.396429 IP localhost.ndmp > localhost.56349: Flags [.], ack 1, win 9186, options [nop,nop,TS val 396158772 ecr 396158772], length 0 10:29:16.396435 IP localhost.ndmp > localhost.56348: Flags [R], seq 2666080833, win 0, length 0

10:29:16.396441 IP localhost.ndmp > localhost.56350: Flags [.], ack 1, win 9186, options [nop,nop,TS val 396158772 ecr 396158772], length 0

10:29:16.396454 IP localhost.ndmp > localhost.56347: Flags [R], seq 2871094528, win 0, length 0

10:29:16.396460 IP localhost.ndmp > localhost.56351: Flags [R], seq 3931313021, win 0, length 0

From unix network programming, connect() will start 3-times hand shake routine, and return when server send syn && ack.

From tcpdump output, top 10 lines told that server reply syn & sck although backlog is 2. Later, client send last-time ack and server return rst.

In my opinion, connect return with value != -1 means connection is established & client is able to send data. But log shows above shows it doesn't work that way.

So can anyone tell me which is correct?

Answer 1

Finally, after spending time googling, I've got an explanation of behavior shows above.

I've checked linux manual and it shows:

Notes

The behavior of the backlog argument on TCP sockets changed with Linux 2.2. Now it specifies the queue length for completely established sockets waiting to be accepted, instead of the number of incomplete connection requests. The maximum length of the queue for incomplete sockets can be set using /proc/sys/net/ipv4/tcp_max_syn_backlog. When syncookies are enabled there is no logical maximum length and this setting is ignored. See tcp(7) for more information.

And also, in freebsd manual (osx's manual doesn't give any details, but since two os are both based on bsd ^.^):

Note that before FreeBSD 4.5 and the introduction of the syncache, the backlog argument also determined the length of the incomplete connection queue, which held TCP sockets in the process of completing TCP's 3-way handshake. These incomplete connections are now held entirely in the syncache, which is unaffected by queue lengths. Inflated backlog values to help handle denial of service attacks are no longer necessary.

In conclusion, it seems like the reason for the change of backlog behavior is to protect service from syn flood attack.

Answer 2

You receive a SIGPIPE signal when you try to send data on a broken/closed pipe. According to your tcpdump data and as you explained:

• All 5 TCP connections are established correctly (with 3-way handshake) from 5 different client ports (56347-56351).
• Then the server sends a Reset in the connections from port 47 and 51. This breaks the pipe.
• Almost at the same time data is being sent ([P], push) from ports 47 and 51 (in the client) to the server. Probably this data already was in the socket send buffer ready to be sent when the kernel decides. Later down the server responds again with Reset [R] but this doesn't change anything.
• Actually the last 4 lines in you tcpdump do not affect this explanation.

connect actually returned OK in your client program. I'm sure because all connections were established OK with 3-way handshake complete. The problem is that after that you immediately try to send data and I bet you are trying to send data more than once in each socket. I assume each thread handles one socket. The second or third (or more) time the thread tries to send data on the socket with port 56347 or 56351, the pipe is broken because of the RESET already received from the server and the OS will send the signal SIGPIPE to the process.

listen backlog seems to work OK in your server. It seems 3 connections were handled correctly and only 2 rejected. Probably the server handled the first connection fast enough to leave place for the next 2 in the backlog. The last two were not able to make it because the backlog is too small (2).