Where are my TIME_WAITs on Mac OS X?

Question

No TIME_WAITs on Mac OS X

Normally, when a TCP connection is closed, the socket on the side where close() is called first is left in the TIME_WAIT state.

When one of the peers is a Mac OS X (Lion) machine, no TIME_WAIT is listed by netstat -an on the Mac if close() is called first on the Mac side. However, it seems that the socket is actually in TIME_WAIT state, because trying to call listen() again (without using the socket option SO_REUSEADDR) causes listen() to fail.

Waiting for 2*MSL (Maximum Segment Lifetime which is 15 seconds on Mac OS X Lion as reported by sysctl net.inet.tcp.msl) clears the TIME_WAIT state, and listen() can be called again without error.

Why can't I see the socket in TIME_WAIT?

Testing

Here are two simple test programs in Python.

Server

#!/usr/bin/env python

import socket

HOST = ''
PORT = 50007
l = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
l.bind((HOST, PORT))
l.listen(1)
print("Listening on %d" % PORT)
(s, _) = l.accept()
print("Connected")
raw_input("Press <enter> to close...")
l.close()
s.close()
print("Closed")

Client

#!/usr/bin/env python

import socket
import sys

HOST = sys.argv[1]
PORT = 50007

print("Opening connection to server")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
raw_input("Press <enter> to close...")
s.close()
print("Closed")

When running both the server and the client on two different Linux machines, the peer that press <enter> to call close() first gets a TIME_WAIT as expected:

$ ./server-timewait.py 
Listening on 50007
Connected
Press <enter> to close...
Closed
$ netstat -an | grep 50007
tcp        0      0 172.16.185.219:50007    172.16.185.42:49818     TIME_WAIT  
$ 

When one of the peers is a Mac (running OS X Lion) I never see a TIME_WAIT when running netstat -an | grep 50007 after closing first on the Mac.

Answer 1

This bug report claims that the problem is in the netstat implementation. The code attached to the bug report shows sockets in TIME_WAIT state correctly. You need to remove the following lines

if (lip == INADDR_LOCALHOST ||
  lip == INADDR_ANY
  ) { continue; }

to make it show sockets bound to localhost.

Answer 2

This is not an answer, but someone maybe able to dig out more from this.

tcpdump -i lo0 -vv port 50007

## Press Enter at the server window

# Server send a FIN (note the flag)
23:33:04.283768 IP (tos 0x0, ttl 64, id 4134, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->2c9c)!)
    localhost.50007 > localhost.56030: Flags [F.], cksum 0xfe28 (incorrect -> 0xeff9), seq 1, ack 1, win 9186, options [nop,nop,TS val 432165676 ecr 432157913], length 0

# Client send back ACK
23:33:04.283803 IP (tos 0x0, ttl 64, id 44906, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->8d57)!)
    localhost.56030 > localhost.50007: Flags [.], cksum 0xfe28 (incorrect -> 0xd1a6), seq 1, ack 2, win 9186, options [nop,nop,TS val 432165676 ecr 432165676], length 0

# Server confirm the ACK is received
23:33:04.283812 IP (tos 0x0, ttl 64, id 18284, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->f555)!)
    localhost.50007 > localhost.56030: Flags [.], cksum 0xfe28 (incorrect -> 0xd1a6), seq 2, ack 1, win 9186, options [nop,nop,TS val 432165676 ecr 432165676], length 0

## After this point, the server process is actually exit but client still running.
## It's strange that re-run server script gives "OSError: [Errno 48] Address already in use"
## and netstat shows this connection is in CLOSE_WAIT status

## Press Enter at the client window

# Client send a FIN to server
23:33:09.731728 IP (tos 0x0, ttl 64, id 51478, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->73ab)!)
    localhost.56030 > localhost.50007: Flags [F.], cksum 0xfe28 (incorrect -> 0xbcb6), seq 1, ack 2, win 9186, options [nop,nop,TS val 432171035 ecr 432165676], length 0

# WTH!? Who send back this packet? The server process is closed!
23:33:09.731764 IP (tos 0x0, ttl 64, id 18754, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->f37f)!)
    localhost.50007 > localhost.56030: Flags [.], cksum 0xfe28 (incorrect -> 0xa7c7), seq 2, ack 2, win 9186, options [nop,nop,TS val 432171035 ecr 432171035], length 0