4

Is there a way to double the number of ephemeral ports and work around the 16-bit limit? I have tried creating virtual ethernet interfaces over eth0, and hope that would lift the limit. Although the application is utilizing the new virtual IPs in outbound traffic, it seems still hitting the same ephemeral-port limit. I suppose the virtual ports have 1-to-1 mapping to the ports on the physical interface.

ifconfig eth0:1 10.10.10.210 netmask 255.255.255.192 ifconfig eht0:2 10.10.10.211 netmask 255.255.255.192

Could someone please advise how I could double the total number of ephemeral ports in Linux, without adding an extra NIC?

(FYI, I have tried increasing ulimit/max open file, changing the port range, enabling tcp recycle/timestamps, reducing tcp fin timeout... I suppose we simply need more than 65k ports for this proxy machine.)

tshepang
  • 12,111
  • 21
  • 91
  • 136
unjc
  • 63
  • 1
  • 9
  • Perhaps I should clarify what I need. I am not looking to increase the 16-bit port limit over a single IP. What I really want is to create extra (virtual) IP in order to allow more ports for the proxy (or Linux OS) to use. If that works, the outbound connections will be binding to different IPs. Hope that makes sense. – unjc Sep 10 '13 at 14:34
  • The destination server sees the tcp connections are bound to the virtual IPs. Should OS allow to reuse the same local port as long as the 5-tuple (protocol, source address, source port, destination address, destination port) doesn't exist already? – unjc Sep 10 '13 at 16:10
  • I have the exact issue. After adding 10 virtual IPs, and binding sockets on them, the sockets are still limited to ephemeral port range. I'm not sure if SO_REUSEPORT is required. Have you resolve this issue ? – Ning Sun Jun 20 '14 at 07:42

4 Answers4

3

If you create virtual interfaces over eth0, then you should be able to assign different IP addresses to those interfaces. With that, you can use the same ephemeral port numbers (they are allocated in the kernel, so you dont really have much control) for multiple sockets each bound to different addresses -- you will probably need to set SO_REUSEADDR option. The reason this will work is because for incoming packets (UDP/TCP), the flow is identified by looking at both local source IP and the port number.

And as @Duck mentioned, since TCP/UDP headers allocate only 16 bits for port numbers, there is not much point in increasing the ephemeral range in the local stack.

Manoj Pandey
  • 4,528
  • 1
  • 17
  • 18
  • Thanks Manoj. I am using squid in the proxy. I just check their website; S_REUSEADDR seems to be the default option already. – unjc Sep 10 '13 at 14:48
1

It's a limitation of the network protocols. Both TCP & UDP, for instance, have 16 bit source and destination ports. Even if you could increase the number of ports no one could address them.

Duck
  • 26,924
  • 5
  • 64
  • 92
  • When the socket binds to the virtual IP, the remote machine sees the virtual IP as the connection's originating IP. In theory, we are still sticking to 16-bit port limit but the extra virtual interface should create a new ephemeral port pool to use. Am I correct? – unjc Sep 09 '13 at 16:08
  • You know, I'm not 100% sure. I'll delete my answer in a bit unless I get time to do some research. – Duck Sep 09 '13 at 17:17
0

It turns out that you cannot use 0 for binding ephemeral port if you want to exceed the 65535 limit. Instead, you need to use an explicit port number.

And also turning on tcp_tw_reuse might be helpful: http://krenel.org/tcp-time_wait-and-ephemeral-ports-bad-friends.html

tlwhitec
  • 1,845
  • 16
  • 15
Ning Sun
  • 2,145
  • 1
  • 19
  • 24
0

It seems there's a way, but it's not for free. It's called "bind before connect". See this short but dense article, which sums it up very nicely.

Having multiple virtual IPs is just a start. Quoting the linked article:

On Linux the ephemeral port range is a global resource, it's not a specific setting local to an IP address.

So that's bad and you have to improve your starting position with few right settings (where most of them you already found) and get around the global limit with a clever socket allocation technique. The result is that you'll control all the outgoing IPs manually. This also seems not to cope well with other apps on the system using the traditional "connect" way.

tlwhitec
  • 1,845
  • 16
  • 15