1

I'm running http_load on OpenBSD 5.2, to test how good my nginx setup is, and I've noticed that cold runs are much faster than warm runs, and with every run the performance goes down very dramatically (e.g. from 3735 replies per second on a cold run, to 2288, 1804, 1553 on subsequent runs).

I've noticed with netstat -n | wc -l that there are several thousand connections after running http_load, most of which are in the (state) of TIME_WAIT.

It might seem like set timeout tcp.finwait 8 for pf.conf would reduce some timeout value from 45s to 8s, but it doesn't seem to affect these TIME_WAIT connections at all, which still stay in netstat -n for exactly 60s from the time they're created through http_load / nginx.

Is there a way to expire these TIME_WAIT connections much sooner than 60s?

cnst
  • 13,848
  • 9
  • 54
  • 76
  • 1
    You should not even consider this if you don't fully understand [RFC 1337](http://www.faqs.org/rfcs/rfc1337.html) -- TIME-WAIT Assassination Hazards in TCP. You should figure out why this is causing you a problem and fix that rather than trying to change the timing. – David Schwartz Feb 08 '13 at 01:16
  • Yes, that RFC seems very relevant, but how exactly does it apply here? If `http_load` makes 1000 connections per second for 4 seconds, for a total of 3999 connections, then I end up with having 3831 connections in a `TIME_WAIT` `(state)` as per `netstat` (not sure why it's 3831, and not 3999), which obviously slows down various OS lookups and insertions, and also rather obviously slows down the speed at which new connections can be established. – cnst Feb 08 '13 at 03:08
  • If former connections have to be kept in the `TIME_WAIT` `(state)` for 60s after `FIN`, doesn't it imply that a single IP-address is only good for at most 64K requests per minute, or, in other words, only 1000 requests per second? That seems kinda wrong, there's got to be a better way. – cnst Feb 08 '13 at 03:16
  • You should find out *why* it slows those things down and fix that. There's no reason it *should* have those effects. If you need to establish more than 64K outbound TCP connections per minute, you're probably doing something *very* wrong. (FWIW, 30 seconds is generally regarded as safe.) – David Schwartz Feb 08 '13 at 03:19
  • @DavidSchwartz, why do you say "outbound"? Don't you have to `TIME_WAIT` for any connection, outbound or inbound? The `TIME_WAIT` timeout is definitely 60s in OpenBSD, and I haven't yet found a way to change it, which will basically indeed limit the server to processing at most 1000 individual connections per second per IP address. – cnst Feb 08 '13 at 04:25
  • The "1,000 connections per second" is based on each connection tying up a local port. All inbound connections to the same server share the same local port, so they don't tie up a port. The limitation exclusively applies to outbound connections which require a unique combination of local port and local IP address. – David Schwartz Feb 08 '13 at 05:16

1 Answers1

1

You can play with these sysctl variables:

/etc/sysctl.conf

net.inet.tcp.keepinittime = 150
net.inet.tcp.keepidle = 14400
net.inet.tcp.keepintvl = 150