1

We are having some network timeout issues on a Debian server that is quite busy, and maintains multiple connections to a number of other servers on the network.

Here are our current TCP keepalive settings in sysctl.conf:

net.ipv4.tcp_keepalive_time=60
net.ipv4.tcp_keepalive_intvl=90
net.ipv4.tcp_keepalive_probes=3

Perhaps there is a problem with these.

What keepalive settings are recommended for a busy server?

UpTheCreek
  • 1,628
  • 10
  • 32
  • 48

2 Answers2

2

so tcp keepalive is different then something like nginx/apache keepalive.

tcp keepalive keeps the connection open in case an error has happened. Like the client didn't get the request so it can re-try it over the same connection. Now that rarely happens and general rule of thumb is you want to keep a high tcp keepalive on a NAT server so it doesn't lose the mapping from client to NATed server behind it. We run Ad servers that hand millions somewhere around 40million connections per day per server and our keepalive looks like 

"net.ipv4.tcp_keepalive_intvl" => 2,
"net.ipv4.tcp_keepalive_probes" => 3,
"net.ipv4.tcp_keepalive_time" => 5,

I still feel 5 seconds for keepalive time is too high and given the nature of our business where if we don't return an ad in 50ms then the client timesout. So I'll probably drop that to 1. I've just been lowering that value slowly so I don't cause any major issues. I'd not recommend the same since all use cases are different.

So as I said its very different then nginx/apache keepalive. That is more persistent connections. So it can connect once and re-use that connection again. That will help reduce the latency between client and host.

Chances are if you aren't running out of tcp ports then changing your tcp keepalive won't change anything you're seeing with timeouts.

vanthome
  • 730
  • 5
  • 13
Mike
  • 22,310
  • 7
  • 56
  • 79
1

What kind of network timeouts do you have? TCP keep alive will not help if the server is to busy to respond in time. It will only help to detect when the TCP connection is no longer alive because the peer crashed or some packet filter in between closed states because of inactivity of the connection.

Steffen Ullrich
  • 13,227
  • 27
  • 39
  • Only timeout I can see is /proc/sys/net/ipv4/tcp_fin_timeout which is 60. Is that what you meant? (Just realised you might be talking about the timeout errors we have been getting - will try to get these for you) – UpTheCreek Nov 03 '14 at 13:20
  • At the application level these errors are 'read ETIMEDOUT', but I'm not sure how I'd give you any lower-level info than that. – UpTheCreek Nov 03 '14 at 13:26
  • Without getting more specific information I can just guess, that you are talking about a web server which is too busy to respond in time to the clients so they give up. In this case no keep alive tuning will help, you need to find out why the server is responding so slowly. Or you need to adjust how long the clients are willing to wait, which is an issue in the client application. If you need more detailed tips you should describe what kind of applications are in use, how they communicate and where to you get the timeouts. – Steffen Ullrich Nov 03 '14 at 14:35
  • Specifically I'm seeing this from an app server communicating with a redis server. Neither server is stressed or responding slowly though. – UpTheCreek Nov 03 '14 at 14:40
  • Hard to say what's exactly going on, but since you consider the system as busy redis might just fight with the rest of the system to get I/O and thus responds slowly unless the data it needs are already in memory. And I don't think that the app has a long timeout because one does not expect a database to be slow. – Steffen Ullrich Nov 03 '14 at 15:36
  • Redis holds all of its data in memory (we don't even have disk snapshotting enabled). Average response times from both servers are in the milliseconds. Yeah, there must be something odd going on, but not sure what yet. – UpTheCreek Nov 03 '14 at 15:49
  • Does redis write log files to disk? If this is the case the I/O contention could be there. – Steffen Ullrich Nov 03 '14 at 16:28