2

I am developing open source socket server library: https://sourceforge.net/projects/socketservers/

And I would to like to add socket reuse feature to this lib. I have implement draft of this feature, but I do not see any benefits in my tests. The client makes 32K connect-disconnects by 8 items to the server and measure the time. But here is no difference between reusing socket and not reusing socket - same time elapsed for this test.

What I am doing wrong in test?

What benefit should server get when reuse sockets, and how to measure this benefit?

vf.
  • 103
  • 2
  • 6

3 Answers3

2

I can explain what happens from an unmanaged point of view and how DisconnectEx() is used, perhaps someone can then map this to the managed scenario.

In unmanaged code you would use DisconnectEx() to reuse a socket for an subsequent AcceptEx() or ConnectEx() call, more likely the former. So you'd initially create x sockets and post your overlapped async accept operations using AcceptEx(). When clients connect to these pending connection you would do your server stuff and then at the end call DisconnectEx() on the socket and post a new AcceptEx() using that socket. This avoids the need to create a new socket at this point and it's thus more efficient for the server. The performance difference is probably pretty small but worth having on heavily loaded servers that are accepting lots of short lived connections.

So I suggest you post some code showing how you're reusing your socket after calling Disconnect(true) on it...

Len Holgate
  • 21,282
  • 4
  • 45
  • 92
  • Thank you. I could upload development version to the sourceforge's SVN, but I use several classes for socket reusing and it require a some time understand the logic. – vf. Nov 09 '10 at 21:46
  • I am agree with your theory. I have implement it in this way (except I do not create sockets before, but reuse created by Accept). But I do not any performance difference in my test. The test clients create 4096 times 8 short lived connection - no difference. – vf. Nov 09 '10 at 21:54
  • Are sure that socket reusing has any performance benefit in real application? – vf. Nov 09 '10 at 21:57
  • How do you reuse the sockets created by accept? How many concurrent connections, how quickly are the connections occurring? How loaded is the CPU? etc. As I said, the perf increase is likely pretty small. In unmanaged code you ARE avoiding some work by reusing the socket and so you should get more other work done because of that. – Len Holgate Nov 10 '10 at 06:47
  • Yes, I understand that it depends on many factors. Is there simple test for this? I have measure (new Socket() & Bind) actions and Disconnect(reuse) in .net application - no difference almost ( ~5% ). – vf. Nov 10 '10 at 10:33
  • What about DisconnectEx, how fast is it vs new socket&bind? – vf. Nov 10 '10 at 10:35
  • Of course there's not a simple test for it. All you can do is profile a server that uses both styles under a variety of loads on a variety of platforms. In general, if written correctly, reusing the socket should work out better in terms of performance I would expect. Also it will depend on how you're shutting down the connection, if you're performing the active close then your disconnect will take the time wait period to complete before the socket can be reused... – Len Holgate Nov 10 '10 at 12:41
  • >active close then your disconnect will take the time wait period to complete before – vf. Nov 10 '10 at 12:46
  • Do you talk about TIME_WAIT? I guess, it is related to {ip:port} tuple not to socket resources? – vf. Nov 10 '10 at 12:49
  • "there's not a simple test for it." It is very sadly, so we have only theory and no real confirmation for all this assumptions. – vf. Nov 10 '10 at 12:51
  • Of course it is possible that I have any bugs in implementations, but if here is no test at all how to validate it? – vf. Nov 10 '10 at 12:53
  • What about that .net Disconnect(for reusing) take same time like new Socket() & Bind() ? – vf. Nov 10 '10 at 12:54
  • My comments re time wait are from the DisconnectEx() msdn page... I think the key point about my "not a simple test for it" is that the testing isn't simple not that you cant test it. – Len Holgate Nov 10 '10 at 13:37
  • msdn: While in the TIME_WAIT state, a socket pair cannot be re-used. – vf. Nov 10 '10 at 14:43
  • Yes, msdn has some confusion. But I believe that the _socket pair_ is IP and port, not resources of socket. – vf. Nov 10 '10 at 14:45
  • Thank you for your comments. I really need a opponent for continue experiments. – vf. Nov 10 '10 at 14:47
  • I agree it's not clear, but "Note The socket level disconnect is subject to the behavior of the underlying transport. For example, a TCP socket may be subject to the TCP TIME_WAIT state, causing the DisconnectEx call to be delayed." implies that the DisconnectEx could be delayed by the time wait. So I think my question about who's doing the active close in your tests is still valid. I don't think I can help further without seeing some code snippets of how you reuse the socket. – Len Holgate Nov 10 '10 at 16:35
  • A client is doing active close. – vf. Nov 10 '10 at 17:53
  • I have upload new version w/ socket reusing to https://sourceforge.net/projects/socketservers/ , you could find socket reusing in BasTcpServer.cs and SocketRecycle.cs files. Thanks. – vf. Nov 12 '10 at 17:42
  • I think you're making things too complex and doing too much work around the socket reuse. It probably defeats the perf advantage of reuse, especially if you reuser locks a collection of reusable sockets as you're adding contention in there which otherwise might not exist. – Len Holgate Nov 12 '10 at 18:32
  • I am using lock-free pool. But it was too complex before I add reuse. So I expect to see difference between complex solution w/o reuse & complex solution w/ reuse. What exactly complex things do you mean - I just add lock-free pool for storing sockets? – vf. Nov 12 '10 at 21:24
  • Have you seen my comment about TCB reusing by a system? Actually the Windows can reuse socket resources itself. – vf. Nov 12 '10 at 21:27
  • Given that socket reuse was added so that when issuing an AcceptEx from an IOCP thread you don't need to do anything 'expensive' (pp 188 Network Programming for MS Windows) and given that DisconnectEx came along after TransmitFile which first had the 'disconnect for reuse' option. I think it's safe to say that the intended usage of disconnected sockets is for subsequent reuse in an AcceptEx that is issued straight after the disconnect completes... So, in summary, I think you're using them wrong, so perhaps that's why you don't get the perf gain you expect. The TCB stuff is not, IMHO relevant. – Len Holgate Nov 12 '10 at 21:54
  • Maybe but not in my case, the test client create-destroy connection w/o delay, so it looks like the server use disconnected socket almost after disconnect(). But can you offer algorithm _for subsequent reuse_? I guess it is impossible in real application. – vf. Nov 13 '10 at 09:01
  • You're trading socket creation time for resource usage. If you reuse straight away this is a worthwhile trade, IMHO, if you add complexity with pooling the sockets for reuse and delay their reuse then you probably may as well not bother unless you can get noticeable perf gains, which, it seems, you cant. Reuse with subsequent accepts is pretty straight forward. – Len Holgate Nov 13 '10 at 09:05
  • I assume you got the TCB info from here http://ssmax.net/archives/916.html or the KB article that it links to (useful stuff by the way). This is at a lower level than we're talking here. Yes there will be a TCB data structure related to the socket that you're reusing but the idea behind reuse is to speed up the allocation of the socket structures and to reduce the time involved in going through the winsock layered providers to allocate a new socket, etc. – Len Holgate Nov 13 '10 at 09:13
  • What is _socket structure_ can you give some ref? I suppose, the TCB is socket structure. – vf. Nov 13 '10 at 09:35
  • I can not imagine _straight way_ (w/o socket pool) for real server application. Ok, connection is closed, what I should to do with disconnected socket? – vf. Nov 13 '10 at 09:38
0

The question is if the OS or the runtime does not perform the reuse automatically when invoking a new socket.
The Socket.Disconnect Method documentation points into this direction:

Closes the socket connection and allows reuse of the socket.
So this seem to be an over-optimization.
weismat
  • 7,195
  • 3
  • 43
  • 58
  • The msdn has no any point about automatically socket reusing in winapi and .net Socket class. – vf. Nov 09 '10 at 12:23
  • Thank you for reply, I do not understand where is point in Socket.Disconnect? I use Socket.Disconnect(true) for socket reusing and Socket.Close when do not reuse socket. – vf. Nov 09 '10 at 12:35
  • For each connection a TCP Control Block (TCB – Data structure using 0.5 KB pagepool and 0.5 KB non-pagepool) is maintained. The TCBs are pre-allocated and stored in a table, to avoid spending time on allocating/deallocating the TCBs every time connections are created/closed. The TCB Table enables reuse/caching of TCBs and improves memory management, but the static size limits how many connections TCP can support simultaneously (Active + TIME_WAIT). – vf. Nov 11 '10 at 21:22
  • I can not find what winsock doing when all TCB are used, but it looks like before winsock reuse socket itself. – vf. Nov 11 '10 at 21:24
0

In case you mean something like SO_REUSEADDR or SO_REUSEPORT:

Socket reuse is essentially important if e.g. your server crashes but there are still connections lingering.

If you restart your server, you'd normally have to wait till the operating system gracefully closed those connections, before you can rebind your socket to that port.

This could mean, that some processes which heavily rely on your server to come to a halt till it has been restarted.

Due to the socket reuse feature, you circumvent this problem.

There might be other uses for this, but I can only think of this one right now. Hope that helped.

lx.
  • 2,317
  • 1
  • 22
  • 32
  • Thank you. No, I mean DisconnectEx with TF_REUSE_SOCKET flag instead closesocket function in win api and then use disconnected socket again. Or I use Socket.Disconnect(true) in .net – vf. Nov 09 '10 at 12:54