0

I have a WCF service hosted in IIS that on request does a request to another server using asynchronous socket requests. The request has to be fast (5 seconds) or it times out. When a lot of users are sending requests at the same time (I guess about 10) then what happens is that socket requests to another server are not handled within those 5 seconds. The logs on another server say that the request is received and handled almost immediately and the response is sent back, but the async event is not fired before 5 - 10 seconds causing the wcf service to have a timeout (note this is not a timeout of the wcf itself, but an internal timeout on a call to another server). We think that that thread pool is either too slow to spawn new threads or has some reasons not to do that, which leads to the situation when there are no threads available to pick up the async receive of the socket. As far as I understand both WCF and async socket handlers are using ThreadPool for that so they really fight for the same resource. Did someone encounter similar situation? What is the best way of handling it?

I see two easy ways out:

1) Limit the number of maximum concurrent requests at the wcf service to 5 or something which may cause it to be slower as some requests will have to wait in the queue.

2) Increase the number of MinSize in the ThreadPool I would be glad for any suggestions on the matter.

P.S. The processor has 8 cores, meaning default minsize of threadpool is 8, so I start seeing the problem when more than 8 concurrent requests are happening.

AlexH
  • 2,650
  • 1
  • 27
  • 35
Ilya Chernomordik
  • 27,817
  • 27
  • 121
  • 207
  • The problem almost certainly stems from the work that you're doing in the ThreadPool. Are you blocking on IO with the "call to another server"? If so, you'll be pointlessly straining the ThreadPool and subject to the latency involved in the ThreadPool's decision to spin up new threads. – spender Dec 06 '13 at 16:24
  • I don't block on the IO itself as it's the async socket, but the wcf thread is blocked and waiting for the results as there is no way to have asynchronous response in wcf – Ilya Chernomordik Dec 06 '13 at 16:27
  • "no way to have asynchronous response"? Are you sure about this? http://msdn.microsoft.com/en-us/library/ms731177.aspx – spender Dec 06 '13 at 16:28
  • Your link is about calling the wcf service asynchronously, I mean I cannot respond asynchronously (like in MVC), I have to respond in a method called by wcf synchronously from a WCF service. P.S. I don't do any wcf calls myself – Ilya Chernomordik Dec 06 '13 at 16:29
  • I'm calling you out on this one. You can write contracts that return `Task` and use async/await in your service method. If you want scalable, it's the only way to go. http://blogs.msdn.com/b/endpoint/archive/2010/11/13/simplified-asynchronous-programming-model-in-wcf-with-async-await.aspx – spender Dec 06 '13 at 16:32
  • I see what you mean, but I cannot change the contract unfortunately... But thanks for the link – Ilya Chernomordik Dec 06 '13 at 16:34
  • How do you handle the async responses? I'm curious about the logs on the other server saying that the requests are handled and responded to immediately but the first server is still timing out. – HasaniH Dec 06 '13 at 16:37
  • I use m_Socket.ReceiveAsync(e) – Ilya Chernomordik Dec 06 '13 at 16:39
  • 1
    It might be better to perform your IO synchronously to avoid using a bogged down ThreadPool. Seeing as you're blocking the ThreadPool while you wait for IO, why not just perform it synchronously anyway? If you've spent the ThreadPool on 8 requests, then there will be no ThreadPool threads available to process the received data callback from the async socket call. It's a kind of slowly relieved deadlock. I'm still convinced that rewriting the contract to allow async/await is best, and what you should fight for. Existing clients won't be aware of the change. – spender Dec 06 '13 at 16:41
  • Is it possible that in handling the data you receive you miss a response say if multiple responses get returned in the same buffer? If you check the wire traffic are the responses missing or just delayed? – HasaniH Dec 06 '13 at 16:47
  • @SpaceghostAli I don't think this is an issue, code can handle multiple or single or partial response in the buffer. – Ilya Chernomordik Dec 09 '13 at 08:21
  • @spender I am on .NET 4, is it available there? As far as I understand it's new in .NET 4.5? – Ilya Chernomordik Dec 09 '13 at 08:24

1 Answers1

2

If you are blocking while waiting for async IO, you are creating a situation that is guaranteed to create ThreadPool thread starvation.

Eight requests come in and use all threads in the ThreadPool. They issue an asynchronous IO request, all is good. IO requests are answered, and async callbacks are queued to the ThreadPool, but now all threads in the TheadPool are tied-up waiting for the operation that we've just queued up to a blocked queue. Now after a while, the ThreadPool notices that the queues aren't shrinking and throws a thread at the problem. Slowly the spare thread in the system relieves the deadlock, maybe the ThreadPool spins up another thread if the queue isn't clearing down quickly enough. The important thing is that it isn't in any kind of rush to deal with this increased workload, and optimistically uses as few thread as it can get away with to clear the backlog. Not good. Loads of latency.

You need to rewrite your service contracts to return Task<T> instead of plain'old T. Your clients won't even notice, but you'll be able to run thousands instead of 8 concurrent clients without the slightest strain (assuming the actual workload is trivial) by adopting async/await.

If this isn't an option, consider performing your IO using synchronous methods, because currently, the thread you're running on and the thread that you're depending on in order to proceed are both running in the same constrained ThreadPool. With sync IO, you'll remove a large part of the problem, although it will still (IMO) be severely hobbled by not writing an async service method.

EDIT

Seeing as you on an older version of .Net, you probably can't use async/await. There are a couple of other options available for writing async services, documented here. IIRC, although you change the contract to implement these methods, the endpoints w.r.t. the client stay the same as long as you follow the conventions laid out in the documentation.

spender
  • 117,338
  • 33
  • 229
  • 351
  • Thank you very much for the answer, unfortunately doing IO synchronously is not an option, as I have one socket and if I will make IO synchronously, this will just be very slow as I will have to do some kind of locking. Regarding another suggestion to return Task why do you say that the client won't notice the difference? This will be a contract change, won't it? I am on .NET 4, is it available there? As far as I understand it's new in .NET 4.5? – Ilya Chernomordik Dec 09 '13 at 08:19