0

I am facing a difficult-to-reproduce problem that I think is related to the use of CancellationToken. I have three async apps (each on a different server) that communicate through ASP.NET Web API as shown in the diagram below

A ---> B ---> C

App A runs a routine from time to time that fires a lot of concurrent requests to app B, which in turn fires another lot of concurrent requests to app C. All these calls are done using HttpClient.SendAsync, and are constrained to a 10 seconds timeout, using a CancellationToken as such:

var cancellationTokenSource = new CancellationTokenSource();
cancellationTokenSource.CancelAfter(TimeSpan.FromSeconds(10));
var token = cancellationTokenSource.Token;

When server C is under heavy load, it starts taking more time to respond, which causes server B to start cancelling requests -- because of the configured Timeout. When this happens, sometimes a request gets "stuck" (see image below).

enter image description here

I am familiarized with C# async best practices, and I am not doing .Result or .Wait() or the like. I am aware that these can cause hangs.

If I pass CancellationToken.None instead, the problem goes away, so my hint is that it has to do with canceling under high concurrency scenario.

When time passes, more and more requests start hanging, and eventually I have no choice but to do an IIS reset to make them go away.

Any clues of what can be going on?

alextercete
  • 4,871
  • 3
  • 22
  • 36

1 Answers1

1

cancellationtoken doesnt dictates cancelation, implementer may inspect it after very long interval, or ignore it completely. the correct way implementing timeout is:

if(yourTask == Task.WhenAny(yourTask, Task.Delay(3000)))
{
   //task completed
}
else
{
  //timeout occured
}
TakeMeAsAGuest
  • 957
  • 6
  • 11
  • I forgot to mention that I am using `HttpClient.SendAsync` to make the calls, so I know for a fact that `CancellationToken` is not ignored completely. I added that to the question. – alextercete Jul 29 '13 at 20:08
  • Also, I think you forgot one `await` there before `Task.WhenAny`. – alextercete Jul 29 '13 at 20:19
  • why you are insisting not to use standart way? i dont think there will be measurable performance difference. – TakeMeAsAGuest Jul 29 '13 at 20:19
  • its hard to answer your question without inspecting even debugging original source code. my guess is it implements cancellation by observing it from another thread. maybe its waiting a free thread from threadpool etc. – TakeMeAsAGuest Jul 29 '13 at 20:22
  • but this way you won't have control over the Task execution. I mean, the Task will continue to run (and so will its side effects). Is there a way to avoid that? – alextercete Jul 30 '13 at 16:42
  • you could combine your approach (cancelationtoken) and check task state if its canceled if not timed out. you didnt have control over task if it hangs too, so there is no easy way to make sure the destiny of it. if its critical process, you could implement something according to your problem (flag the database record etc.) – TakeMeAsAGuest Jul 30 '13 at 20:21
  • I thought about what you said and it makes sense. I did what you suggested and the problem seems to have gone away. Thanks! – alextercete Aug 01 '13 at 15:50