0

I have a long list of API URL calls in urlList each of them returning quite an amount of JSON data. Some of these take milliseconds, others may well take several minutes to complete. That's why I set the timeout to 30 minutes. For some reason the HttpClient instance times out after just 5 minutes and I have no idea why. When I open the same URLs in a browser window or in Postman it takes between 8 and 20 minutes, but they will eventually load while my C# code (see below) throws these exceptions:

For some of the URLs I then get a CanceledException: Task CanceledException - System.Threading.Tasks.TaskCanceledException: A task was canceled. at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at ...

private async Task<string> ProcessUrls(urlList)
{
    string userAndPasswordToken = Convert.ToBase64String(Encoding.UTF8.GetBytes(username + ":" + password);
    ServicePointManager.ServerCertificateValidationCallback += this.ValidateRemoteCertificate;
    ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
    this.httpclient.DefaultRequestHeaders.Accept.Clear();
    this.httpclient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
    this.httpclient.Timeout = TimeSpan.FromMinutes(30);
    this.httpclient.DefaultRequestHeaders.TryAddWithoutValidation("Authorization",$"Basic {userAndPasswordToken}");

    SemaphoreSlim throttler = new SemaphoreSlim(initialCount: 20);
    List<Task> requests = new List<Task>();
    foreach (var url in urlList)
    {
        await throttler.WaitAsync();
        requests.Add(
            Task.Run(async () =>
            {
                try
                {
                    result = await this.httpclient.GetAsync(url);
                }
                finally
                {
                    throttler.Release();
                }
            }
            )
        );
    }
}

UPDATE: I tried the same thing without the Semaphore, still, the same problem occurs - So I guess I can rule out that the Semaphore causes the issue.

What am I missing here? (Note: The APIs I am accessing are not under my control)

This is a .NET project (targetFramework net452), currently running in VS2019 on Win10.

Aileron79
  • 523
  • 5
  • 27
  • Are you in full .net or .net core and what OS/platform? – zaitsman Jul 31 '20 at 06:48
  • 1
    A 30-minute transaction in HTTP is abusing the protocol. There are routers and proxies and things that are in your network path that may react badly. Consider [chunking your data](https://stackoverflow.com/a/11725261/2791540) – John Wu Jul 31 '20 at 06:48
  • @zaitsman: .NET, targetFramework net452. Currently running in VS2019 on Win10. Will add that information. – Aileron79 Jul 31 '20 at 07:01
  • @John: I am aware of this but as I mentioned, the API is not under my control and these requests unfortunately cannot be chunked. But as the requests work in the browser/postman I assume it's not a problem with the network path. – Aileron79 Jul 31 '20 at 07:01
  • 1
    Check the server part. In many servers, do not allow over than 5 mins connection without data transfer. If you cannot control the server, you have to ping some data to preserve the connection. –  Jul 31 '20 at 08:16
  • @donggas90: Thanks a lot, this actually is what causes the problem. In Wireshark I can clearly see Keep-Alive pings being sent from Chrome, but not from my .net app. Funnily enough, Postman (Chrome-Plugin) sends the Keep-Alive, thus it works. Postman (Standalone) does not - therefore it fails. However, so far I have not found a solution for sending Keep-Alive packets through an HttpClient connection. Any suggestions? – Aileron79 Jul 31 '20 at 12:58

0 Answers0