SocketException existing connection forcibly closed with Flurl

Question

I'm attempting to query the public NPPES NPI registry using Flurl as my HTTP client library, version 2.4.2.

After 4-5 successful async requests (debugging by stepping through each request in the loop) it always fails with a SocketException that the connection was forcibly closed. I'm assuming the API is rate limited, but slowing down the requests doesn't seem to be working. Here is my relevant code.

static async Task<Result> GetNpiEntry(string npiNumber)
{
    var npiEntry = await "https://npiregistry.cms.hhs.gov"
        .AppendPathSegment("api/")
        .SetQueryParams(new { version = "2.1", number = npiNumber }).GetJsonAsync<RootObject>();

    return npiEntry.results[0];
}

and the loop calling it with a hefty sleep between requests.

List<Result> npiResults = new List<Result>(npiTable.Rows.Count);

foreach (DataRow row in npiTable.Rows)
{
    Result npiEntry = Task.Run(() => GetNpiEntry((string)row[0])).Result;
    npiResults.Add(npiEntry);
    Thread.Sleep(2000);
}

Here's the actual exception.

 ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.  

Is there some more appropriate way for me to debug or do this? I imagine I'd need to rate limit the client, but shouldn't a generous wait time between requests work at least for debugging?

Answer 1

You're not awaiting the call to GetNpiEntry in the Task.Run lambda. If the call to GetNpiEntry executed synchronously then you have no problems. If the call to GetNpiEntry executes asynchronously then the Task.Run lambda will not await its result.

Try the following instead:

foreach (DataRow row in npiTable.Rows)
{
    Result npiEntry = Task.Run( async () => await GetNpiEntry((string)row[0])).GetAwaiter( ).GetResult( );
    npiResults.Add(npiEntry);
    Thread.Sleep(2000);
}

I also see you're using .NetCore so you should be able to use the following main instead.

static async Task Main( string[ ] args )

This allows you to use await within Main .

Answer 2

The problem is not Flurl or API throttling, but blocking on an async call (as @WBuck said).

The best practice is to avoid .Result and even .GetAwaiter().GetResult(). Those and Task.Run() are all examples of mixing sync and async code. There are good articles online for why this is bad - search for .net mix sync async for more background info.

The correct solution is to use async "all the way down". Pretty much every entry point can be marked async now, including console apps.

This works on my machine even without the sleeps:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Flurl;
using Flurl.Http;

namespace SO_59567958
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var ids = new[] { "1023086709", "1659325215", "1912946427", "1740219450", "1750497640", "1538260823", "1275625626", "1144303488", "1205919107", "1730281890", "1568453561" };
            Console.WriteLine($"Retrieving {ids.Length} NPI records...");

            var npiResults = new List<Result>();
            foreach (var id in ids)
            {
                var retrievedFromApi = await GetNpiEntry(id);
                npiResults.Add(retrievedFromApi);
            }

            Console.WriteLine("Done");
            npiResults.ForEach(x => Console.WriteLine(x.Number));
        }

        static async Task<Result> GetNpiEntry(string npiNumber)
        {
            var npiEntry = await "https://npiregistry.cms.hhs.gov"
                .AppendPathSegment("api/")
                .SetQueryParams(new { version = "2.1", number = npiNumber })
                .GetJsonAsync<RootObject>();

            return npiEntry.Results[0];
        }
    }

    public class RootObject
    {
        public int ResultCount { get; set; }

        public IReadOnlyList<Result> Results { get; set; }
    }

    public class Result
    {
        public int Number { get; set; }
    }
}