HttpClient Failing in asp.net web api

Question

I have a very strange scenario I keep experiencing on one of my systems.

I have an asp.net web API hosted in IIS. Web requests come into it from mobile devices, and then it posts HTTP requests to another server that executes the specific job and then returns the result. (see flow below)

Phone => API => HTTP => ( Service Executor ) => HTTP => API => PHONE

This works perfectly 95% of the time. However it sometimes gets into this state where NONE of the http requests complete.

I have observed the following:

• After the issues happens once, it will continue to fail indefinitely until i restart the application pool.
• Setting the IIS application pool recycle time to 1 hour has significantly reduced the occurrence. However not totally removed it.

The following method is called from my Asp.net Controller:

Here is my code:

public static async Task<string> MakeRequest(string deviceId, BaseCommand baseCommand)
{
    using (var client = new HttpClient())
    {
        // Setup Client                
        client.BaseAddress = await GetBaseAddressForDevice(deviceId);
        client.DefaultRequestHeaders.Accept.Clear();
        //Only wait 90 seconds for responses
        client.Timeout = new TimeSpan(0, 0, 0, 90);

        var requestId = Guid.NewGuid();

        Debug.WriteLine(">Posting  RequestId: " + requestId + " commandId:" + ((int)(baseCommand.CommandId)).ToString(CultureInfo.InvariantCulture) + " data: " + JsonConvert.SerializeObject(baseCommand));
        //

        // Make request
        HttpResponseMessage response = null;
        string responseContent = null;
        try
        {
            response = await client.PostAsJsonAsync(((int)baseCommand.CommandId).ToString(CultureInfo.InvariantCulture), baseCommand);
            // read response content
            responseContent = await response.Content.ReadAsStringAsync();
        }
        catch (Exception ex)
        {
            //TODO: Log excepton
            throw ex;
        }

        Debug.WriteLine(">Response  RequestId: " + requestId + " data: " + JsonConvert.SerializeObject(baseCommand));
        //

        if (response.IsSuccessStatusCode)
        {
            return responseContent;
        }
        HandleErrorResponse(response, responseContent);
        //Wont hit here as an exception will be thrown
        return null;
    }
}

The above example is designed to create and dispose of an HttpClient instance each time. I believe that HttpClient is thread safe and I have tested before with a single instance of HttpClient, and I had the same issues.

Further Behavior that I have observed.

I added logging on the server that Receives the http posts. When this phenomenon happens, it does receive the HTTP Post, however it then immediately fails. Almost like it establishes the connection, and then fails.

I have some exceptions which Application Insights has collected:

 - System.Net.Sockets.SocketException
An error occurred while sending the request.The underlying connection was closed: An unexpected error occurred on a receive.Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.An existing connection was forcibly closed by the remote host

I would really love some input on the issue, maybe some further suggestions to try and debug / location this issue further.