HttpClient and stream issues

Question

A year ago I wrote a cmdlet that handles Multipart/form-data requests and it is leveraging the .net class HttpClient in order to do so. I described it in details here.

In a nutshell this is the core of my cmdlet:

$networkCredential = New-Object -TypeName System.Net.NetworkCredential -ArgumentList @($Credential.UserName, $Credential.Password)
$httpClientHandler = New-Object -TypeName System.Net.Http.HttpClientHandler
$httpClientHandler.Credentials = $networkCredential

$httpClient = New-Object -TypeName System.Net.Http.Httpclient -ArgumentList @($httpClientHandler)

$packageFileStream = New-Object -TypeName System.IO.FileStream -ArgumentList @($packagePath, [System.IO.FileMode]::Open)

$contentDispositionHeaderValue = New-Object -TypeName  System.Net.Http.Headers.ContentDispositionHeaderValue -ArgumentList @("form-data")
$contentDispositionHeaderValue.Name = "fileData"
$contentDispositionHeaderValue.FileName = $fileName

$streamContent = New-Object -TypeName System.Net.Http.StreamContent -ArgumentList @($packageFileStream)
$streamContent.Headers.ContentDisposition = $contentDispositionHeaderValue
$streamContent.Headers.ContentType = New-Object -TypeName System.Net.Http.Headers.MediaTypeHeaderValue -ArgumentList @("application/octet-stream")

$content = New-Object -TypeName System.Net.Http.MultipartFormDataContent
$content.Add($streamContent)

try
{
    $response = $httpClient.PostAsync("$EndpointUrl/package/upload/$fileName", $content).GetAwaiter().GetResult()

    if (!$response.IsSuccessStatusCode)
    {
        $responseBody = $response.Content.ReadAsStringAsync().GetAwaiter().GetResult()
        $errorMessage = "Status code {0}. Reason {1}. Server reported the following message: {2}." -f $response.StatusCode, $response.ReasonPhrase, $responseBody

        throw [System.Net.Http.HttpRequestException] $errorMessage
    }

    return [xml]$response.Content.ReadAsStringAsync().GetAwaiter().GetResult()
}
catch [Exception]
{
    throw
}
finally
{
    if($null -ne $httpClient)
    {
        $httpClient.Dispose()
    }

    if($null -ne $response)
    {
        $response.Dispose()
    }
}

I was using this code in a VSTS Build task for over a year with success. Recently it started intermittently failing. In one run it succeeds then the next one fails and so on. I can't understand why this is the case and I would use some help.

The code fails on PostAsync method invocation and following is the exception I do see:

2017-03-24T15:17:38.4470248Z ##[debug]System.NotSupportedException: The stream does not support concurrent IO read or write operations.
2017-03-24T15:17:38.4626512Z ##[debug]   at System.Net.ConnectStream.InternalWrite(Boolean async, Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
2017-03-24T15:17:38.4626512Z ##[debug]   at System.Net.ConnectStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
2017-03-24T15:17:38.4626512Z ##[debug]   at System.Net.Http.StreamToStreamCopy.TryStartWriteSync(Int32 bytesRead)
2017-03-24T15:17:38.4626512Z ##[debug]   at System.Net.Http.StreamToStreamCopy.BufferReadCallback(IAsyncResult ar)
2017-03-24T15:17:38.4626512Z ##[debug]--- End of stack trace from previous location where exception was thrown ---
2017-03-24T15:17:38.4626512Z ##[debug]   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
2017-03-24T15:17:38.4626512Z ##[debug]   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
2017-03-24T15:17:38.4626512Z ##[debug]   at CallSite.Target(Closure , CallSite , Object )
2017-03-24T15:17:38.4939015Z ##[error]The stream does not support concurrent IO read or write operations.

I have tried to execute this on different build servers, on hosted agent, etc. but the result is the same. Also if I try to execute this code from an interactive session on my build server from ISE I can't have the code to fail. I posed a similar question on the VSTS build task project here but till now, no luck.

Does anyone have a suggestion or a test I can do in order to understand what is going on and why is this failing?

I would appreciate any tip.

UPDATE 1:

I executed my code by running it as a background job via a Start-Job cmdlet and I can't get it to fail. So it needs to be something related with the way the VSTS agent executes my code. I'll keep digging. If you have any suggestions, they are still more then welcome.

Answer 1

System.Net.Http.StreamContent and System.IO.FileStream are instantiated but not disposed.

Add the code to invoke Dispose like you did in finally block.

try
{
    # enclose the program
}
finally
{
    if($null -ne $streamContent)
    {
        $streamContent.Dispose()
    }
    if($null -ne $packageFileStream)
    {
        $packageFileStream.Dispose()
    }
}

Answer 2

That exception is clearly threading-related.

Your method calls use async method calls, but not correctly:

.ReadAsStringAsync().GetAwaiter().GetResult()

just blocks the current thread while waiting for another: it actually takes up an additional thread.

So, because you're a) not using threading in a useful fashion and b) getting an exception for your efforts, I suggest you take anywhere in your code which uses that construct and replace it with a normal method call.

Edit:

Threading issues aside, you could try

$content.Add($streamContent)
$content.LoadIntoBufferAsync().GetAwaiter().GetResult()

to force the content stream to load prior to the next async post operation.

Answer 3

Be careful - this NotSupportedException can mask a different error, such as an authorization problem (e.g. incorrect credentials passed in the NetworkCredential object).

I found this out by running Fiddler and watching the responses.