HttpResponseMessage - Decompression of Gzip respone from an API Call throws the following exceptiong - The response ended prematurely

Question

I am actually consuming a REST Service from which I receive the response compressed as Gzip so, following is my code

HttpClientHandler handler = new HttpClientHandler()
{
    AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
var httpClient = new HttpClient(handler);
httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var result = await httpClient.GetAsync(url);

I get an exception of the last line "await httpClient.GetAsync(url)"

Following is the stack trace :

at System.Net.Http.HttpConnection.<FillAsync>d__87.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.<CopyToAsyncCore>d__10.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   
at System.IO.Compression.DeflateStream.CopyToStream.<CopyFromSourceToDestinationAsync>d__6.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.DecompressionHandler.DecompressedContent.<SerializeToStreamAsync>d__5.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()  
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.HttpContent.<LoadIntoBufferAsyncCore>d__54.MoveNext()

As a quick update: I did try a different implementation of the call as follow :

HttpWebRequest Http = (HttpWebRequest)WebRequest.Create("url");

Http.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");
HttpWebResponse WebResponse = (HttpWebResponse)Http.GetResponse();
Stream responseStream = responseStream = WebResponse.GetResponseStream();
if (WebResponse.ContentEncoding.ToLower().Contains("gzip"))
{
    responseStream = new GZipStream(responseStream, CompressionMode.Decompress);
}
StreamReader Reader = new StreamReader(responseStream, Encoding.Default);
string responseAsString= Reader.ReadToEnd();
WebResponse.Close();
responseStream.Close();

I am getting an exception on the following line string responseAsString= Reader.ReadToEnd();

The exception message is also "The response ended prematurely."

Following is the stack trace:

at System.Net.Http.HttpConnection.Fill()
at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.Read(Span`1 buffer)
at System.Net.Http.HttpBaseStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Compression.DeflateStream.ReadCore(Span`1 buffer)
at System.IO.Compression.DeflateStream.Read(Byte[] array, Int32 offset, Int32 count)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadToEnd()
at ApiCall.Program.<Main>d__0.MoveNext() in 
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at ApiCall.Program.<Main>(String[] args)

See following : https://weblog.west-wind.com/posts/2007/jun/29/httpwebrequest-and-gzip-http-responses — jdweng, Dec 16 '19 at 15:11
@BaoucheIqbal the error message is "The response ended prematurely" I did also post the stack trace — Mohamed Kamel Bouzekria, Dec 16 '19 at 15:16
You have the same error when not using the compression? Just to make sure that the calling party is indeed returning a valid response? And if so, what if you use a tool like postman and do a similar call (setting the headers so it will return a compressed version)? — Michiel, Dec 16 '19 at 15:26
@Michiel Yes, I tried using postman and I do get the response with the "Content-Encoding" header set to "gzip" — Mohamed Kamel Bouzekria, Dec 16 '19 at 15:34
What line is giving error. Two possibilities 1) The transmit end did not zip properly 2) You do not have the entire zip data before you deflat. Check file size of send and receive zip to make sure they are the same. If you are getting http 1.1 (chunk mode) you are not getting the entire data in one chunk which would account for the error. — jdweng, Dec 16 '19 at 17:24
Calling another site, using compression, from your code; does that work? Just to eliminate where it can occur? This is also an async call (as you only list a small portion of the code), do you handle the await correctly? As it might close the connection as the application already ends and the reading of the stream is on another thread? — Michiel, Dec 16 '19 at 21:07
@jdweng The line throwing the exception is the "await httpClient.GetAsync(url)" To comment further on the two possibilities listed above : 1) I am getting the response pretty well on postman, so the zipped response should be fine 2) From where can I check the size of sent and received response. Also from postman the response has a header "Transfer-Encoding" set to "Chunked", would that be related to what you said ? — Mohamed Kamel Bouzekria, Dec 17 '19 at 08:15
@Michiel I am actually handling the await properly (I even tried both with and without the await keyword) Also, is there any sites that use compression so that I can try to consume their service in order to test ? — Mohamed Kamel Bouzekria, Dec 17 '19 at 08:21
I don't know why that line would give an error. Use a sniffer like wireshark or fiddler and compare the headers in the 1st request from postman with the headers in your c# app. Make the headers in c# look exactly like the postman results. If you are in chunk mode (http 1.1) than I would use stream mode (http 1.0). Add following line : Http.ProtocolVersion = HttpVersion.Version10; — jdweng, Dec 17 '19 at 09:51
@jdweng I did make them exactly the same ! The problem I think is when the StreamReader try to ReadToEnd the response as a String. — Mohamed Kamel Bouzekria, Dec 17 '19 at 11:11
You are not waiting until the entire response is competed. As a simple test add a wait for 30 seconds before ReadToEnd and see if it works so you know the issue that you are reading before all the data is received. — jdweng, Dec 17 '19 at 11:24
@jdweng I've tried to put delay of 30 seconds but it doesn't work either — Mohamed Kamel Bouzekria, Dec 17 '19 at 11:39
How long does file take to download in postman? Are you getting any exceptions? Is StreamReader == null? Is ResponseString == null? — jdweng, Dec 17 '19 at 11:43
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/204411/discussion-between-mohamed-kamel-bouzekria-and-jdweng). — Mohamed Kamel Bouzekria, Dec 17 '19 at 13:29

score 3 · Accepted Answer · answered Mar 20 '20 at 18:30

3

Try adding HttpCompletionOption.ResponseContentRead to your GetAsync call:

var result = await httpClient.GetAsync(url, HttpCompletionOption.ResponseContentRead);

This should make your code wait until the entire response has been read.

answered Mar 20 '20 at 18:30

rkralston

386
1
5
20

score 0 · Answer 2 · answered Dec 20 '19 at 07:04

0

Have you tried using RestSharp for this request? It should take care of everything for you.

http://restsharp.org/

answered Dec 20 '19 at 07:04

Erwin Mayer

18,076
9
88
126

HttpResponseMessage - Decompression of Gzip respone from an API Call throws the following exceptiong - The response ended prematurely

2 Answers2