Can I consume an entity response after having it initialized GZIPInputStream?

Question

My program is reading this large gzip file and it runs for an hour and so and it fails with the following stack trace:

java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:196)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
    at sun.security.ssl.InputRecord.read(InputRecord.java:480)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
    at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
    at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
    at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
    at org.apache.http.impl.conn.LoggingSessionInputBuffer.read(LoggingSessionInputBuffer.java:82)
    at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:182)
    at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:138)
    at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:238)
    at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
    at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:116)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.fill(BufferedReader.java:154)
    at java.io.BufferedReader.readLine(BufferedReader.java:317)
    at java.io.BufferedReader.readLine(BufferedReader.java:382)
    at com.trainchaser.feed.connections.StaticConnect.getScheduleFile(StaticConnect.java:116)
    at com.trainchaser.app.App.main(App.java:32)

where line StaticConnect.getScheduleFile(StaticConnect.java:116) is the while loop in the code below.

I've read similar posts and I do close the reader (in) after its done reading the while loop but it was still giving me the same error. So I think maybe if I consume the entity it would work, like so:

    HttpEntity entity=getResponse.getEntity();

    BufferedReader in = new BufferedReader(new InputStreamReader(
                        new GZIPInputStream(entity.getContent())));

   EntityUtils.consumeQuietly(entity);

   try
   {
      while ((content = in.readLine()) != null)
      {...

Would that work? I was thinking maybe if I store the gzip file temporarily and not having the connection still allocated continuously would help prevent the error. I would test it myself but I am currently testing the fix of turning off firewall to see if it still errors out.

I am using org.apache.http.HttpEntity if it helps

How about telling us which http library you are using? – Stephen C Feb 16 '14 at 04:25 — Stephen C, Feb 16 '14 at 04:25
@StephenC, thanks, I have edited the question accordingly. – obsessiveCookie Feb 16 '14 at 09:44 — obsessiveCookie, Feb 16 '14 at 09:44

Stephen C · Answer 1 · 2014-02-17T07:55:19.033

I had a quick scan of the Apache 4.x HTTP library source code, and I can see nothing that would cause this call:

      consumeQuietly(entity);

to fail. However, the consumeQuietly call is going to close the underlying input stream ... from beneath the feet of your in stream. If you keep reading from in your code is going to get an IOException ... next time it needs to fill its buffer.

However, I suspect that does not explain the exception you are seeing. If my reading of the BufferedReader code is correct, you would see IOException("stream closed"), not a SocketException.

Either way, continuing to use in after you have "consumed" the entity seems a bad idea.

Can I consume an entity response after having it initialized BufferedReader?

That's a different issue. Yes you can do that.

The problem is with what you do with the reader after consuming the response. If you do something other than closing it, you are asking for trouble ... IMO.

By the way, the source code for the Apache HTTP libraries, and the core Java IO classes are all freely available. The best way to understand how they are going to behave is to read the source code ... just like I did.

Ahh okay thanks. It looks the problem with the java.net.SocketException: Connection reset lay in how i was inserting the data from the file to the db.. it was taking so long so it eventually reset it. It was fixed by having it so I use spring transaction — obsessiveCookie, Feb 16 '14 at 17:34

Can I consume an entity response after having it initialized GZIPInputStream?

1 Answers1