S3AbortableInputStream: Not all bytes were read from the S3ObjectInputStream

Question

I am getting this warning when doing s3 GET Object request

WARN S3AbortableInputStream: Not all bytes were read from the 
S3ObjectInputStream, aborting HTTP connection. This is likely an error 
and may result in sub-optimal behavior. Request only the bytes you 
need via a ranged GET or drain the input stream after use

I looked at AmazonS3: Getting warning: S3AbortableInputStream:Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection

I do not have a clear understanding of what needs to be done.

Below is the code snippet I am using

 try {
    s3obj = s3Client.getObject(s3Bucket, objectSummary.getKey)
    val name = s3obj.getKey.substring(dirPath.length + 1)
    val filename = name.replace(".xml", "")
    endpoints += base_url + filename
  } catch {
    case e@(_: IOException |_ : JSONException | _: Exception) =>
      println(e)
  } finally {
    if (s3obj != null)
      s3obj.close()
  }

score 1 · Answer 1 · answered Apr 04 '19 at 19:01

Basically what it does is, if S3 client tries to close an InputStream without consuming it fully after reading N bytes, it gives the warning message because they need to close the HttpRequest and the underlying http connection with removing the connection from the connection pool.
This could happen when:
Web application reads data from S3 initially and S3 DataStore normally caches the data in local file system and while this is happening, if web visitor disconnects the HTTP connection, and so if Tomcat indicates it, then the http processing thread will be halted and as a result, the InputStream will be closed.

Please refer the following link: https://docs.aws.amazon.com/AmazonS3/latest/dev/RetrievingObjectUsingJava.html

Thanks for your response. What do I have to do fix the warning in my case? — fledgling, Apr 05 '19 at 05:44

score 0 · Answer 2 · answered Oct 19 '20 at 07:12

If you add

  s3obj.abort()

to your exception block, it should resolve the issue. It would look like this:

  } catch {
    case e@(_: IOException |_ : JSONException | _: Exception) =>
      println(e)
      if (s3obj != null) {
        s3obj.abort()
      }
  } finally {
    if (s3obj != null)
      s3obj.close()
  }

It is a complexity of the AWS SDK in that it wants to avoid the normal close() operations to short cut streaming lots of data when it is not necessary.

S3AbortableInputStream: Not all bytes were read from the S3ObjectInputStream

2 Answers2