java.lang.IllegalStateException: InputStream has already been read - do not use InputStreamResource if a stream needs to be read multiple times

Question

I am trying to read a file from aws s3 bucket and set it as resource inside my spring batch reader class. When I test the application on aws lambda function I got below error. any suggestion experts?

    Caused by: java.lang.IllegalStateException: InputStream has already been read - do not use InputStreamResource if a stream needs to be read multiple times
    at org.springframework.core.io.InputStreamResource.getInputStream(InputStreamResource.java:97) ~[task/:na]
    at org.springframework.batch.item.file.DefaultBufferedReaderFactory.create(DefaultBufferedReaderFactory.java:34) ~[task/:na]
    at org.springframework.batch.item.file.FlatFileItemReader.doOpen(FlatFileItemReader.java:266) ~[task/:na]
    at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.open(AbstractItemCountingItemStreamItemReader.java:146) ~[task/:na]

Class to read from s3 bucket
@Service
public class S3BucketProcessing {
private static final AmazonS3 s3 = AmazonS3ClientBuilder.standard().build();

public InputStreamResource readFile() throws IOException{

   String bucketName = "mybuckey";
   String key = "File.txt";

   S3Object object = s3.getObject(new GetObjectRequest(bucketName, key));    

   return new InputStreamResource(object.getObjectContent());

}

Spring batch reader class

    @Component
public class MyReader extends FlatFileItemReader<MyEntity> {

    MyLineMapper mapper;
    MyTokenizer tokenizer;
    S3BucketProcessing s3BucketProcessing;

    @Autowired
    public MyReader(MyTokenizer tokenizer, MyLineMapper mapper, S3BucketProcessing s3BucketProcessing) throws Exception{
        LOG.info("CardCustomerNotificationReader constructor");
        this.mapper = mapper;
        this.tokenizer = tokenizer;
        this.s3BucketProcessing= s3BucketProcessing;
        this.setResource(s3BucketProcessing.readFile());
        mapper.setLineTokenizer(tokenizer);
        this.setLineMapper(mapper);
    }
}

Teddy · Accepted Answer · 2018-11-16T21:57:13.660

18

The docs suggest using ByteArrayResource to cache the content in memory, rather than InputStreamResource.

https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/core/io/InputStreamResource.html

Just change the returns part like this:

//As suggested by berzerk
byte[] content = IOUtils.toByteArray(object.getObjectContent()); 

//Then
return new ByteArrayResource( content );

edited Nov 16 '18 at 21:57

answered Nov 16 '18 at 21:42

Teddy

4,009
2
33
55

1

thanks, can you please share an example how it can implemented in my context. – Maana Nov 16 '18 at 21:50
Will this cause out of memory issues for higher file? – VinuIsNotUnix Nov 25 '22 at 13:35
Yes. If you hold one big file like 100MB+ in memory, or several smaller files, it's going to take up that much of heap space. You need to see if it is worth having such a lot of data in memory. Also, will it be there for a short while? Or, will it hang around after it is not needed anymore (eg., there is some static reference to it, even though it is not needed anymore)? – Teddy Nov 25 '22 at 13:42

score 1 · Answer 2 · answered Nov 16 '18 at 21:35

1

Instead of returning InputStreamResource , you shud return content of the stream may be byte[ ].
byte[] content = IOUtils.toByteArray(object.getObjectContent()); return content ;

answered Nov 16 '18 at 21:35

batflix

196
5

then how do I set -> this.setResource(s3BucketProcessing.readFile()); – Maana Nov 16 '18 at 21:42

score 0 · Answer 3 · answered Jul 24 '20 at 11:12

So I had to send in JSON/XML object as Output Stream. I was using the InputStreamResource and was getting the same error as OP.

Here's the solution that worked for me.

    @Override
    public Resource dataExportForFieldExtractorModel() {

        ObjectMapper xmlMapper = new XmlMapper().enable(SerializationFeature.INDENT_OUTPUT);
        byte[] data;
        Resource resource = null;
        try {
            data = xmlMapper.writerWithDefaultPrettyPrinter().writeValueAsBytes(new DataExportResponse());
            resource = new ByteArrayResource(data);
        } catch (JsonProcessingException e) {
            e.printStackTrace();
        }
        return resource;
    }

As the resource needs to be byte[], I changed resource = new InputStreamResource(new ByteArrayInputStream(data)); to resource = new ByteArrayResource(data);.

java.lang.IllegalStateException: InputStream has already been read - do not use InputStreamResource if a stream needs to be read multiple times

3 Answers3