Spring Batch - Read multiple files from S3

Question

Like for reading single file in spring batch from s3, we use

@Bean
public FlatFileItemReader<Map<String, Object>> itemReader() {
    FlatFileItemReader<Map<String, Object>> reader = new FlatFileItemReader<>();
    reader.setLineMapper(new JsonLineMapper());
    reader.setRecordSeparatorPolicy(new JsonRecordSeparatorPolicy());
    reader.setResource(resourceLoader.getResource("s3://" + amazonS3Bucket + "/" + file));
    return reader;
}

But if i want to read all the files from some specific folder/key then is there something to MultiResourceItemReader, like below(which we use for local filesystem)

MultiResourceItemReader<UserData> reader = new MultiResourceItemReader<>();
reader.setResources(resources);

score 2 · Answer 1 · answered Sep 18 '20 at 09:38

2

No, it is up to you to create the Resource array and pass it to the MultiResourceItemReader.

answered Sep 18 '20 at 09:38

Mahmoud Ben Hassine

28,519
3
32
50

2

I'm asking...can i use MultiResourceItemReader to read multiple s3 files, if yes, then how? – Gaurav Raghav Sep 19 '20 at 12:23
Yes you can. You need to create an array of s3 resources and pass them to the `MultiResourceItemReader`. – Mahmoud Ben Hassine Sep 19 '20 at 18:57
@GauravRaghav - Were you able to implement this? Could you show some code please ? – PAA Jan 06 '21 at 05:44
@Pra_A - yes just posted my answer – Gaurav Raghav Jan 07 '21 at 06:28

score 2 · Answer 2 · answered Jan 07 '21 at 06:26

Create a MultiResourceItemReader like this,

@Autowired
private AmazonS3 s3;

@Autowired
private ResourceLoader resourceLoader;


public MultiResourceItemReader<String> fileItemReader() throws Exception {

    List<Resource> resourceList = new ArrayList<>();

    String s3ResponseFilePath = "s3://bucket/path/"; //put your s3 path here

    //TODO: warn: this functn can only return max 1000 objects
    s3objects = s3.listObjects("bucket", s3ResponseFilePath).getObjectSummaries();

    for(S3ObjectSummary it:s3objects)
        resourceList.add(resourceLoader.getResource( "s3://" + s3Config.getBucket() + "/" + it.getKey()));

    Resource[] resources = resourceList.toArray(new Resource[resourceList.size()]);

    MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
    reader.setResources(resources);
    reader.setDelegate(flatFileItemReader());

    return reader;
}

This reader will need a delegate and lineMapper, you can implement it like this,

private FlatFileItemReader<String> flatFileItemReader() throws Exception {

    FlatFileItemReader<String> reader = new FlatFileItemReader<>();
    JsonLineMapper lineMapper = new JsonLineMapper();
    reader.setLineMapper(lineMapper);
    reader.afterPropertiesSet();

    return reader;
}


public class JsonLineMapper implements LineMapper<String> {

    private ObjectMapper mapper = new ObjectMapper();

    @Override
    public String mapLine(String s, int i) throws Exception {

        return s;
    }
}

Seems to be not working using the same strategy in [my question](https://stackoverflow.com/questions/70144975/input-resource-must-exist-for-aws-s3-resources) — Braham Shakti, Nov 28 '21 at 16:09
I created the reader in same approach with -> SynchronizedItemStreamReader delegating to MultiResourceItemReader which delegates to FlatFileItemReader but got an error - resource must exist (s3 resource urls shown in logs is same as my file available in s3 bucket). any idea what could be wrong ? — Ashish Awasthi, Aug 21 '22 at 23:58

Spring Batch - Read multiple files from S3

2 Answers2