Can spring-integration-aws for s3 replicate bucket subdirectories?

Question

I was using the sample configuration from https://github.com/spring-projects/spring-integration-aws for an Inbound Channel Adapter, but I have a bucket that will contain subdirectories with CSV's inside.

Is there a way to replicate locally the bucket tree structure? I have only managed to copy the files from inside the subdirs, but they end up being created right in the root of the directory that I set with messageSource.setLocalDirectory(LOCAL_FOLDER);

Or is there a way to identify what bucket subdir the file came from?

@Bean
@InboundChannelAdapter(value = "s3FilesChannel", poller = @Poller(fixedDelay = "100"))
public S3InboundFileSynchronizingMessageSource s3InboundFileSynchronizingMessageSource() {
    S3InboundFileSynchronizingMessageSource messageSource =
            new S3InboundFileSynchronizingMessageSource(s3InboundFileSynchronizer());
    messageSource.setAutoCreateLocalDirectory(true);
    messageSource.setLocalDirectory(LOCAL_FOLDER);
    messageSource.setLocalFilter(new AcceptOnceFileListFilter<File>());
    return messageSource;
}

score 0 · Accepted Answer · answered Oct 05 '18 at 15:17

You need to use

/**
 * Switch the local {@link FileReadingMessageSource} to use a custom
 * {@link DirectoryScanner}.
 * @param scanner the {@link DirectoryScanner} to use.
 * @since 5.0
 */
public void setScanner(DirectoryScanner scanner) {

And specify a RecursiveDirectoryScanner: https://docs.spring.io/spring-integration/docs/current/reference/html/ftp.html#_recovering_from_failures:

Starting with version 5.0, the Inbound Channel Adapter can build sub-directories locally according the generated local file name.

You need to pull from bucket and the S3InboundFileSynchronizingMessageSource will restore a structure in the local dir.

Note: you need to use, of course, the latest spring-integration-aws-2.0.0.RELEASE.

Can spring-integration-aws for s3 replicate bucket subdirectories?

1 Answers1