1

I am looking for AWS s3 Spring Integration File Modification Monitor sample.

Requirement: whenever a file gets modified from the AWS S3 bucket, the latest changes has to be transferred from s3 to EC2 instance.

We have the watch-event for file inbound adapter, but I am looking for the similar adapter or logic for s3 operation.

What we have right now:

<int-file:inbound-channel-adapter id="modifiedFiles" directory="${input.directory}" use-watch-service="true" filter="acceptAllFilter" watch-events="MODIFY"/>

XML Config:

<bean id="acceptOnceFilter"
    class="org.springframework.integration.file.filters.AcceptOnceFileListFilter" />

<integration:channel id="s3FilesChannel" />
<int-aws:s3-inbound-channel-adapter id="s3FilesChannelId" channel="s3FilesChannel"
             session-factory="s3SessionFactory"
             auto-create-local-directory="false"
             delete-remote-files="false"
             preserve-timestamp="true"
             local-directory="file:${localFilePath}"
             local-filter="acceptOnceFilter"
             remote-directory-expression="${bucket_name}"
             auto-startup="false"
             filename-regex="${regx_expresion}" 
             remote-file-separator="/" >
    <integration:poller fixed-delay="1000"/>
</int-aws:s3-inbound-channel-adapter>

<integration:channel id="s3FilesChannel">
    <integration:queue/>
</integration:channel>
Las
  • 37
  • 4
  • 1
    Please provide us with the code you worked upon so we can help you in a better way. – Himanshu Bansal Sep 06 '18 at 05:07
  • Himanshu , Thanks for your reply. I haven't started coding yet just analyzing this type of requirement. – Las Sep 06 '18 at 05:15
  • Looking similar kind of solution for s3 bucket . – Las Sep 06 '18 at 05:22
  • Welcome to Stack Overflow! Other users marked your question for low quality and need for improvement. I re-worded/formatted your input to make it easier to read/understand. Please review my changes to ensure they reflect your intentions. Feel free to drop me a comment in case you have further questions or feedback for me. – GhostCat Sep 06 '18 at 10:39
  • 1
    Please note: never add more info in comments. Always update your question instead. – GhostCat Sep 06 '18 at 10:40

1 Answers1

0

There is no such a monitor for AWS S3. This is a remote protocol over HTTP unlike plain file system monitor based on the WatchService. You don't compare apples with apples.

You need to use a standard <int-aws:s3-inbound-channel-adapter>. There is an S3PersistentAcceptOnceFileListFilter used by default which reacts to the modification for the remote files. Well, essentially it polls the remote file and check its lastmodified.

See more info in the Reference Manual:

Also, if you configure the filter to use a FtpPersistentAcceptOnceFileListFilter, and the remote file timestamp changes (causing it to be re-fetched)

The same rules are applied to AWS S3 Channel Adapters.

Artem Bilan
  • 113,505
  • 11
  • 91
  • 118
  • Yes Right Artem . I am trying to implement the solution mentioned above . Also i guess , i should use S3InboundFileSynchronizer class also for achieving this. am i right ?. – Las Sep 10 '18 at 05:53
  • That’s correct . Also you need a `S3InboundFileSynchronizingMessageSource` – Artem Bilan Sep 10 '18 at 11:45
  • Hi Artem, I am getting the followingerror while configuring org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected exception parsing XML document from class path resource nested exception is java.lang.AbstractMethodError: org.springframework.integration.file.config.AbstractRemoteFileInboundChannelAdapterParser.getPersistentAcceptOnceFileListFilterClass()Ljava/lang/Class; – Las Sep 10 '18 at 11:46
  • You don’t use compatible versions. Bottom line that Spring Integration version should come from this `spring-integration-aws`. Also would be great to rely on the versions from Spring Boot and add respective SI-AWS dependency – Artem Bilan Sep 10 '18 at 11:50
  • Either java configuration or xml configuration is OK .. right ? – Las Sep 10 '18 at 11:50
  • Right. The problem with incompatible versions – Artem Bilan Sep 10 '18 at 11:55
  • Hi Artem, I have tried java configuration and its working fine . But If i try XML configuration in flow its not working. Not giving any error but file are not moving to EC2 instance . Attached my XML config settings. – Las Sep 11 '18 at 11:36
  • You have there this `auto-startup="false"`, so your ` – Artem Bilan Sep 11 '18 at 13:42
  • Thank you Artem. Now i am facing this issue... IllegalArgumentException: No poller has been defined for endpoint 's3FilesChannel.adapter', and no default poller is available within the context. – Las Sep 12 '18 at 07:14
  • Well, first of all I see it in your config above, but on other hand here: https://docs.spring.io/spring-integration/docs/5.0.8.RELEASE/reference/html/messaging-channels-section.html#channel-adapter-namespace-inbound. An Inbound Channel Adapter always has to be with the poller - global or its own. – Artem Bilan Sep 12 '18 at 12:13
  • Thanks a lot Artem . Either ways working fine now. The issue was because of remote-directory-expression property. I just tried remote-directory to set remote path now it working fine. – Las Sep 18 '18 at 10:18