Delete source file from s3 bucket : s3-inbound-streaming-channel-adapter

Question

Source files are not deleting from s3 bucket once after successfully transfer it to target directory. steps 1. Using Inbound streaming channel adapter to stream source files from S3 to local directory.(working fine) step 2 : Want to delete source files once successfully transferred (not working) configuration code are below

                        <int-aws:s3-inbound-streaming-channel-adapter id="s3FilesInbound"
                                        channel="s3FilesChannel" 
                                        session-factory="s3SessionFactory" 
                                        filename-regex="^.*\\.(txt|csv)$" 
                                        remote-directory-expression="bucket_name" 
                                        auto-startup="true" >

                                <integration:poller id="s3FilesChannelPoller"
                                                    fixed-delay="1000"
                                                    max-messages-per-poll="1">
                                </integration:poller>
                            </int-aws:s3-inbound-streaming-channel-adapter>


                    <integration:stream-transformer id="streamTransformer" input-channel="s3FilesChannel" output-channel="s3FilesChannelOut"/>

        <integration:chain id="filesS3ChannelChain"
                input-channel="s3FilesChannelOut">
                <file:outbound-gateway
                    id="fileInS3ArchiveChannel"
                    directory="local_directory"
                    filename-generator-expression="headers.file_remoteFile">
                    <file:request-handler-advice-chain>
                        <ref bean="retryAdvice" />
                    </file:request-handler-advice-chain>
                </file:outbound-gateway>
                <integration:gateway request-channel="nullChannel"
                        error-channel="errorChannel" />
            </integration:chain>

Regards,

score 0 · Accepted Answer · answered Nov 07 '18 at 14:44

Since you use there a <integration:stream-transformer>, I don't see reason to rely on the <int-aws:s3-inbound-streaming-channel-adapter>. With the first one you just eliminate the streaming purpose of the last one.

I'd suggest you take a look into the regular <int-aws:s3-inbound-channel-adapter>, which already has a delete-remote-files="true" option.

On the other hand you still can do that with what you have so far, but you need to something like <integration:outbound-channel-adapter expression="@s3SessionFactory.getSession().remove(headers[file_remoteDirectory] + '/' + headers[file_remoteFile])">.

Those headers are populated by the AbstractRemoteFileStreamingMessageSource.

Delete source file from s3 bucket : s3-inbound-streaming-channel-adapter

1 Answers1