Poller/Adapter in s3-inbound-channel-adapter not working properly

Question

My scenario is to transfer one file from AWS S3 bucket to EC2 instance only once whenever the file gets modified. I used the following configuration and manually started the adapter when server starts.

The issue is the execution is repeated 5 or 6 times when server starts. Looks like different threads execution are happening. I am able to see different task-executor in the log don't know whether its a poller issue or adapter issue.

I am using service activator to do some other operation based on the file change in S3 location.

Note : This issue is happening only once on start up . It's working fine further file modification.

Config:

<bean id="s3SessionFactory" 
            class="org.springframework.integration.aws.support.S3SessionFactory"></bean>
        <bean id="acceptOnceFilter"
            class="org.springframework.integration.file.filters.AcceptOnceFileListFilter" />
        <task:executor id="s3PollingExecutor" pool-size="1" queue-capacity="10" />
        <integration:channel id="s3FilesChannel"/>
        <int-aws:s3-inbound-channel-adapter id="s3FileInbound"
                     channel="s3FilesChannel" 
                     session-factory="s3SessionFactory" 
                     auto-create-local-directory="false"
                     delete-remote-files="false" 
                     preserve-timestamp="true"
                     filter="acceptOnceFilter"
                     local-directory="local_directory"
                     auto-startup="false" 
                     remote-directory="s3_bucket">
            <integration:poller id="s3FilesChannelPoller" 
                                fixed-rate="1000" 
                                max-messages-per-poll="1" time-unit="MILLISECONDS" 
                                task-executor="s3PollingExecutor">
            </integration:poller>
        </int-aws:s3-inbound-channel-adapter>
        <integration:service-activator id="s3FilesChannelWatcher" 
                                       input-channel="s3FilesChannel" 
                                       output-channel="nullChannel"
                                       ref="configurationFileWatcher" 
                                       method="getConfigurationFileWatcher">
        </integration:service-activator>

As you suggested i have tried the following .

<bean id="acceptOnceFilterRegion"
      class="cS3FileFilterOnLastModifiedTime">
    <constructor-arg index="0" ref="metaDataStoreRegion"/>
    <constructor-arg index="1" value="*"/>
</bean>

logic added for checking last modified time

import org.springframework.integration.aws.support.filters.S3PersistentAcceptOnceFileListFilter;
import org.springframework.integration.metadata.ConcurrentMetadataStore;
import com.amazonaws.services.s3.model.S3ObjectSummary;
public class S3FileFilterOnLastModifiedTime extends S3PersistentAcceptOnceFileListFilter {

    Long delayTime = 1000L;

    public S3FileFilterOnLastModifiedTime(ConcurrentMetadataStore store, String prefix) {
        super(store, prefix);
    }

    @Override
    public boolean accept(S3ObjectSummary file) {
        long lastModified = modified(file);
        long currentTime = System.currentTimeMillis();
        long timeDifference = currentTime - lastModified;
        return timeDifference > delayTime;
    }   
}

Still no hopes .logs are like this .......

[INFO ] 2018-10-11 11:22:10,888 [s3PollingExecutor-1] ConfigurationSettingWatcher {} - ConfigurationSettingWatcher Started succesfully
[INFO ] 2018-10-11 11:22:10,892 [s3PollingExecutor-2] ConfigurationSettingWatcher {} - ConfigurationSettingWatcher Started succesfully
[INFO ] 2018-10-11 11:22:10,892 [s3PollingExecutor-3] ConfigurationSettingWatcher {} - ConfigurationSettingWatcher Started succesfully
[INFO ] 2018-10-11 11:22:10,893 [s3PollingExecutor-4] ConfigurationSettingWatcher {} - ConfigurationSettingWatcher Started succesfully
[INFO ] 2018-10-11 11:22:10,893 [s3PollingExecutor-5] ConfigurationSettingWatcher {} - ConfigurationSettingWatcher Started succesfully
[INFO ] 2018-10-11 11:22:10,894 [s3PollingExecutor-6] ConfigurationSettingWatcher {} - ConfigurationSettingWatcher Started succesfully

score 0 · Accepted Answer · answered Oct 10 '18 at 18:54

0

I think you are missing the fact that your s3_bucket is not empty on start up and the <int-aws:s3-inbound-channel-adapter> picks up all the files on its start. That's how you see those several task: each for every file.

If you really worry only about new files in that bucket, you need to consider do not use an in-memory AcceptOnceFileListFilter, but switch into some persistent implementation, based on the shared MetadataStore implementation. For this purpose there is a S3PersistentAcceptOnceFileListFilter in the spring-integration-aws and DynamoDbMetadataStore to persist a filtering results into the DynamoDb on AWS: https://github.com/spring-projects/spring-integration-aws#metadata-store-for-amazon-dynamodb

answered Oct 10 '18 at 18:54

Artem Bilan

113,505
11
91
118

Yes Artem. This bucket will not be empty and it will have only one file always . it will be final with static name.Only we can modify this file as per the requirement. The latest changes has to be transferred to local machine (I mean EC2 instance) based on the delay mentioned in the poller. – Las Oct 11 '18 at 08:50
Also, pay attention that your `local-directory` might not be empty in start up, so Channel Adapter picks those local files up – Artem Bilan Oct 11 '18 at 11:18
Yes . The same file i have in the local directory as well.It not empty . – Las Oct 11 '18 at 11:50
So, clean up everything and starts over. You also can consider to use `local-filter` as persistent one as well to skip local files after restarting – Artem Bilan Oct 11 '18 at 12:02
The `ConfigurationSettingWatcher Started succesfully` is somehow not related to Spring Integration. It fully looks like your custom code. Not sure how you connect it with the S3 Channel Adapter. You can turn on DEBUG logging level for the `org.springframework.integration` category and observer the messages that channel adapter produces. Maybe this one will give you some clue what's going on. – Artem Bilan Oct 11 '18 at 13:15
Yes. ConfigurationSettingWatcher is a custom class will be called only once when file gets modified. – Las Oct 13 '18 at 07:31

Poller/Adapter in s3-inbound-channel-adapter not working properly

1 Answers1