1

I need to get a file daily via SFTP. I would like to use Spring Integration with Java config. The file is generally available at a specific time each day. The application should try to get the file near that time each day. If the file is not available, it should continue to retry for x attempts. After x attempts, it should send an email to let the admin know that the file is still not available on the SFTP site.

One option is to use SftpInboundFileSynchronizingMessageSource. In the MessageHandler, I can kick off a job to process the file. However, I really don't need synchronization with the remote file system. After all, it is a scheduled delivery of the file. Plus, I need to delay at most 15 minutes for the next retry and to poll every 15 minutes seems a bit overkill for a daily file. I guess that I could use this but would need some mechanism to send email after a certain time elapsed and no file was received.

The other option seems to be using get of the SFTP Outbound Gateway. But the only examples I can find seem to be XML config.

Update

Adding code after using help provided by Artem Bilan's answer below:

Configuration class:

@Bean
@InboundChannelAdapter(autoStartup="true", channel = "sftpChannel", poller = @Poller("pollerMetadata"))
public SftpInboundFileSynchronizingMessageSource sftpMessageSource(ApplicationProperties applicationProperties, PropertiesPersistingMetadataStore store) {
    SftpInboundFileSynchronizingMessageSource source =
            new SftpInboundFileSynchronizingMessageSource(sftpInboundFileSynchronizer(applicationProperties));
    source.setLocalDirectory(new File("ftp-inbound"));
    source.setAutoCreateLocalDirectory(true);
    FileSystemPersistentAcceptOnceFileListFilter local = new FileSystemPersistentAcceptOnceFileListFilter(store,"test");
    source.setLocalFilter(local);
    source.setCountsEnabled(true);        
    return source;
}

@Bean
public PollerMetadata pollerMetadata() {
    PollerMetadata pollerMetadata = new PollerMetadata();
    List<Advice> adviceChain = new ArrayList<Advice>();
    adviceChain.add(retryCompoundTriggerAdvice());
    pollerMetadata.setAdviceChain(adviceChain);
    pollerMetadata.setTrigger(compoundTrigger());
    return pollerMetadata;
}

@Bean
public RetryCompoundTriggerAdvice retryCompoundTriggerAdvice() {
    return new RetryCompoundTriggerAdvice(compoundTrigger(), secondaryTrigger());
}

@Bean
public CompoundTrigger compoundTrigger() {
    CompoundTrigger compoundTrigger = new CompoundTrigger(primaryTrigger());
    return compoundTrigger;
}

@Bean
public Trigger primaryTrigger() {
    return new CronTrigger("*/60 * * * * *");
}

@Bean
public Trigger secondaryTrigger() {
    return new PeriodicTrigger(10000);
}

@Bean
@ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler(PropertiesPersistingMetadataStore store) {
    return new MessageHandler() {

        @Override
        public void handleMessage(Message<?> message) throws MessagingException {
            System.out.println(message.getPayload());
            store.flush();
        }

    };
}

RetryCompoundTriggerAdvice class:

public class RetryCompoundTriggerAdvice extends AbstractMessageSourceAdvice {

    private final CompoundTrigger compoundTrigger;

    private final Trigger override;

    private int count = 0;

    public RetryCompoundTriggerAdvice(CompoundTrigger compoundTrigger, Trigger overrideTrigger) {
        Assert.notNull(compoundTrigger, "'compoundTrigger' cannot be null");
        this.compoundTrigger = compoundTrigger;
        this.override = overrideTrigger;
    }

    @Override
    public boolean beforeReceive(MessageSource<?> source) {
        return true;
    }

    @Override
    public Message<?> afterReceive(Message<?> result, MessageSource<?> source) {
        if (result == null && count <= 5) {
            count++;
            this.compoundTrigger.setOverride(this.override);
        }
        else {
            this.compoundTrigger.setOverride(null);
            if (count > 5) {
                 //send email
            }
            count = 0;
        }
        return result;
    }
}
James
  • 2,876
  • 18
  • 72
  • 116

1 Answers1

2

Since Spring Integration 4.3 there is CompoundTrigger:

* A {@link Trigger} that delegates the {@link #nextExecutionTime(TriggerContext)}
* to one of two Triggers. If the {@link #setOverride(Trigger) override} trigger is
* {@code null}, the primary trigger is invoked; otherwise the override trigger is
* invoked.

With the combination of CompoundTriggerAdvice:

* An {@link AbstractMessageSourceAdvice} that uses a {@link CompoundTrigger} to adjust
* the poller - when a message is present, the compound trigger's primary trigger is
* used to determine the next poll. When no message is present, the override trigger is
* used.

it can be used to reach your task:

The primaryTrigger can be a CronTrigger to run the task only once a day.

The override could be a PeriodicTrigger with desired short period to retry.

The retry logic you can utilize with one more Advice for poller or just extend that CompoundTriggerAdvice to add count logic to send an email eventually.

Since there is no file, therefore no message to kick the flow. And we don't have choice unless dance around the poller infrastructure.

Artem Bilan
  • 113,505
  • 11
  • 91
  • 118
  • 1
    `@Poller` has `value()` to accept `PollerMetadata` bean name. And there you can configure the `trigger`, `maxMessagesPerPoll` and `adviceChain` with desired `CompoundTriggerAdvice`. http://docs.spring.io/spring-integration/docs/4.3.1.RELEASE/reference/html/configuration.html#annotations and search for `@Poller` – Artem Bilan Aug 31 '16 at 15:25
  • Thanks. I found that documentation right after posting the comment and then deleted my comment, but apparently you still received / saw my comment. Sorry about that. – James Aug 31 '16 at 15:57
  • Glad to be helpful! So, it's time to accept my answer and let other people know that we have a solution here! And we will be free to do our own work :) – Artem Bilan Aug 31 '16 at 15:59
  • Almost done. Just trying to work through extending `CompoundTriggerAdvice` to implement the `retry` with `count` logic. – James Aug 31 '16 at 16:16
  • I cannot get this to work. I'm sure that I'm missing something but was unable to extend `CompoundTriggerAdvice` since its `CompoundTrigger` and `Trigger` are private. Instead, I extended `AbstractMessageSourceAdvice`. I notice though that once I set the override polling continues to occur using the override even once I set the override back to null after receiving a null `Message> result`. I noticed this same behavior with `CompoundTriggerAdvice`. Once the override is set, polling will be based on the override trigger even if a subsequent non-null `Message> result` is received. – James Aug 31 '16 at 17:00
  • Well, you can just borrow some idea from the `CompoundTriggerAdvice` and implement your own `AbstractMessageSourceAdvice`. Sorry, I didn't think properly about the solution for you, so just sharing ideas where and how to dig. – Artem Bilan Aug 31 '16 at 17:03
  • Yeah, that's what I'm doing but I guess I don't understand how `CompoundTrigger` works to dynamically set polling. I figured setting its override to null would allow the polling to resume using the primary. But it doesn't. Once the override is set (in `afterReceive`), it is forever locked to poll using that trigger. – James Aug 31 '16 at 17:10
  • Well, `CompoundTriggerAdvice` does it basing on the `null` result, but you can switch to the `primary` back to any other required condition. – Artem Bilan Aug 31 '16 at 17:14
  • That's what I'm doing. I see the same behavior using `CompoundTriggerAdvice`. So, perhaps there some other issue with my configuration. I'm going to update the OP with my code. I greatly appreciate your continued help. – James Aug 31 '16 at 17:20
  • I've noticed that after `afterReceive` is called with a non-null `result`, it is immediately called again. That seems to be causing the issue because on that immediate next call the `result` is null and so override trigger is set again. – James Aug 31 '16 at 18:33
  • 1
    I found the issue. Multiple messages were being generated per poll. The first message contained a non-null `result` and the very next message (presumably from the same poll) contained a null `result`. Once I set the max message per poll to one, the issue went away. I had it set with the annotation but forgot to set it when moving to Java config. Sorry that I missed that. Thanks again for your help. I accepted your answer. (I also upvoted it and your first comment.) – James Aug 31 '16 at 20:24