2

I am using IntegrationFlow as Sftp Inbound DSL Configuration where I am using CustomTriggerAdvice to handle manual trigger. Please see below code snippet for reference.

I am also using RotatingServerAdvice for handling multiple path in same host.

But when I start the Sftp Inbound it fetch file for the first time from every path but it does not work for second time and onward. Sftp Inbound Starts but does not fetch file from paths. I couldn't figure out the problem. Is there anything that I am missing?

SftpConfiguration

public IntegrationFlow fileFlow() {
    SftpInboundChannelAdapterSpec spec = Sftp
            .inboundAdapter(dSF())
            .preserveTimestamp(true)
            .remoteDirectory(".")
            .autoCreateLocalDirectory(true)
            .deleteRemoteFiles(false)
            .localDirectory(new File(getDestinationLocation()));

    return IntegrationFlows
            .from(spec, e -> e.id(BEAN_ID)
                    .autoStartup(false)
                    .poller(Pollers
                            .fixedDelay(5000)
                            .advice(
                                    customRotatingServerAdvice(dSF()),
                                    customTriggerAdvice()
                            )
                    )
            )
            .channel(sftpReceiverChannel())
            .handle(sftpInboundMessageHandler())
            .get();
}

private MessageChannel sftpReceiverChannel() {
    return MessageChannels.direct().get();
}

... ... ...

@Bean
public RotatingServerAdvice customRotatingServerAdvice(
      DelegatingSessionFactory<LsEntry> dSF
) {
    List<String> pathList = getSourcePathList();
    for (String path : pathList) {
        keyDirectories.add(new RotationPolicy.KeyDirectory(KEY, path));
    }

    return new RotatingServerAdvice(
            dSF,
            keyDirectories
    );
}

@Bean
public CustomTriggerAdvice customTriggerAdvice() {
    return new CustomTriggerAdvice(customControlChannel(),BEAN_ID);
}


@Bean
public IntegrationFlow customControlBus() {
    return IntegrationFlows.from(customControlChannel())
            .controlBus()
            .get();
}


@Bean
public MessageChannel customControlChannel() {
    return MessageChannels.direct().get();
}

CustomTriggerAdvice

public class CustomTriggerAdvice extends AbstractMessageSourceAdvice {
    private final MessageChannel controlChannel;
    private final String BEAN_ID;

    public CustomTriggerAdvice(MessageChannel controlChannel, String beanID) {
        this.controlChannel = controlChannel;
        this.BEAN_ID = beanID;
    }

    @Override
    public boolean beforeReceive(MessageSource<?> source) {
        return true;
    }

    @Override
    public Message<?> afterReceive(Message<?> result, MessageSource<?> source) {
        if (result == null) {
            controlChannel.send(new GenericMessage<>("@" + BEAN_ID + ".stop()"));
        } 
        return result;
    }
}

Starting Sftp Inbound using MessageChannel

@Qualifier("customControlChannel") MessageChannel controlChannel;


public void startSftpInbound(){
       controlChannel.send(new GenericMessage<>("@" + beanID + ".start()"));
}
Towfiqul Islam
  • 456
  • 6
  • 14
  • Your `CustomTriggerAdvice` stops your `SourcePollingChannelAdapter` with `BEAN_ID` for your `SftpInboundChannelAdapterSpec`, but I don't see where you call that `startSftpInbound()`... – Artem Bilan Jun 15 '20 at 15:07
  • @ArtemBilan, I am calling it to anther service. – Towfiqul Islam Jun 15 '20 at 16:11
  • So, what do you expect then? You stop it when no new files and you definitely need to start it again eventually to be able to pick new files up. If there are still no new files on the restart, it is going to stop again. That's your logic. Not sure what you are looking for... – Artem Bilan Jun 15 '20 at 16:13
  • Yes, you are right. But when I am restarting the **Sftp Inbound** using `startSftpInbound()` it is restarting but it is not fetching new files. – Towfiqul Islam Jun 15 '20 at 16:15
  • Are you sure that you have new files in your remote folders and their names don't clash with already fetched? The `SftpInboundFileSynchronizingMessageSource` is configured with the `SftpPersistentAcceptOnceFileListFilter` by default to avoid refetching files with the same name and timestamp. You need to investigate logs in DEBUG level to be sure what is going on. – Artem Bilan Jun 15 '20 at 16:20
  • 1
    I have tried all kinds of **DEBUG** and I couldn't find anything and quite sure that the file is not is not fetched before and file is not stored in the `MetadataStore`. – Towfiqul Islam Jun 15 '20 at 16:22
  • OK. How does it work if you remove your `customTriggerAdvice()`? Kinda let's try to narrow the boundaries for the problem – Artem Bilan Jun 15 '20 at 16:24
  • 1
    It works as usual. It fetches new file within a moment after polling each path. – Towfiqul Islam Jun 15 '20 at 16:26
  • OK. Can you now to stop and restart that channel adapter to be sure that the problem with rotation is really because we stop in between? – Artem Bilan Jun 15 '20 at 16:32
  • Are you asking me to stop the **Sftp Inbound** manually instead of using it in `afterReceive()`? – Towfiqul Islam Jun 15 '20 at 16:38
  • Right. That's what I want to see. Please, stop it after fetching some files. Ten restart it back to be sure where we are. – Artem Bilan Jun 15 '20 at 16:41
  • 1
    Yes . Its working when I removed `controlChannel.send(new GenericMessage<>("@" + BEAN_ID + ".stop()"));` from `afterReceive()` its restarting and fetching new files. I used manual trigger for stopping **Sftp Inbound**. – Towfiqul Islam Jun 15 '20 at 17:33
  • 1
    Hope you got my logic and how I want to use stop **Sftp Inbound** after polling every directory at least one time. – Towfiqul Islam Jun 15 '20 at 17:35
  • Well, your logic is a bit not stable: when the first server doesn't have files to poll, it returns `null` for the advice and stop polling over here without any chances to let another server to take a look into its files. We definitely don't poll all the servers at once: we move to the next one only on the next poll. Also pay attention to the `fair` option of the `RotatingServerAdvice`. – Artem Bilan Jun 15 '20 at 17:43
  • Well, I got your point. I also tried with `fair` option. But I need the system to start on demand and fetch file completing one cycle. If it is not stopped after that, it will continue polling and won't stop and my system will fall into an infinite loop. Is there any way to get that when the `RotatingServerAdvice` is completing polling from all servers at least one? Does it throw any event or something like that ? Otherwise I have to stop here with **Sftp Inbound Adapter**. Thanks. – Towfiqul Islam Jun 15 '20 at 18:13

1 Answers1

1

I need the system to start on demand and fetch file completing one cycle. If it is not stopped after that, it will continue polling and won't stop and my system will fall into an infinite loop. Is there any way to get that when the RotatingServerAdvice is completing polling from all servers at least one? Does it throw any event or something like that ?

You probably misunderstood the logic with the afterReceive(@Nullable Message<?> result, MessageSource<?> source) contract. You can't stop a channel adapter for your requirements when one of the servers has returned nothing to poll. This way you don't give a chance for another server to poll on the next polling cycle.

I think your idea is to iterate over all the servers only once and then stop. Probably independently of the result from any of them. It looks like the best way to stop for you it is to use a RotatingServerAdvice with the fair = true to move to the next server every time. The stop might be performed from the custom afterReceive() independently of the result when you see that the RotationPolicy.getCurrent() is equal to a last one in the list. So, this way you iterate over all of them and stop moving the first one for the next poling cycle.

Artem Bilan
  • 113,505
  • 11
  • 91
  • 118
  • 1
    Do `afterReceive()` have anything like `RotationPolicy.getCurrent()`? Or where to find this `RotationPolicy.getCurrent()` ? – Towfiqul Islam Jun 15 '20 at 18:44
  • The `RotatingServerAdvice` has the `rotationPolicy` property. You need a custom `RotatingServerAdvice` with `afterReceive()` overridden to perform a logic to be sure that you are in a state to stop a channel adapter. – Artem Bilan Jun 15 '20 at 18:46