Sftp Outbound Gateway multiple delete request getting stuck after first "File not found" issue

Question

I am trying to delete multiple files over Sftp and using the Spring Integration SftpOutboundGateway approach.

I am using QueueChannel for request and response. Also I am going Async way so that I can submit multiple request. I have also added an error channel recently.

Now the issue is that my first request is for a non-existing file, so I am fine to get a "2: No such file" exception message. But once that happens the other requests get stuck.

Following is the error message for the file not found case :

DEBUG o.s.integration.util.SimplePool - Obtained new org.springframework.integration.sftp.session.SftpSession@4ec427c0.
DEBUG o.s.i.f.r.s.CachingSessionFactory - Releasing Session org.springframework.integration.sftp.session.SftpSession@4ec427c0 back to the pool.
INFO  com.jcraft.jsch - Disconnecting from xxxx
DEBUG o.s.integration.util.SimplePool - Releasing org.springframework.integration.sftp.session.SftpSession@4ec427c0 back to the pool
INFO  com.jcraft.jsch - Caught an exception, leaving main loop due to Socket closed
Caused by: org.springframework.messaging.MessageHandlingException: error occurred in message handler [sftpDeleteFileHandler]; nested exception is org.springframework.messaging.MessagingException: Failed to execute on session; nested exception is org.springframework.core.NestedIOException: Failed to remove file.; nested exception is 2: No such file, failedMessage=GenericMessage [xxx]
    at org.springframework.integration.support.utils.IntegrationUtils.wrapInHandlingExceptionIfNecessary(IntegrationUtils.java:189)
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:186)
    at org.springframework.integration.endpoint.PollingConsumer.handleMessage(PollingConsumer.java:143)
    at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:390)
    at org.springframework.integration.endpoint.AbstractPollingEndpoint.pollForMessage(AbstractPollingEndpoint.java:329)
    at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$null$1(AbstractPollingEndpoint.java:277)
    at org.springframework.integration.util.ErrorHandlingTaskExecutor.lambda$execute$0(ErrorHandlingTaskExecutor.java:57)
    at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
    at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:55)
    at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$createPoller$2(AbstractPollingEndpoint.java:274)
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
    at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    ... 1 more
Caused by: org.springframework.messaging.MessagingException: Failed to execute on session; nested exception is org.springframework.core.NestedIOException: Failed to remove file.; nested exception is 2: No such file
    at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:446)
    at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.doRm(AbstractRemoteFileOutboundGateway.java:566)
    at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.handleRequestMessage(AbstractRemoteFileOutboundGateway.java:459)
    at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:123)
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:169)
    ... 17 more
Caused by: org.springframework.core.NestedIOException: Failed to remove file.; nested exception is 2: No such file
    at org.springframework.integration.sftp.session.SftpSession.remove(SftpSession.java:83)
    at org.springframework.integration.file.remote.session.CachingSessionFactory$CachedSession.remove(CachingSessionFactory.java:225)
    at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.rm(AbstractRemoteFileOutboundGateway.java:586)
    at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.lambda$doRm$7(AbstractRemoteFileOutboundGateway.java:566)
    at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:437)
    ... 21 more
Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2873)
    at com.jcraft.jsch.ChannelSftp.rm(ChannelSftp.java:1985)
    at org.springframework.integration.sftp.session.SftpSession.remove(SftpSession.java:79)
    ... 25 more

Update 1 :

I am using Spring Boot 2.1.8 -> Spring Integration 5.1.7

Config:

    @Bean
    public SessionFactory<LsEntry> sftpSessionFactory() {
        DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
        factory.setHost(host);
        factory.setPort(port);
        factory.setUser(user);
        factory.setPassword(password);
        factory.setAllowUnknownKeys(true);
        return new CachingSessionFactory<LsEntry>(factory);
    }

    @Bean(name = PollerMetadata.DEFAULT_POLLER)
    public PollerMetadata defaultPoller() {

        PollerMetadata pollerMetadata = new PollerMetadata();
        pollerMetadata.setTrigger(new PeriodicTrigger(10));
        return pollerMetadata;
    }

    @Bean(name = "sftp.file.delete.request.channel")
    public MessageChannel sftpFileDeleteRequestChannel() {
        return new QueueChannel();
    }

    @Bean(name = "sftp.file.delete.response.channel")
    public MessageChannel sftpFileDeleteResponseChannel() {
        return new QueueChannel();
    }

    @Bean(name = "sftp.error.channel")
    public MessageChannel sftpErrorChannel() {
        return MessageChannels.queue("sftp.error.channel").get();
    }

    @Bean
    @ServiceActivator(inputChannel = "sftp.file.delete.request.channel", async = "true")
    public MessageHandler sftpDeleteFileHandler() {
        SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory(), Command.RM.getCommand(),
                "headers['file_remoteDirectory'] + headers['file_remoteFile']");
        sftpOutboundGateway.setRequiresReply(true);
        return sftpOutboundGateway;
    }

    @ServiceActivator(inputChannel = "sftp.error.channel")
    public void sftpErrorHandler(final Message<MessageHandlingException> excpMessage) {
        log.error(excpMessage.getPayload().getCause());
    }

    @MessagingGateway(errorChannel = "sftp.error.channel")
    public interface SftpDeleteMessagingGateway {
        @Gateway(requestChannel = "sftp.file.delete.request.channel", replyChannel = "sftp.file.delete.response.channel")
        CompletableFuture<Message<Boolean>> deleteFile(final Message<Boolean> message);
    }

Code:

List<CompletableFuture<Message<Boolean>>> fileDeleteResults = new ArrayList<>();

foreach(...) {
        Message<Boolean> fileDeleteRequest = MessageBuilder.withPayload(true)
.setHeader(FileHeaders.REMOTE_DIRECTORY, directory)
.setHeader(FileHeaders.REMOTE_FILE, name).build();

 fileDeleteResults.add(sftpDeleteMessagingGateway.deleteFile(fileDeleteRequest));

}

try {
       CompletableFuture.allOf(fileDeleteResults.toArray(new CompletableFuture[fileDeleteResults.size()])).join();

       for (CompletableFuture<Message<Boolean>> fileDeleteResult : fileDeleteResults){

                Message<Boolean> message = fileDeleteResult.get();
                log.debug((String) message.getHeaders().get(FileHeaders.REMOTE_DIRECTORY)
                             + (String) message.getHeaders().get(FileHeaders.REMOTE_FILE) 
                             + ": " + message.getPayload());

                }
    } catch (CompletionException | InterruptedException | ExecutionException excp) {
            log.error(excp);
      }

Update 2 :

I modified the config as per the suggestion, but still facing the same issue. Below is the modified config -

    @Bean
    @ServiceActivator(inputChannel = "sftp.file.delete.request.channel")
    public MessageHandler sftpDeleteFileHandler() {
        SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory(), Command.RM.getCommand(),
                "headers['file_remoteDirectory'] + headers['file_remoteFile']");
        sftpOutboundGateway.setRequiresReply(true);
        return sftpOutboundGateway;
    }

    @ServiceActivator(inputChannel = "sftp.error.channel")
    public boolean sftpErrorHandler(final Message<MessageHandlingException> excpMessage) {
        log.error(excpMessage.getPayload().getCause());
        return false;
    }

I added Spring versions, config and code in the main question. I also added error channel recently. — Chimu, Sep 19 '19 at 01:41
Just want to give an update that I have tested the construct with single valid delete request and it worked fine. — Chimu, Sep 19 '19 at 02:10
This is wrong construction: `+ (String) message.getHeaders().get(FileHeaders.REMOTE_FILE), + ": " + message.getPayload()`. I can't compile it. What was your idea here? Why do we need that coma? — Artem Bilan, Sep 19 '19 at 14:02
What is that `foreach(...) {`. Doesn't look like valid Java construction... — Artem Bilan, Sep 19 '19 at 14:04
The "," was just a typing mistake, I corrected that. The foreach(..) is a placeholder. I am actually looping on a output from database, formatted in a way that suits me. I just omitted that part as I thought that might not be any use in this issue. — Chimu, Sep 19 '19 at 14:34

score 0 · Accepted Answer · answered Sep 19 '19 at 14:15

0

OK. Looks like your problem that you don't catch exceptions to go ahead.

See async option on the ServiceActivatingHandler:

/**
 * Allow async replies. If the handler reply is a {@link ListenableFuture}, send
 * the output when it is satisfied rather than sending the future as the result.
 * Ignored for return types other than {@link ListenableFuture}.
 * @param async true to allow.
 * @since 4.3
 */
public final void setAsync(boolean async) {

So, it is really an async only if a ListenableFuture (or Reactive Publsiher, though) is returned from the target handleRequestMessage() implementation. It is not a fact of the SftpOutboundGateway. Therefore your errorChannel = "sftp.error.channel" on the gateway definition is the right way to go. Although you need to return something from that sftpErrorHandler, which is going to be a return for the gateway call. Otherwise we are stuck waiting for reply or error.

answered Sep 19 '19 at 14:15

Artem Bilan

113,505
11
91
118

Thanks a lot for the clarification. I just have two questions regarding that. - Me using "CompletableFuture" is not actually making it async ? - What kind of object I can return from "sftpErrorHandler" ? Is that needs to be same as my gateway call return type ? – Chimu Sep 19 '19 at 14:39
Yes, it must be the same return type. So, in your case it is `boolean`. I guess errorHandler must return `false` since you have not deleted file. – Artem Bilan Sep 19 '19 at 14:46
Well, `CompletableFuture` made is as an `async. That's true, but it doesn't rely on the return type of the target message handler. – Artem Bilan Sep 19 '19 at 14:49
No, it is not. See https://docs.spring.io/spring-integration/reference/html/#gw-completable-future in the Docs. Also keep in mind that `async` on the `@ServiceActivator` has a value only if your return `ListenableFuture`, otherwise it is ignored. I just mean you don't need it since it doesn't help you with the `SftpOutboundGateway`. – Artem Bilan Sep 19 '19 at 14:52
Thanks for making me understand the async part. I removed that and added a boolean return from the error handler. But still after the first error, the polling stops and the processing of rest of the requests get stuck. – Chimu Sep 19 '19 at 15:06
Works for me well. Share, please, with us a simple Spring Boot project to let us to play with. Don't worry about SFTP server over there: I can spin an embedded one based on Apache Mina. – Artem Bilan Sep 19 '19 at 15:10
Will do. Please let me know how to share a project. – Chimu Sep 19 '19 at 15:23
Via GitHub. You check it in and we check it out. – Artem Bilan Sep 19 '19 at 15:32
I moved to Direct Channel as async is out of picture now and things worked. Is there anything wrong happening with the Poller, like it is failing due to this file not found error. I am suspecting that because while using queue and poller it didn't work and also the polling stopped after the first error. – Chimu Sep 19 '19 at 15:49
It must not, but yeah... Errors are really go into the `errorChannel` of the poller. And it is global one just with the `logger` subscriber. However things must go into your `errorChannel = "sftp.error.channel"` anyway and you should handle it properly there. – Artem Bilan Sep 19 '19 at 15:52
Is there a way to setup the error channel in the Poller Metadata bean ```@Bean(name = PollerMetadata.DEFAULT_POLLER) public PollerMetadata defaultPoller() { PollerMetadata pollerMetadata = new PollerMetadata(); pollerMetadata.setTrigger(new PeriodicTrigger(10)); return pollerMetadata; }``` – Chimu Sep 19 '19 at 15:58
There is an `errorHandler` option and you can consider to use a `MessagePublishingErrorHandler` for that purpose. Although I can recommend to take a look into Java DSL with its `Pollers` factory. – Artem Bilan Sep 19 '19 at 16:01
I have accepted the answer. Thanks a lot for giving me the right direction. Not returning any message from the error handler was actually the main issue. Currently I am going with a combination of Queue and Direct channel as that will serve my purpose. – Chimu Sep 20 '19 at 22:04

Sftp Outbound Gateway multiple delete request getting stuck after first "File not found" issue

1 Answers1