Recursively read files with Spring Integration SFTP DSL

Question

I am trying to make Spring Integration SFTP read files (.txt) from a remote server recursively from all subfolders. The remote folder is something like "/tmp/remoteFolder" and all subfolders are date folders like "/tmp/remoteFolder/20180830", "/tmp/remoteFolder/20170902".

This is the code that I have until now

@Bean
@InboundChannelAdapter(value = "sftpMgetInputChannel",
    poller = @Poller(fixedDelay = "5000"))
public IntegrationFlow sftpMGetFlow() {
    return IntegrationFlows.from("sftpMgetInputChannel")
            .handleWithAdapter(h -> h.sftpGateway(this.sftpSessionFactory,
             Command.MGET, "'/tmp/remoteDirectory/*'")
            .options(Option.RECURSIVE)
            .regexFileNameFilter("((\\d{8})|*\\.txt)")
            .localDirectoryExpression("sftp-inbound" + "/" + "#remoteDirectory"))
            .channel(remoteFileOutputChannel())
            .get();
}

@Bean
public MessageChannel sftpMgetInboundChannel(){
   return new DirectChannel();
}

@Bean
public PollableChannel remoteFileOutputChannel() {
    return new QueueChannel();
}

How do I specify the root remote directory for sftp mget to be /tmp/remoteFolder? Why isn't this working? Why do I need to specifiy the output channel?

Update: Instead of calling channel(remoteFileOutputChannel()) I call a handler like this

@Bean
public MessageHandler messageHandler(){
 return new MessageHandler() { ... }
}

Code updated:


    @InboundChannelAdapter(value = "sftpMgetInputChannel",
        poller = @Poller(fixedDelay = "5000"))
    public String filesForMGET(){
      return "'/tmp/input/remoteDirectory/*'";
    }

    @Bean
    public IntegrationFlow sftpMGetFlow() {
        return IntegrationFlows.from("sftpMgetInputChannel")
                .handleWithAdapter(h -> h.sftpGateway(this.sftpSessionFactory,
                 Command.MGET, "payload")
                .options(Option.RECURSIVE)
                .regexFileNameFilter("((\\d{8})|*\\.txt)")
                .localDirectoryExpression("'sftp-inbound/'" + "#remoteDirectory"))
                .handler(messageHandler())
                .get();
    }

    @Bean
    public MessageChannel sftpMgetInboundChannel(){
       return new DirectChannel();
    }

    @Bean
    public MessageHandler messageHandler(){
     return new MessageHandler() { ... }
    }

With this updated code, I get the following error:


    rg.springframework.core.NestedIOException: failed to read file; nested exception is 2: No such file
        at org.springframework.integration.sftp.session.SftpSession.read(SftpSession.java:100)
        at org.springframework.integration.file.remote.session.CachingSessionFactory$CachedSession.read(CachingSessionFactory.java:137)
        at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.copyFileToLocalDirectory(AbstractInboundFileSynchronizer.java:176)
        at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:138)
        at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.receive(AbstractInboundFileSynchronizingMessageSource.java:144)
        at org.springframework.integration.endpoint.SourcePollingChannelAdapter.doPoll(SourcePollingChannelAdapter.java:89)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:146)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:144)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:207)
        at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
        at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
        at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)
        at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:202)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:51)
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:680)
    Caused by: 2: No such file
        at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2289)
        at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:1741)
        at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:1011)
        at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:986)
        at org.springframework.integration.sftp.session.SftpSession.read(SftpSession.java:96)
        ... 22 more

Gary Russell · Accepted Answer · 2017-09-11T13:39:22.530

1

With the expression set to payload (as was the case in your question before the edit), the message payload sent to the gateway should be /tmp/remoteFolder/* which internally is split into remote directory and remote filename (*).

Why do I need to specifiy the output channel?

The result of the MGET (list of retrieved files) needs to go somewhere.

EDIT

You misunderstond; you can't add the @InboundChannelAdapter annotation to the flow; you need something like this...

@InboundChannelAdapter(value = "sftpMgetInputChannel",
    poller = @Poller(fixedDelay = "5000"))
public String filesForMGET() {
    return "/tmp/remoteDirectory/";
}

@Bean
public IntegrationFlow sftpMGetFlow() {
    return IntegrationFlows.from("sftpMgetInputChannel")
            .handleWithAdapter(h -> h.sftpGateway(this.sftpSessionFactory,
             Command.MGET, "payload")
            .options(Option.RECURSIVE)
            .regexFileNameFilter("((\\d{8})|*\\.txt)")
            .localDirectoryExpression("sftp-inbound" + "/" + "#remoteDirectory"))
            .channel(remoteFileOutputChannel())
            .get();
}

edited Sep 11 '17 at 13:39

answered Aug 31 '17 at 11:47

Gary Russell

166,535
14
146
179

I've updated the code like it is now but I don't see any files being pulled from the remote server. Is * still necesary if I specify the subfolder regex? – 2dor Aug 31 '17 at 13:06
1

`Command.MGET, "/tmp/remoteDirectory/*")` No; the String is an expression; if you want a literal value (instead of using, say, `payload`), then add quotes: `Command.MGET, "'/tmp/remoteDirectory/*'")`. You can also [turn on DEBUG logging for JSCH](https://docs.spring.io/spring-integration/reference/html/sftp.html#sftp-jsch-logging) (and `org.springframework.integration`) to see the interaction with the server. – Gary Russell Aug 31 '17 at 13:12
I don't see any interaction. I did before a simple inbound adapter to get one file, not all recursively and that fetched the file and I could see interaction in the logs. Now nothing happens, I can't see even the ssh handshake. Do I need to add anything like autoStartup(true) anywhere? – 2dor Aug 31 '17 at 13:41
1

You have to send a message to the gateway to trigger the interaction; it's not polled like the inbound adapter. That's what I meant when I said "send a message with payload `/tmp/remoteFolder/*` (when the expression was `payload`). Send the message to `sftpMgetInputChannel`. – Gary Russell Aug 31 '17 at 13:46
1

The documentation is so low on quality on this one. so I can't have a listener on that location, a poller? I need to run this let's say one time per hour and fetch the files. I could do a scheduler but I would have liked to have a poller.cron on that integration flow. What exactly do I need to push to that MessageChannel to trigger the mget from /tmp/remoteDirectory? – 2dor Aug 31 '17 at 13:58
1

You can use an `@InboundChannelAdapter` which is polled to send a message to the gateway; or you can use a `MessagingGateway` to trigger the request; see the [SFTP Sample app](https://github.com/spring-projects/spring-integration-samples/tree/master/basic/sftp) for an example of the latter. – Gary Russell Aug 31 '17 at 14:08
I updated my code (see above) but the code still doesn't triggers the sftp connectivity. Do you have any advices for me? – 2dor Sep 11 '17 at 11:15
And actually my remote folder is /tmp/remote/remoteDirectory. And I managed to handle a response but /tmp/remote is set as remote directory and "remoteDirectory" is seen as the remote file. Why is that? – 2dor Sep 11 '17 at 13:24
Now I receive "Property of type 'sftp' cannot be found on object of type '...GenericMessage' - maybe not public?". Why would I receive this? I need to set that property somehow? – 2dor Sep 12 '17 at 12:05
Sounds like a problem with one of your expressions; you need to edit the question to show your current configuration, as well as a full stack trace. – Gary Russell Sep 12 '17 at 13:33
I've updated the code. It was because I didn't add the ' to the payload value. Now I get the error that the file is not there but it is there, the connection is succesfull and I think it's something else. What can it be? – 2dor Sep 12 '17 at 14:29
Remove the `'...'` from `return "'/tmp/input/remoteDirectory/*'";`. You are sending `'/tmp/input/remoteDirectory/*'` to the server instead of `/tmp/input/remoteDirectory/*` – Gary Russell Sep 12 '17 at 14:49
Gary, I tried putting the poller with a cron expresion to run once per 10 minutes. It activates after 10 minutes but after that it does not stop. How to get it to run once at a given time pattern? – 2dor Sep 27 '17 at 08:13
It's not clear what you mean; it's better to ask a new question showing your current configuration as well as some logs showing what you mean. – Gary Russell Sep 27 '17 at 12:46

Recursively read files with Spring Integration SFTP DSL

1 Answers1