Moving processed files in remote S(ftp) using Java DSL

Question

I'm trying to move files on remote SFTP once the batch has successfully processed the files using Spring integration and Java DSL.

What would be the best way to achieve that?

Adding a step in batch to move remote files ?
Or using FTP Outbound Gateway and provide the MV command ?

I tend to prefer the second solution and let the batch focus on the logic only, but I've hard times trying to implement it with java dsl.

I've read http://docs.spring.io/spring-integration/reference/html/ftp.html#ftp-outbound-gateway and tried to implement like that :

@Bean
public MessageHandler ftpOutboundGateway() {
    return Sftp.outboundGateway(SftpSessionFactory(), 
            AbstractRemoteFileOutboundGateway.Command.MV, "payload")
            .localDirectory(new File("/home/blabla/"))
            .get();

}

@Bean
public IntegrationFlow ftpInboundFlow() {
    return IntegrationFlows
            .from(
                Sftp.inboundAdapter(SftpSessionFactory())
                .regexFilter(".*\\.xml.mini$")
                ...             
               , 
                e -> e.id("sftpInboundAdapter")
                .poller(
                        Pollers.fixedRate(intCfg.getSftpPollerInMinutes(), TimeUnit.MINUTES)
                        .maxMessagesPerPoll(-1)
                        .advice(retryAdvice())
                        )
            )
            .enrichHeaders( h -> h
                    .header(FileHeaders.REMOTE_DIRECTORY,"/home/filedrop/")
                    .header(FileHeaders.REMOTE_FILE, "/home/filedrop/OFFERS.xml.mini")
                    .header(FileHeaders.RENAME_TO, "/home/filedrop/done/OFFERS.xml.mini")
            )
            .transform(fileToJobLaunchRequestTransformer())         
            .handle(jobLaunchingGw()))
            .transform(jobExecutionToFileStringTransformer())
            .handle(ftpOutboundGateway())
            .handle(logger())
            .get();
}

I know my headers should be dynamic but I don't know how to do it so for now I use the name of an existing file. I get this error message (he's trying to delete the file in the destination directory!):

Caused by: org.springframework.messaging.MessagingException: Failed to execute on session; nested exception is org.springframework.core.NestedIOException: Failed to delete file /home/filedrop/done/OFFERS.xml.mini; nested exception is org.springframework.core.NestedIOException: Failed to remove file: 2: No such file     at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:343)
    at org.springframework.integration.file.remote.RemoteFileTemplate.rename(RemoteFileTemplate.java:290)
    at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.doMv(AbstractRemoteFileOutboundGateway.java:482)
    at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.handleRequestMessage(AbstractRemoteFileOutboundGateway.java:400)
    at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:99)
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
    at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
    ... 94 more
 Caused by: org.springframework.core.NestedIOException: Failed to delete file /home/filedrop/done/OFFERS.xml.mini; nested exception is org.springframework.core.NestedIOException: Failed to remove file: 2: No such file
    at org.springframework.integration.sftp.session.SftpSession.rename(SftpSession.java:211)
    at org.springframework.integration.file.remote.RemoteFileTemplate$3.doInSessionWithoutResult(RemoteFileTemplate.java:300)
    at org.springframework.integration.file.remote.SessionCallbackWithoutResult.doInSession(SessionCallbackWithoutResult.java:34)
    at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:334)
    ... 100 more
 Caused by: org.springframework.core.NestedIOException: Failed to remove file: 2: No such file
    at org.springframework.integration.sftp.session.SftpSession.remove(SftpSession.java:83)
    at org.springframework.integration.sftp.session.SftpSession.rename(SftpSession.java:205)
    ... 103 more

Thanks for any help!

EDIT working flow, I've then simplified it a lot but here the solution of myprevious problem:

@Bean
public IntegrationFlow ftpInboundFlow() {
    return IntegrationFlows
            .from(
                Sftp.inboundAdapter(SftpSessionFactory())
                .regexFilter(".*\\.xml$")
                ...
                , 
                e -> e.id("sftpInboundAdapter")
                .poller(Pollers.fixedRate(intCfg.getSftpPollerInMinutes(), TimeUnit.MINUTES)
                        .maxMessagesPerPoll(-1)
                        )
            )
            .enrichHeaders( h -> h
                    // headers necessary for moving remote files (ftpOutboundGateway)
                    .headerExpression(FileHeaders.RENAME_TO, "'/home/blabla/done/' + payload.getName()")
                    .headerExpression(FileHeaders.REMOTE_FILE, "payload.getName()")
                    .header(FileHeaders.REMOTE_DIRECTORY,"/home/blabla/")
                    // headers necessary for moving local files (fileOutboundGateway_MoveToProcessedDirectory)
                    .headerExpression(FileHeaders.ORIGINAL_FILE,  "payload.getAbsolutePath()" )
                    .headerExpression(FileHeaders.FILENAME,  "payload.getName()")
            )
            .transform(fileToJobLaunchRequestTransformer())         
            .handle(jobLaunchingGw(), e-> e.advice(retryAdvice()))

            .<JobExecution, Boolean>route(p -> BatchStatus.COMPLETED.equals(p.getStatus()),
                                            mapping -> mapping
                                            .subFlowMapping("true", sf -> sf


                                                .handle(org.springframework.batch.core.JobExecution.class,
                                                         (p, h) -> myServiceActivator.jobExecutionToString(p, 
                                                                 (String) h.get(FileHeaders.REMOTE_DIRECTORY),
                                                                 (String) h.get(FileHeaders.REMOTE_FILE)))
                                                .handle(ftpOutboundGateway())
                                                .handle(Boolean.class,
                                                         (p, h) -> myServiceActivator.BooleanToString(p, 
                                                                 (String) h.get(FileHeaders.FILENAME)))
                                                .handle(fileOutboundGateway_MoveToProcessedDirectory())

                                                                                    )


                                        .subFlowMapping("false", sf -> sf
                                            .channel("nullChannel")     

                                            )
            )

            .handle(logger())
            .get();
}

@Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
    return Pollers.fixedRate(500).get();
}


@Bean
public MessageHandler ftpOutboundGateway() {
    return Sftp
            .outboundGateway(SftpSessionFactory(),
                    AbstractRemoteFileOutboundGateway.Command.MV,
                    "payload")
            .renameExpression("headers['file_renameTo']").get();
}

I have quite the same task to accomplish. Can you better describe the solution ("answer to your own question" mayben a better "stack-overflow way" to do it.). Thnks! — lrkwz, Aug 04 '15 at 17:42
Where does myServiceActivator come from? Which dependencies in your pom? Thank you — lrkwz, Aug 04 '15 at 17:43
Please don't use this code to do what you want, it's not working and I'm not doing that anymore. to answer your question myServiceActivator is a component with an annotated @ServiceActivator method and is autowired inside my flow. for the pom you need spring-integration-sftp spring-integration-file spring-integration-java-dsl. Please also have a look to https://github.com/spring-projects/spring-integration-java-dsl/wiki/Spring-Integration-Java-DSL-Reference . I may write an article later on this topic and update this post. In meantime best of luck. — landbit, Aug 05 '15 at 08:34

score 1 · Accepted Answer · answered Jul 01 '15 at 13:30

1

Perhaps you don't have permissions to do the rename, or the rename failed for some other reason; this exception is an attempt to remove the "to" filename because an initial rename failed. Turn on DEBUG logging and you should see this log...

if (logger.isDebugEnabled()){
    logger.debug("Initial File rename failed, possibly because file already exists. Will attempt to delete file: "
            + pathTo + " and execute rename again.");
}
try {
    this.remove(pathTo);

Since the failure is on this remove() operation, your failure indicates that the rename failed due to some other reason (because clearly the "to" file does not exist).

answered Jul 01 '15 at 13:30

Gary Russell

166,535
14
146
179

Thanks, the thing is I didn't know what to blame wrong path? permissions? flow configuration? I then implement it in XML where documentation and samples are far better than Java dsl (examples are sometimes too simple imo for new comers, also still confused with `@Transformer`, `.transform()`, `handle()` and `@ServiceActivator` what is the best practice to transform a message ), somehow it makes me understand few things and it was then easier to switch to java dsl. I've updated my post hoping that could help someone in future. Anyway thanks for making developers life easier :) – landbit Jul 04 '15 at 09:47
The XML has been around a lot longer; we are trying to improve the Java Config/DSL documentation as time permits. If you have any suggestions for documentation or sample improvements, create a JIRA [here](https://jira.spring.io/browse/INT) or [here](https://jira.spring.io/browse/INTSAMPLES) respectively. It would be even better if you'd like to [make a contribution](https://github.com/spring-projects/spring-integration/blob/master/CONTRIBUTING.md). – Gary Russell Jul 04 '15 at 14:05
Dsl seems much better than the xml configuration as far it simplifies it ... but it lacks documentation from a novice point of view. Real world examples such this one would be of great help (things like "read file from ftp, process file line by line, send confirmation mail, move file on remote ftp") – lrkwz Aug 04 '15 at 17:45
Good idea - please open a [JIRA issue](https://jira.spring.io/browse/INTEXT) (JavaDSL component) and we'll take a look at it as time permits. – Gary Russell Aug 04 '15 at 18:16

score 1 · Answer 2 · edited May 23 '17 at 12:31

1

Take a look at Move file after successful ftp transfer using Java DSL where I deleted the file in the ftpInbound and the reuploaded in the new directory after processing.

edited May 23 '17 at 12:31

Community

1
1

answered Aug 04 '15 at 21:06

lrkwz

6,105
3
36
59

Moving processed files in remote S(ftp) using Java DSL

2 Answers2

Linked