Apache Camel: How to download multiple files from SFTP with Premove, Move and MoveFailed options?

Question

I have two file names in following format:

Gontagrator_1.xml
Gontagrator_2.xml

As of now i am picking just Gontagrater_1.xml and rename it to processing and failed once done.

 + "&fileName=" + sftpFileName
            + "&preMove="+sftpFileName+".$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.processing"
            + "&move="+sftpFileName+".$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}"
            + "&moveFailed="+sftpFileName+".$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.failed"

Now Gontagrator_2 comes into picture. It is suggested to create separate routes for both.

Can we download both in one route and rename accordingly? If yes, what values i need to pass?

Update1: There are multiple files with different names but i need to use both above file names only

update 2: Whole from component is:

    "{{m.protocol}}://{{m.hostname}}{{t.directory}}"
            + "?username={{m.username}}"
            + "&password={{m.password}}"
            + "&download=true"
            + "&useList=false"
            + "&stepwise=false"
            + "&disconnect=true"
            + "&passiveMode=true"
            + "&reconnectDelay=10000"
            + "&bridgeErrorHandler=true"
            + "&delay=30000"
            //+ "&fileName=" + sftpFileName
            + "&include="+ sftpFileName
            + "&preMove=$simple{file:onlyname}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.processing"
            + "&move=$simple{file:onlyname}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}"
            + "&moveFailed=$simple{file:onlyname}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.failed"
            + "&readLock=idempotent-changed"
            + "&idempotentRepository=#infinispan"
            + "&readLockRemoveOnCommit=true")
        .onException(GenericFileOperationFailedException.class)
            .onWhen(exchange -> { 
                    Throwable cause = exchange.getException(GenericFileOperationFailedException.class).getCause();
                    return (cause != null && cause.toString().equalsIgnoreCase("2: No such file"));
                }).handled(true)
                .logExhausted(true)
                .logExhaustedMessageHistory(true)
                 .log("Could not find file")
                .end()
        .log("Downloading xml file")
        //.to(archiveReceivedFile(sftpFileName))       
        .split(body().tokenizeXML("ERequest", "ERequests")).streaming()
            .inOnly(E_QUEUE)

hk6279 · Accepted Answer · 2019-08-16T08:40:28.417

2

Yes, you can download all files in one route.

You need to

Remove fileName option (such that the route will pick all file)

Use file expression language (file:onlyname) to refer to filename currently handled by Camel SFTP component

+ "&preMove=$simple{file:onlyname}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.processing"
+ "&move=$simple{file:onlyname}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}"
+ "&moveFailed=$simple{file:onlyname}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.failed"

Use include option to control file to be picked up (REGEX to match file name pattern)
```
+ "&include=Gontagrator_(1|2)\.xml"
```

edited Aug 16 '19 at 08:40

answered Aug 16 '19 at 08:19

hk6279

1,881
2
22
32

So i pass it as $simple{file:Gontagrator_1.xml, Gontagrator_2.xml}. Is this understanding correct? I do not want it to pick any other file – fatherazrael Aug 16 '19 at 08:23
If you have many files inside the folder, but only want to get the two files (Gontagrator_1.xml and Gontagrator_2.xml), you could use `include` option to limit the file to be picked up by file name pattern – hk6279 Aug 16 '19 at 08:28
It appears regex will be -> "&include="+sftpFileName+"_*.xml" where sftpFileName will be Gontagrator – fatherazrael Aug 16 '19 at 08:36
It is base on your example, you could define variable to make it more dynamic. – hk6279 Aug 16 '19 at 08:40
It says ""Caused by: java.lang.IllegalArgumentException: Endpoint is configured with useList=false, then fileName must be configured also"". I have updated whole From Component. Setting useList to true – fatherazrael Aug 16 '19 at 08:53
1

Change the `useList` to `true`, you need the list command to help you to scan on server – hk6279 Aug 16 '19 at 08:58
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/198038/discussion-between-fatherazrael-and-hk6279). – fatherazrael Aug 16 '19 at 09:52
I think it worked with issue that after successful, file name becomes -> Gontagrator_2.xml.2019-08-16T11-23-58.processing.2019-08-16T11-24-04 . It appears {File:onlyname} is taking previous name and appending date to it – fatherazrael Aug 16 '19 at 09:53
Another issue is when both Files are in place then it picks another say Gontagrator_2 and rename it to Gontagrator_1 and Gontagrator_1 got vanishes – fatherazrael Aug 16 '19 at 10:09
1

For filename problem, maybe you can use below workaround. Replace `$simple{file:onlyname}` by `$simple{file:onlyname.noext}.xml` in both option `move` and `moveFailed`. – hk6279 Aug 19 '19 at 02:26
I am sorry asking few more related questions. 1) as per your logic, 2 Routes created successfully. But it is picking "Gontagrator_2.xml" and removing it from SFTP and file is gone where as "Gontagrator_1.xml" is still there. No error is shown in logs. Do you know why this is happening?.... 2) streamDownload option is not working with set up you told and only download option is working and one of file is >400MB. Any suggestions here? – fatherazrael Aug 20 '19 at 04:09
For (1), maybe it is cause by competition from multiple node, you might try create a testing server and have the server log to check what happen at server side. For (2), not sure what you mean by not working, but you could also try option `localWorkDirectory` – hk6279 Aug 20 '19 at 07:59

Apache Camel: How to download multiple files from SFTP with Premove, Move and MoveFailed options?

1 Answers1