camel ftp in recursive manner results in ftp error code: 226

Question

I have a very simple ftp route that should recursively download files from an URL. It would be very important to get it running still today.

There are no proxies, no extra authentication, no firewall. However, it only downloads the first file and then the socket will be closed. I have experimented with the different timeouts but they did not solve the issue. Either error code 226 is retuned if I don't use any timeout or extra config option or if I use the commented out things, error code 221 is returned. The 226 does not seem to be an error as it just counts as an indication that the server completed the transfer. Stack traces I copied below. I would appreciate the responses and thanks also in advance.

The route I am using in pollEnrich as I have to start it depending on a timer.

The code:

[...]
                        from("direct:ftp").routeId("ftp")
                                .log("### FTP is in progress ")
                                .pollEnrich().simple(ConfigData.getConfigData().getFtpUrl() 
                                        + "?binary=true&"
                                        + "recursive=true"
/*                                        + "soTimeout=300000&"
                                        + "stepwise=true&"
                                        + "ignoreFileNotFoundOrPermissionError=true&"
                                        + "ftpClient.dataTimeout=30000&"
                                        + "disconnect=true&"
                                        + "consumer.delay=1000" */
                                                    )
                                .to("file:modelFiles")
                                .end();

[...]

Update 1

I removed pollEnrich() and without it it works fine. However, I cannot start it from another route e.g. from a timer. So this is a quick hack to get FTP running. I also copy the code here as I did it with an idempotent consumer, so that only those files will be downloaded that are not yet on the disk. It can maybe be useful also to other people. The idempotent consumer example you can find here (Apache Camel ftp consumer loads the same files again and again) for more information.

Any further comments?

from(ConfigData.getConfigData().getFtpUrl() 
            + "?binary=true&"
            + "recursive=true&"
            + "passiveMode=true&"
            + "ftpClient.bufferSize=10000000&"
            + "localWorkDirectory=" + ConfigData.getConfigData().getLocalTmpDirectory())
    .idempotentConsumer(header("CamelFileName"), FileIdempotentRepository.fileIdempotentRepository(new File("data", "repo.dat")))
    .to("file:modelFiles")
    .log("Downloaded file ${file:name} complete.")
    .end();

Stack Trace

2016-03-19 15:27:53,921 [WARN|org.apache.camel.component.file.remote.FtpConsumer|MarkerIgnoringBase] Error processing file RemoteFile[2009-03-25/BioModels_Database-r13-sbml_files.tar.gz] due to File operation failed: null socket closed. Code: 221. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - File operation failed: null socket closed. Code: 221]
org.apache.camel.component.file.GenericFileOperationFailedException: File **operation failed: null socket closed. Code: 221**
[...]
Caused by: java.net.SocketException: socket closed
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(Unknown Source)
        at java.net.SocketInputStream.read(Unknown Source)
        at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
        at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
        at sun.nio.cs.StreamDecoder.read(Unknown Source)
        at java.io.InputStreamReader.read(Unknown Source)
        at java.io.BufferedReader.fill(Unknown Source)
        at java.io.BufferedReader.read(Unknown Source)
        at org.apache.commons.net.io.CRLFLineReader.readLine(CRLFLineReader.java:58)
        at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:314)
        at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:294)
        at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:483)
        at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:608)
        at org.apache.commons.net.ftp.FTP.cwd(FTP.java:828)
        at org.apache.commons.net.ftp.FTPClient.changeWorkingDirectory(FTPClient.java:1128)
        at org.apache.camel.component.file.remote.FtpOperations.doChangeDirectory(FtpOperations.java:769)

2016-03-19 12:42:16,068 [WARN|org.apache.camel.component.file.remote.FtpConsumer|MarkerIgnoringBase] Error processing file RemoteFile[2008-12-03/BioModels_Database-r12-sbml_files.tar.gz] due to File operation failed: null Socket Closed. Code: 226. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - **File operation failed: null Socket Closed. Code: 226]**
[...]
Caused by: java.net.SocketException: Socket Closed
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(Unknown Source)
    at java.net.SocketInputStream.read(Unknown Source)
    at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
    at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
    at sun.nio.cs.StreamDecoder.read(Unknown Source)
    at java.io.InputStreamReader.read(Unknown Source)
    at java.io.BufferedReader.fill(Unknown Source)
    at java.io.BufferedReader.read(Unknown Source)
    at org.apache.commons.net.io.CRLFLineReader.readLine(CRLFLineReader.java:58)
    at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:314)
    at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:294)
    at org.apache.commons.net.ftp.FTP.getReply(FTP.java:692)
    at org.apache.commons.net.ftp.FTPClient.completePendingCommand(FTPClient.java:1813)
    at org.apache.commons.net.ftp.FTPClient._retrieveFile(FTPClient.java:1885)
    at org.apache.commons.net.ftp.FTPClient.retrieveFile(FTPClient.java:1845)
    at org.apache.camel.component.file.remote.FtpOperations.retrieveFileToStreamInBody(FtpOperations.java:367)

Solution:

As added in Upadete 1, I do not use pollEnrich(). The route cannot be started from a timer now but it works. So I will close the question. I really like the idea of the idempotent consumer (unrelated to the original question).

What version of Camel do you use? And you may want to set maxMessagesPerPoll=1 in your poll enrich to only poll 1 file. — Claus Ibsen, Mar 20 '16 at 07:31
See the FTP example if you want to download many files from a FTP server and save to file or another FTP server: http://camel.apache.org/ftp-example.html — Claus Ibsen, Mar 20 '16 at 07:31
Thanks for the comment. I had to get on with it yesterday. So after giving it a try, I got the whole pollEnrich out and just had it run without the timer. That way it works fine. I will check the maxMessagesPerPoll. I bumped in new problems while trying the extract the tar.gzip files downloaded but it is already a different problem with camel. I will write an update to my post. Actually, I also wanted to give bounty for the right answer but removed that part of the message for not finding the bounty button, however I have the bounty piviledge. Thanks nevertheless. — Tamas, Mar 20 '16 at 12:44
I tested it with maxMessagesPerPoll=1 in the FTP URI. In that case there was no exception but just one of the files from the subtree was downloaded. This did not change after introducing consumer.delay=1000. — Tamas, Mar 20 '16 at 14:51

score 0 · Answer 1 · answered Mar 21 '16 at 10:38

0

Don't forget that a pollEnrich delete the oldExchange and keep the newExchange.

So you don't want to do a from("ftp").pollEnrich("timer") but a from("timer").pollEnrich("ftp").

answered Mar 21 '16 at 10:38

Jonathan Schoreels

1,660
1
12
20

Thanks Jonathan, The first problem is that the FTP does not seem to work over pollEnrich: from("smg").pollEnrich().simple().to("file:directoryforthefiles"). If it worked, I could route a timer to the route. – Tamas Mar 22 '16 at 13:02
Do you tried using the right way to put simple in endpoint : .pollEnrich(simple("yoururlwithsimplevalue")).to ? – Jonathan Schoreels Mar 22 '16 at 13:05
I construct the URL in the simple expression. It works for the first item in the subtree to download. It is downloaded all right. Then at starting to download the next item, the exception occures. – Tamas Mar 22 '16 at 13:24

score 0 · Accepted Answer · answered Mar 25 '16 at 11:17

0

See the Solution at the end of the question.

answered Mar 25 '16 at 11:17

Tamas

736
2
11
27

camel ftp in recursive manner results in ftp error code: 226

The code:

Update 1

Stack Trace

2 Answers2