Camel SFTP,quartz2 component gives:Cannot retrieve file :caused by GenericFileOperationFailedException,when polling multiple files from sftp

Question

I'm trying to poll files from sftp location using the below camel route and using quartz2 scheduler, after polling couple of files i'm getting a GenricFileOperationFailedException caused by java.io.IOException: Pipe closed:

JaxbDataFormat dataFormat = new JaxbDataFormat();
dataFormat.setContext(JAXBContext.newInstance(MyBean.class));

    from(sftphost + sourcedirpath + 
                          "?password=" + password  
                        + "&username=" + userName  
                        + "&include=" + filePattern
                        + "&localWorkDirectory=" + localWorkingDirectory
                        + "&flatten=true"
                        + "&delete=true"
                        + "&scheduler=quartz2"
                        + "&scheduler.cron=0/45 * * * * ?" 
                        + "&stepwise=false"
                        + "&disconnect=true")
                .onException(Exception.class)
                    .to(sftphost + errdirpath + "?password=" + password + "&username=" + userName+"&disconnect=true")
                    .handled(true)
                    .end()
                .log(LoggingLevel.INFO, "Processing File : ${file:onlyname}").log(LoggingLevel.INFO, "Before Unmarshalling XML").unmarshal(dataFormat)
                .log(LoggingLevel.INFO, "after UnMarshalling the XML").beanRef("service", "processData(${body}, ${file:name})")
                .to(sftphost + processedDirPAth + "?password=" + password + "&username=" + userName+ "&disconnect=true");

I face this issue only when I use the quartz2 Scheduler. The above camel route works completely fine when I use the '&delay=10000' parameter in the camel route.

Error Stacktrace:

2016-04-22 10:33:15,634 | WARN  | terface_Worker-1 | SftpConsumer                    ]
    | ?                                     ? | 264 - 1.0.0.SNAPSHOT | Error processing file RemoteFile[/home/source/file_one.XML
    ] due to Cannot retrieve file: /home/source/file_one.XML. Caused by: [org.apache.camel.component.file.GenericFileOperat
    ionFailedException - Cannot retrieve file: /home/source/file_one.XML]
    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot retrieve file: /home/source/file_one.XML
            at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToFileInLocalWorkDirectory(SftpOperations.java:735)[264:myBundle:1.0.0.SNAPSHOT]
            at org.apache.camel.component.file.remote.SftpOperations.retrieveFile(SftpOperations.java:591)[264:myBundle:1.0.0.SNAPSHOT]
            at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:396)[264:myBundle:1.0.0.SNAPSHOT]
            at org.apache.camel.component.file.remote.RemoteFileConsumer.processExchange(RemoteFileConsumer.java:137)[264:myBundle:1.0.0.SNAPSHOT]
            at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:211)[264:myBundle:1.0.0.SNAPSHOT]
            at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:175)[264:myBundle:1.0.0.SNAPSHOT]
            at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)[264:myBundle:1.0.0.SNAPSHOT]
            at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)[264:myBundle:1.0.0.SNAPSHOT]
            at org.apache.camel.pollconsumer.quartz2.QuartzScheduledPollConsumerJob.execute(QuartzScheduledPollConsumerJob.java:59)[264:myBundle:1.0.0.SNAPSHOT]
            at org.quartz.core.JobRunShell.run(JobRunShell.java:202)[264:myBundle
    :1.0.0.SNAPSHOT]
            at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)[264:myBundle:1.0.0.SNAPSHOT]
    Caused by: 4:
            at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:1160)[264:myBundle:1.0.0.SNAPSHOT]
            at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:1006)[264:myBundle:1.0.0.SNAPSHOT]
            at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:984)[264:myBundle
    :1.0.0.SNAPSHOT]
            at org.apache.camel.component.file.remote.SftpOperations.retrieveFileToFileInLocalWorkDirectory(SftpOperations.java:724)[264:myBundle:1.0.0.SNAPSHOT]
            ... 10 more
    Caused by: java.io.IOException: Pipe closed
            at java.io.PipedInputStream.read(PipedInputStream.java:307)[:1.8.0_45]
            at java.io.PipedInputStream.read(PipedInputStream.java:377)[:1.8.0_45]
            at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2882)[264:myBundle:1.0.0.SNAPSHOT]
            at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java:2908)[264:myBundle:1.0.0.SNAPSHOT]
            at com.jcraft.jsch.ChannelSftp._get(ChannelSftp.java:1025)[264:myBundle:1.0.0.SNAPSHOT]
            ... 13 more

I have not used any other IO streams which are to be closed in the camel route.

May I ask what version of Camel are you using? this error looks very familiar to me.. — Ramin Arabbagheri, Apr 22 '16 at 20:57
Thanks for the response. The camel version i'm using is 2.15.2. — Nikhilesh Y, Apr 22 '16 at 21:49
@RaminArabbagheri I still have an issue, but I do not wanna connect to FTP after the my job is done — Nikhilesh Y, Apr 23 '16 at 21:29

score 0 · Answer 1 · answered Apr 22 '16 at 21:56

0

This looks like a permissions issue for the SFTP user on the target machine. Please review https://winscp.net/eng/docs/sftp_codes#code_4 for more details.

It's possible that it's only happening with the delay because the SSL context is not configured/initialized in the Camel FTP2 component immediately and you will need to delay a small time for that to happen before starting polling.

answered Apr 22 '16 at 21:56

Ninja Code Monkey

216
2
6

Thank you for your response, I get this issue only when I use the parameter 'scheduler=quartz2' and I do not get this issue when I use delay parameter '&delay=15000' in mycamel route. – Nikhilesh Y Apr 22 '16 at 22:17

Camel SFTP,quartz2 component gives:Cannot retrieve file :caused by GenericFileOperationFailedException,when polling multiple files from sftp

1 Answers1