1

Requirement is to not to download 1 GB File into memory but to download file as stream of XML.

private String sftpComponent(String sftpFileName) {
    return "sftp://xxx/xxx"
            + "?username=xxx"
            + "&password=xxx"
            + "&streamDownload=true" 
            + "&useList=true"
            + "&stepwise=false"
            + "&disconnect=true"
            + "&passiveMode=true"
            + "&reconnectDelay=10000"
            + "&bridgeErrorHandler=true"
            + "&delay=30000"
            + "&include="+ sftpFileName
            + "&preMove=$simple{file:onlyname}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.underproc"
            + "&move=$simple{file:onlyname.noext}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.done"
            + "&moveFailed=$simple{file:onlyname.noext}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.error"
            + "&readLock=idempotent-changed"
            + "&idempotentRepository=#infinispanCache"
            + "&readLockRemoveOnCommit=true";
}

from(sftpComponent(sftpFileName))
    .split(body().tokenizeXML("case", "cases"))
    .streaming()
    .inOnly("jms/caseQueue")

<cases> 
    <case>
    </case>
    <case>
    </case> 
</cases>

The file is successfully picked and marked as success but flow does not split the content and move it to caseQueue.

When i use "download=true" then it works like charm.

Can anyone suggest what more settings are required in this case or alternative approach to fix above?

fatherazrael
  • 5,511
  • 16
  • 71
  • 155
  • Your code has "Case" and "Cases" but your XML has "cases" and "case". Ironically, it is case sensitive... :-) By the way - why do you have .streaming() twice after the split? – Screwtape Oct 17 '19 at 13:12
  • @Screwtape: Corrected question. (Sometimes we change xml a bit for privacy concerns and typo left). I will take care of this in future. Could you please suggest an answer to this – fatherazrael Oct 18 '19 at 04:32
  • Reading https://camel.apache.org/components/latest/sftp-component.html, it appears that SFTP has download=false as a default, and therefore reads the file details but provides a null body. Therefore this is doing what the manual says it would do. Are you finding that when you have download=true that you are not getting the file streamed, but fully loaded into memory or a temporary file before processing? – Screwtape Oct 18 '19 at 10:15
  • @Screwtape: I have checked it on local and found that it is fetching data successfully and printing when checked using {$body} but when it does not send it to queue. – fatherazrael Oct 18 '19 at 11:11
  • @Screwtape: Also do you think that infinispan cache is any hurdle here? – fatherazrael Oct 18 '19 at 11:13
  • 1
    That’s going beyond my pay grade! – Screwtape Oct 18 '19 at 18:23

0 Answers0