Apache Camel ftp consumer loads the same files again and again

Question

I have following spring configuration

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
       http://www.springframework.org/schema/beans 
       http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
       http://camel.apache.org/schema/spring 
       http://camel.apache.org/schema/spring/camel-spring.xsd">

    <bean id="downloadLogger" class="com.thomsonreuters.oa.sdi.camel.DownloadLogger" />

    <bean id="fileFilter" class="com.thomsonreuters.oa.sdi.camel.IgnoreReadyFilesFilter" />

    <camelContext xmlns="http://camel.apache.org/schema/spring">
        <route>
            <from uri="ftp://url_to_ftp?password=*******&amp;noop=true&amp;stepwise=false&amp;binary=true&amp;consumer.delay=10s&amp;recursive=true&amp;filter=#fileFilter" />
            <process ref="downloadLogger" />
            <to uri="file:data/outbox" />
        </route>
    </camelContext>

</beans>

At the ftp side I have 3 folders with files which I want to download. I want to achieve following scenario:

On ftp is fixed amount of files (for isntance 5) at the first data pull consumer loads these files to the destination folder
At the second attempt to load files, ftp state still the same (5 files) and camel ftp consumer just does nothing (except check for new files)
To ftp arrives new 2 files, and at this data pull consumer downloads only these new two files

At the moment my current solutions downloads all files each time when I run dataload process, how I can manage information about downloaded files to prevent downloads of duplicates (I mean already copied files from ftp), I can write my own filter which will filter out already downloaded files but I belive there should be built in feature which will give me controle of this (maybe idempotentRepository, actually I am not sure)...

What version of Camel are you using? And the 5 files are they in the same folder, or in different sub-folders? — Claus Ibsen, Apr 19 '11 at 07:42
Camel version is 2.7.0. There is 1 root directory with 3 subfolders, each subfolders may contain different number of files. To be more specific lets consider that one subfolder initially contains 5 files and then to this folder arrives 2 more files. And I want to prevent download of files which is already in destination folder. — endryha, Apr 19 '11 at 08:35
Hmm noop=true would imply idempotent as well. But try enabling it using idempotent=true in the uri — Claus Ibsen, Apr 19 '11 at 14:05
My app is run as standalone, and it seems that idempotent repository works but only in memory, so once all files were downloaded camel just stops and do periodic check of remote ftp for new files (correct behavior). But when I terminate my standalone app and run it again it starts to download all files again. It looks like I need some persistence idempotent repository, maybe file based... — endryha, Apr 20 '11 at 08:14

score 12 · Accepted Answer · answered Apr 20 '11 at 13:52

You need to use a persistent idempotent repository if you want Camel to be able to remember which files it previously have downloaded, between restarts.

You need to set this option on the ftp endpoint: idempotentRepository

See more details here: http://camel.apache.org/file2 (Note: The FTP component inherits the options from the file component.)

There are some examples on the wiki page how to use different stores. And you can also build you custom store.

Thank you, this is exactly what I want to achieve. – endryha Apr 20 '11 at 15:42 — endryha, Apr 20 '11 at 15:42

endryha · Answer 2 · 2014-07-11T11:37:01.753

4

Finally I end up with following solution:

public class SdiRouteBuilder extends RouteBuilder {
    @Override
    public void configure() throws Exception {
        from("ftp://login@url_to_ftp/RootFolder?" +
                "password=******&noop=true&stepwise=false&binary=true&consumer.delay=10s&recursive=true&filter=#fileFilter")
                .idempotentConsumer(header("CamelFileName"), FileIdempotentRepository.fileIdempotentRepository(new File("data", "repo.dat")))
                .process(new DownloadLogger())
                .to("file:data/outbox");
    }
}

edited Jul 11 '14 at 11:37

answered Apr 20 '11 at 15:41

endryha

7,166
8
40
64

Thanks for this hint. It seems not to work directly for ftp (setting properties like for file consumer). it seem only to work if using idempotentConsumer on route. – dermoritz Jul 10 '14 at 07:26

score 1 · Answer 3 · answered Jan 23 '18 at 08:59

Maybe @endryha answer work well in 2011, but not with Camel 2.20.1

In Camel 2.20.1, these code will create two idempotentRepository

ftp default memory idempotentRepository
idempotentConsumer custom idempotentRepository(file based in this case)

So the correct way to use idempotentRepository is (I remove most parameter for readability)

"ftp://login@url_to_ftp/RootFolder?&idempotent=true&idempotentRepository=#myIdempotentRepo"

and a Bean

@Bean
private IdempotentRepository<String> myIdempotentRepo() {
    return FileIdempotentRepository.fileIdempotentRepository(new File("data", "repo.dat");
}

Apache Camel ftp consumer loads the same files again and again

3 Answers3

Linked