20

I would like to monitor all of the files in a given directory for changes, ie an updated timestamp. This use case seems natural for Camel using the file component, but I can't seem to find a way to configure this behavior.

A uri like:

file:/some/directory

will consume the files in the provided directory but will delete them.

A uri like:

file:/some/directory?noop=true

consumes each file once when it is added or when the route is started.

It's surprising that there isn't an option along the lines of

consumeOnChange=true

Is there a straightforward way to monitor file changes and not delete the file after consuming?

Janx
  • 3,285
  • 3
  • 19
  • 24

5 Answers5

26

You can do this by setting up the idempotentKey to tell Camel how a file is considered changed. For example if the file size changes, or its timestamp changes etc.

See more details at the Camel file documentation at: https://camel.apache.org/components/latest/file-component.html

See the section Avoiding reading the same file more than once (idempotent consumer). And read about idempotent and idempotentKey.

So something alike

from("file:/somedir?noop=true&idempotentKey=${file:name}-${file:size}")

Or

from("file:/somedir?noop=true&idempotentKey=${file:name}-${file:modified}")

You can read here about the various ${file:xxx} tokens you can use: http://camel.apache.org/file-language.html

Madhu Tomy
  • 662
  • 11
  • 25
Claus Ibsen
  • 56,060
  • 7
  • 50
  • 65
  • Thanks for taking the time to respond, Claus. I really appreciate your work on Camel. I just tried this for 20 minutes before my next meeting and I can't seem to get it working. I tried with size and modified to no avail. I now understand that noop turns idempotent true by default which makes sense. I'll keep looking at the file tokens when I have a bit more time, but I expected it to start picking up modified files when I added ${file:name}-${file:modified}. Is there more to it? Thanks again! – Janx Nov 20 '13 at 18:53
  • Ah just remembered I fixed a bug recently with idempotentKey, though reported against FTP. Its CAMEL-6936. So will be fixed in upcoming releases. Sorry for that bug. – Claus Ibsen Nov 20 '13 at 20:20
  • no worries - so a url like the second one you posted should work on a future release? Are SNAPSHOT artifacts published anywhere? – Janx Nov 20 '13 at 20:26
  • Hmm - I tried the latest 2.12.2-SNAPSHOT and 2.13-SNAPSHOT. No luck. To be clear, my url now looks like " ?noop=true&idempotentKey=${file:name}-${file:modified} Maybe it's not in the builds yet? – Janx Nov 20 '13 at 22:28
  • Ah thanks the first fix from CAMEL-6936 didn't cover it all. I have fixed this again. So any future SNAPSHOT updates should include the fix. – Claus Ibsen Nov 21 '13 at 08:11
  • Still no luck on the current SNAPSHOTs - should this feature be working now? – Janx Nov 29 '13 at 12:47
  • Thanks for this post, I also need it. On the size vs modified, I will use modified since size might the same but different content like "message a" to "message b" would be missed in the process. – Ricky Vene Aug 12 '22 at 15:29
2

Setting noop to true will result in Camel setting idempotent=true as well, despite the fact that idempotent is false by default.

Simplest solution to monitor files would be:

.from("file:path?noop=true&idempotent=false&delay=60s")

This will monitor changes to all files in the given directory every one minute.

This can be found in the Camel documentation at: http://camel.apache.org/file2.html.

Piotr Niewinski
  • 1,298
  • 2
  • 15
  • 27
1

I don't think Camel supports that specific feature but with the existent options you can come up with a similar solution of monitoring a directory.

What you need to do is set a small delay value to check the directory and maintain a repository of the already read files. Depending on how you configure the repository (by size, by filename, by a mix of them...) this solution would be able to provide you information about news files and modified files. As a caveat it would be consuming the files in the directory very often.

Maybe you could use other solutions different from Camel like Apache Commons VFS2 (I wrote a explanation about how to use it for this scenario: WatchService locks some files?

Community
  • 1
  • 1
hveiga
  • 6,725
  • 7
  • 54
  • 78
  • Well said. This is the pragmatic answer :-) In my case (a Camel FTP client shipping files as soon as they're made available) I added a JMS queue where the file producer posts a message that wakes up the FTP client. – Marco Faustinelli Jan 17 '17 at 15:39
1

I faced the same problem i.e. wanted to copy updated files also (along with new files). Below is my configuration,

public static void main(String[] a) throws Exception {

    CamelContext cc = new DefaultCamelContext();

    cc.addRoutes(createRouteBuilder());

    cc.start();

    Thread.sleep(10 * 60 * 1000);

    cc.stop();
}


protected static RouteBuilder createRouteBuilder() {
    return new RouteBuilder() {
        public void configure() {
            from("file://D:/Production"
                    + "?idempotent=true"
                    + "&idempotentKey=${file:name}-${file:size}"
                    + "&include=.*.log"
                    + "&noop=true"
                    + "&readLock=changed")

            .to("file://D:/LogRepository");
        }
    };
}

My testing steps:

  1. Run the program and it copies few .log files from D:/Production to D:/LogRepository and then continues to poll D:/Production directory
  2. I opened a already copied log say A.log from D:/Production (since noop=true nothing is moved) and edited it with some editor tool. This doubled the file size and save it.

At this point I think Camel is supposed to copy that particular file again since its size is modified and in my route definition I used "idempotent=true&idempotentKey=${file:name}-${file:size}&readLock=changed". But camel ignores the file. When I use TRACE for logging it says "Skipping as file is already in progress...", but I did not find any lock file in D:/Production directory when I editted and saved the file.

I also checked that camel still ignores the file if I replace A.log (with same name but bigger size) in D:/Production directory from outside.

But I found, everything is working as expected if I remove noop=true option.

Am I missing something?

Abhishek Chatterjee
  • 1,962
  • 2
  • 23
  • 31
1

If you want monitor file changes in camel, use file-watch component.

Example -> RECURSIVE WATCH ALL EVENTS (FILE CREATION, FILE DELETION, FILE MODIFICATION):

from("file-watch://some-directory")
.log("File event: ${header.CamelFileEventType} occurred on file ${header.CamelFileName} at ${header.CamelFileLastModified}");

You can see the complete documentation here: Camel file-watch component

Amin
  • 138
  • 7