0

I have this camel route that consumes from Kafka in a constant flow. Is there some way to define that in each 10 min, for example, a new file be created? I need that each file has different names since when a file cannot have the same name as other. I've tried with timer and quartz camel component without success.

from("kafka:testTopic?brokers=localhost:9092&groupId=01")
.to("file:C://DESTINATION?fileExist=Append&fileName=prefixName_${variable}.json");
Erick Roseira
  • 13
  • 1
  • 5
  • 1
    This is not a "please do this for me" page, but a "I've done this, it does not work, can you help me?" page. Show the people what you did and they can help you! – Marvin Fischer Feb 20 '18 at 16:25
  • The file you append to is using year, month, day only in the file name, so even after 10 min then, you cannot have 2 files with the same name. Can you explain more what your use-case is. For example do you want something to happen at 10 min interval, or 10 min inactivity or something? – Claus Ibsen Feb 20 '18 at 16:34
  • Yeah, @ClausIbsen. Is needed that each file has different names. I have tried with timer and quartz component, but they just trigger the route at a specific moment. This doesn't fulfil my requirements. – Erick Roseira Feb 20 '18 at 16:41
  • @MarvinFischer I know, I am trying to specify at most what I am trying to do. And unfortunately, this is what I have achieved until now, I am working hard to try to resolve this as well. – Erick Roseira Feb 20 '18 at 16:43
  • A comment, if you just want unique filenames why don't you just add a timestamp to your filenames? – Souciance Eqdam Rashti Feb 20 '18 at 23:28
  • 1
    The "constant flow" from Kafka does not match with "one file in 10 minutes". Can you further explain what should happen all 10 minutes? Do you want to write all 10 minutes one file that contains everything that was received from Kafka in this time slot? Then you should have a look at Camel Aggregator: http://camel.apache.org/aggregator.html – burki Feb 21 '18 at 06:46
  • @burki. Yeah, For example, Let's suppose that my route started at 09:00 A.M, at 09:10 A.M I'll have FILE1.json (that contains the data consumed in this interval). After that, at 09:20 A.M, another File will be created for example FILE2.json (that contains the data consumed in this new 10min interval) and so on. – Erick Roseira Feb 21 '18 at 11:31
  • Yeah, for this case Camel Aggregator is your friend. You can configure it to "close" an open aggregation after 10 mins by setting `completionTimeout`. You can implement in an `AggregationStrategy` how exactly the messages should be "collected". And for the filename you could just add the time to it. – burki Feb 21 '18 at 12:19
  • @burki, but I don't want aggregate data, I just want to split up the data into different files in every 10 minutes. Did you get it? – Erick Roseira Feb 21 '18 at 13:23
  • Yeah, split and aggregate is just a matter of your point of view. you say you want to split the data stream in 10-minute-chunks. I say you aggregate data in 10-minute-intervals. It is the same. You use Camel aggregator to collect data for 10 minutes. Otherwise every piece of data would result in an individual file. – burki Feb 21 '18 at 13:35

0 Answers0