0

I have a EIP design related query.I have a requirement to process csv file by chunks and call a Rest API.After completion of processing of whole file i need to call another Rest API telling processing is complete.I wanted the route to be transacted so i have queue in between in case of end system not available the retry will happen at broker level.

My flow is as below. First flow: csv File->Split by chunk of 100 records->Place message in queue

the second flow(Transacted route): Picks message from queue ->call the rest API

the second flow is transacted.Since iam breaking the flow and it is asynchronous iam not sure how to call to the completion call.I do not have a persistent store to status of each chunk processing.

is there anyway i can achive it using JMS functionality or Camel?

Ravi
  • 1,247
  • 4
  • 15
  • 35
  • Yes you can - You learn better by trying out yourself a bit first. This questions is a bit too broad for a SO question. – Claus Ibsen Nov 14 '17 at 08:14
  • Sure claus i was not sure where to post EIP design related queries,so i posted here. – Ravi Nov 15 '17 at 09:33

1 Answers1

0

What you can use for your first flow is the Camel Splitter EIP:
http://camel.apache.org/splitter.html

And closely looking at the doc, you will find that there are three exchange properties available for each split exchange:

  • CamelSplitIndex: A split counter that increases for each Exchange being split. The counter starts from 0.
  • CamelSplitSize: The total number of Exchanges that was splitted. This header is not applied for stream based splitting. From Camel 2.9 onwards this header is also set in stream based splitting, but only on the completed Exchange.
  • CamelSplitComplete: Whether or not this Exchange is the last.

As they are exchange properties, you should put them to JMS headers before sending the messages to a queue. But then you should be able to make use of the information at the second flow, so you can know which is the last message.

Keep in mind, though, that it's all asynchronous so the CamelSplitComplete flag doesn't necessarily mean the last message at the second flow. You may create a stateful counter or utilise the Resequencer EIP http://camel.apache.org/resequencer.html to deal with the asynchronicity.

Tadayoshi Sato
  • 1,401
  • 11
  • 18
  • Hi Tadayoshi,Thanks for the hints will do some research and get back to you shortly. – Ravi Nov 14 '17 at 05:09
  • Hi I understand from the below post resequencer is memory based and it cannot be part of jms trasaction. http://camel.465427.n5.nabble.com/Camel-Resequencer-and-JMs-transactions-td5766843.html – Ravi Nov 15 '17 at 09:26
  • @Ravi Ah, you are right. So you might need to go with a custom counter impl with Camel's UnitOfWork http://camel.apache.org/oncompletion.html that allows you to write a custom compensation upon transaction failures. – Tadayoshi Sato Nov 15 '17 at 11:08