I have requirement to iterate over close to million records. The current code was written in Dataweave with filter and ordering logic. However, I'm seeing a performance issue. I was thinking of converting this DataWeave logic to Java using the Java Component and seeing if that improves performance.
How can I improve the performance of my code?
Data weave has some performance issues if you are using Global functions or using 'p()' functions.
If you are having any such functions in your dwl, avoid it.
Since you are processing high volume of records, if the records are identical, you can use scatter-gather pattern and leverage the asynchronous processing of records. You can further tune the performance by configuring the thread pool in which you are doing the transformation/filtering logic.
For implementation of scatter gather pattern you can refer this link. The order logic that you have implemented in data weave, can be moved to a custom aggregator and you can reorder records as per your custom logic
If nothing helps, consider using Java8 Streams API in your custom Java component to both filter and order the records.
Dataweave is best at what it does. It Doesn't matter how many records you are willing to process through your app. Moreover, the primary constraints are not with dataweave, but with the App memory and Vcores assigned. You must consider to recess the processing of records if it's over a million. Also, you must do your processing action in chunks/batches at regular intervals with a reasonable time delay.
From what I have tested, any App which is running on 0.1Vvores and 1 worker, usually will hit the Mule Health Monitor, eventually leading to a crash, if ran straight for 15 hours or more. A good thumbrule is to never cross the system resource usage or the CPU beyond 70 %.
Note : It is strongly recommended to not use Mule Java Components for complex, repetitive, higher load executions.
