0

I have a spring batch app that's runs on tomcat 8.5. This batch works with lots of data such as ten Million records and it is too slow. I want to find most time consuming parts such as database queries E.G, socket IO, thread blocking or waiting, CPU consuming, or garbage collection that maybe slows down the app. I 'm mostly suspicious to jdbc queries E.G, socket IO. I tried to use local partitioning to scale it up and give more memory to tomcat and increase commit interval in spring batch settings. I had a look at socketIO tab in Jmc and logged execution time of one of the methods it shows,but it only takes 15 up to 30 milliseconds. Another problem is that Jmc only shows percentages not exact time. So, I could not figure out how long it takes. I'm a little confused.
Thanks too much in advance.

hamidreza75
  • 567
  • 6
  • 15
  • 1
    "ten Million records" with a run time of "15 up to 30 milliseconds", is that per record? If so, then that is still a significant amount of time. Let's assume an average runtime of 23ms per record, so over 10mil records it would result in a total processing time of 63-64h in that one method alone. If you could optimize that method to take half the time then you would be saving 32h of processing time. – sorifiend Sep 27 '21 at 23:52
  • 1
    Yes @sorifiend you're right. it called for every record.I want to find most time consuming methods sorted by time that they take. – hamidreza75 Sep 28 '21 at 00:03
  • 1
    Is this synchronous I/O? So you only start on record #2 after record #1 is finished with? Consider an asynch model. – user16632363 Sep 28 '21 at 01:30
  • I used local partitioning. every partition processed in separate thread. can I use asynchronous processing with local partitioning together? – hamidreza75 Sep 28 '21 at 05:42

0 Answers0