0

I am new to threading and have very less knowledge in Java I am facing some serious performance issues with my data insertion in mysql. Data is huge chunk and takes huge amount of time, even when done with batch, prepared statement.

I want to read keys and values from hashtable and divide particular say-5 indexes that are to be inserted in mysql table using threads. Which I assume would boost up my performance. Suppose there are 100 index; so I want to create 5 threads, individual of which inserts 20 values in database. I can create a thread array and pass them executor pool and submit, but I am stuck how to proceed there after.

I have following doubts: how do I keep track of thread how much it has inserted or how do I limit the thread for 20 insertion? Can I further divide work of 20 insertion by each thread into batch of 4?

I am not looking for code, but right guidance which might help.

Death Metal
  • 830
  • 3
  • 9
  • 26
  • Use [`ExecutorService`](http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ExecutorService.html) to handle multiple threads at the same time. There's a great explanation of this in online tutorials, here's an example: vogella.com/articles/JavaConcurrency/article.html#threadpools – Luiggi Mendoza Sep 16 '13 at 03:39
  • "I am new to threading" - so don't. – Mitch Wheat Sep 16 '13 at 03:40
  • @MitchWheat don't what? – Luiggi Mendoza Sep 16 '13 at 03:46
  • Threading! The Data insertion speed is more likely to be I/O bound than CPU. Odds are that introducing threading will result in ZERO speedup. – Mitch Wheat Sep 16 '13 at 03:48
  • @MitchWheat have you tested this using a profiler to really say multi threading won't help, even using the advantage of a database connection pool? – Luiggi Mendoza Sep 16 '13 at 03:50
  • @ Mitch Wheat - Thank you for your reply, but that is what I am trying to learn. In this way, I would never be able to learn threading – Death Metal Sep 16 '13 at 03:52
  • @Luiggi Mendoza : No, I'm relying upon years of 25+ experience. – Mitch Wheat Sep 16 '13 at 03:54
  • @MitchWheat then you don't have anything to prove it... – Luiggi Mendoza Sep 16 '13 at 03:55
  • @ Luiggi Mendoza I have gone through vogella.com/articles/JavaConcurrency/article.html#threadpools; I am able to use Future, thread pool, executor. However, dividing index and inserting is a challenge for me now. Apologies if the comments are getting converted to discussion. – Death Metal Sep 16 '13 at 03:56
  • @MitchWheat I mean if you have an article, blog entry or somewhere when you can prove this to the world apart from personal experience. If you don't have anything that other people can really make sure this design won't apply, then you're not helping not OP nor any reader of this question. – Luiggi Mendoza Sep 16 '13 at 03:59

1 Answers1

0

I would, personally, use the Executor API.

Basically, I would create a fixed pooled executor and simply submit the tasks you want to execute to it and allow it to manage the pool of tasks as an existing thread becomes available.

Take a look at Executors tutorial for more details

MadProgrammer
  • 343,457
  • 22
  • 230
  • 366
  • @ MadProgrammer - I would try with this. Also, how do I limit the number of index from hashtable? – Death Metal Sep 16 '13 at 03:51
  • I probably wouldn't. I would simply create the number of "insert tasks" there are numbers and submit them to the executor and allow it to manage handing them out to the threads as they become available... – MadProgrammer Sep 16 '13 at 03:53
  • @ MadProgrammer - Probably, instead of hashmap, creation of linkedhashmap can help me to keep track of position and insert into mysql from that particular position. Please correct me if I am wrong. – Death Metal Sep 16 '13 at 04:03