Parallel processing of List in java

Question

I have a list of objects with list size of 500K. Each object in the list have a unique identifier. I want to validate the list for any duplicate objects(having the same identifier) and mark them with a flag. Since there are lot of elements in the list, the processing takes quite long time and i am not able to achieve within allotted time frame.

Is there any way to process the list in parallel(Using java Future/Callable)?

Thanks in advance. Mayuran

If *identifier* is the only criteria for 2 objects being equal, then why don't you override `equals()` and `hashCode()` and use a `Set` ? — TheLostMind, Oct 14 '14 at 05:58
List is already prepared and i need to get the duplicate objects as well. If two objects is having same identifier, the first one is accepted and the second one is the rejected one. — Mayuran, Oct 14 '14 at 06:01
@subhrajyoti Majumder, If i divide the list, how can i check for duplicate? The duplicate object may be in the another divided list — Mayuran, Oct 14 '14 at 06:03
@isnot2bad, Currently i iterate through the list and validate one by one — Mayuran, Oct 14 '14 at 06:04
like map-reduce in java 8 you will get this with parallel streaming — Subhrajyoti Majumder, Oct 14 '14 at 06:09
Searching for duplicates in a list is more than just iterating the list one by one. Again: Post your code! — isnot2bad, Oct 14 '14 at 12:24
put all objects in an array and sort them (using special Comparator or make objects to implement Comparable). Then scan the array and find duplicates. Only then decide if you need parallel processing. — Alexei Kaigorodov, Oct 14 '14 at 16:55

score 3 · Answer 1 · answered Oct 14 '14 at 06:00

3

I don't know what your definition of a long time is, but 500K object is not a lot. Most likely your algorithm is quite inefficient. Rather than relying on parallelisation (how would you parallelise this anyway?), consider using a better data structure/algorithm for this.

HashMap or HashSet should work quite well for this.

answered Oct 14 '14 at 06:00

NPE

486,780
108
951
1,012

Thanks, but each object has to go through lot of validation scenarios. So validation for each object takes some time. So that i want to process in parallel – Mayuran Oct 14 '14 at 06:06
1

@Mayuran: That's not what your questions says. You might want to work on improving the clarity of the question then. – NPE Oct 14 '14 at 06:16

Parallel processing of List in java

1 Answers1