0

Now I have plenty of JSON files on local, I need read these files and transfer them to JSONObject then do some business logical, such as find out the child-node then return, count some special value then return.

As the single file, I have finished the code of read the JSON file-->transfer it to JSONObject-->do the search of child-node/ count some special value.

But if when there was plenty of JSON files, how can I do this job? I believe multi-thread could resolve my problem, but how can I collect every thread's result and finally get one result?

By the way, if I need create a thread pool and how ?

leo
  • 35
  • 1
  • 1
  • 7
  • 2
    What is the problem when doing it sequentially, in a single thread? Start doing that first. There is a big chance it will be fast enough, or even faster than a multi-thread solution, given that you main bottleneck is disk IO, and multithreading won't improve that. – JB Nizet Aug 23 '15 at 07:12

1 Answers1

1

You can try multi-threading it using Streams. There is no guarantee this will be faster, or if your tasks use lots of memory, you could run out of memory but this is a simple way to try this.

List<Result> results= Files.list(path).parallel()
                           .map(p -> turnFileIntoJSONObject(p))
                           .map(json -> process(json))
                           .collect(Collectors.toList());

This will distribute the files to be processed across your available CPUs and collect the results at the end.

You can also drop the .parallel() if it is not faster for you.

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130