Clojure structure multiple calculation/writes to work in parallel

Question

Let's say I have the following code :

(defn multiple-writes []
  (doseq [[x y] (map list [1 2] [3 4])] ;; let's imagine those are paths to files
     (when-not (exists? x y) ;; could be left off, I feel it is faster to check before overwriting
       (write-to-disk! (do-something x y)))))

That I call like this (parameters omitted) :

   (go (multiple-writes))

I use go to execute some code "in the background", but I do not know if I am using the right tool here. Some more information about those functions :

• this is not high-priority code at all. It could even fail - multiple-writes could be seen as a cache-filling function.
• I consequently do not care about the return value.
• do-something takes a between 100 and 500 milliseconds depending of the input
• do-something consumes some memory (uses image buffers, some images can be 2000px * 2000px)
• there are 10 to 40 elements/images to be processed every time multiple-writes is called.
• every call to write-to-disk will create a new file (or overwrite it if any, though that should not happen)
• write-to-disk writes always in the same directory

So I would like to speed up things by executing (write-to-disk! (do-something x y)) in parallel to go as fast as possible. But I don't want to overload the system at all, since this is not a high-priority task.

How should I go about this ?

Note : despite the title, this is not a duplicate of this question since I don't want to restrict to 3 threads (not saying that the answer can't be the same, but I feel this question differs).

Answer 1

Consider basing your design on streams or fork/join.

I would a single component that does IO. Every processing node can then send their results to be saved there. This is easy to model with streams. With fork/join, it can be achieved by not returning the result up in the hierarchy but sending it to eg. an agent.

If memory consumption is an issue, perhaps you can divide work even more. Like 100x100 patches.

Answer 2

Take a look at the claypoole library, which gives some good and simple abstractions filling the void between pmap and fork/join reducers, which otherwise would need to be coded by hand with futures and promises.

With pmap all results of a parallel batch need to have returned before the next batch is executed, because return order is preserved. This can be a problem with widely varying processing times (be they calculation, http requests, or work items of different "size"). This is what usually slows down pmap to single threaded map + unneeded overhead performance.

With claypoole's unordered pmap and unordered for (upmap and upfor), slower function calls in one thread (core) can be overtaken by faster ones on another thread because ordering doesn't need to be preserved, as long as not all cores are clogged by slow calls.

This might not help much in case of IO to one disk being the only bottleneck, but since claypoole has configurable thread pool sizes and functions to detect the number of available cores, it will help with restricting the amount of cores.

And where fork/join reducers would optimize CPU usage by work stealing, it might greatly increase memory use, since there is no option to restrict the amount of parallel processes without altering the reducer library.