how to parallelize M sequential operations over N objects (with sync point)

Question

Suppose I have N objects and M operations (some of which are doing network I/O). I want to call the sequence of operations in order for each of the N objects but allowing parallelism (across the objects) where possible. There is one synch (fan-in) point in the pipeline let's say at operation M-1. What's the best/easiest way to do this in core.async? [Also, this is in ClojureScript so thread is not an option).

Could the operations be functions that receive and return a number? Could the objects be numbers? Is it like a matrix of results you are after? Do you want the operations/functions to be done sequentially, so that for instance the first function is being done in parallel (many working at once) on each of the objects/numbers? Not sure about that 'fan in' point - I would expect there to be a 'handler' of some sort that just gets all the objects and operations. — Chris Murphy, Feb 27 '16 at 06:45
Possible duplicate of [Parallel doseq for Clojure](http://stackoverflow.com/questions/10969708/parallel-doseq-for-clojure) — muhuk, Feb 27 '16 at 06:59
@ChrisMurphyThe operations are async web requests (e.g.). I specified the type of sequential and parallel operation I need in the original request. Is there something not clear about that? 'Fan in' == sync point; i.e., all operations across the N objects at that point must complete before progress is made on the remaining operations after that point (also known as 'join' in UNIX parlance). — Jonathan Leonard, Feb 27 '16 at 19:35

Jonathan Leonard · Accepted Answer · 2016-02-29T03:28:42.143

The combination of cats' alet, promesa/promise & promesa/all will do it.

Here's the code (with elisions):

(ns example
  (:require [[cats.context :as ctx :include-macros true]
            [cats.core :as m :include-macros true]
            [promesa.monad :as pm]
            [promesa.core :as p :include-macros true]]))

(defn foo []
  (ctx/with-context pm/promise-context
    (let [last-stage (fn [obj] (final-operation obj)) ;; could be chain a la first-stage
          first-stage (fn [obj]
                        (p/chain
                         (operation-m0 args)
                         (partial operation-m1 some-args)
                         (partial operation-m2 some-more-args)))
          first-stage-results (p/all (mapv first-stage objects))]
      (p/then (m/alet [_ (p/then first-stage-results global-operation-requiring-fan-in)
                       z (p/all (mapv last-stage objects))] z)
              (fn [_] (.info logger "Finished all operations."))))))

operation-m0, operation-m1, operation-m2, & final-operation should be functions that return promises immediately and resolve the promises when results are available.

this link won't work, I suggest you post the code and mark the question solved. — Valentin Waeselynck, Feb 28 '16 at 10:22

score 0 · Answer 2 · answered Mar 01 '16 at 22:35

0

Perhaps you can use core.async/pipeline-async which applies a function asynchronously to the elements from a channel, placing the call results into a channel in the same order as the input. Here is an example.

answered Mar 01 '16 at 22:35

Terje Norderhaug

3,649
22
25

how to parallelize M sequential operations over N objects (with sync point)

2 Answers2