julia: pmap passing in vectors with indexes

Question

I would like to use julia's pmap to do something like this:

pmap( f, v, inits[i])

where f is the function I want to call in parallel on v which is some array. I also want to pass some parameters to f (inits), but inits itself is an array, and I want ith parameter to be passed along to the correct process. Does this make sense?

I could do this by sorting of 'rolling my own' version of pmap, since it can be easily done with remotecall_fetch. Here is that implementation if the above was confusing:

    i=1
    nextidx() = (idx=i; i+=1; idx)
    @sync begin
      for k in 1:np
        if k != myid() || np == 1
          @async begin
            while true
              idx = nextidx()
              if idx > chains
                break
              end
              result[idx] = remotecall_fetch(k, mcmc_sub, m, iters, burnin, 
                                             thin, idx, ps[idx], p)
            end
          end
        end
      end
    end

How about a new 'v' whose elements are a composition of the old 'v' and its 'inits'? Then use the standard 'pmap' with a modified 'f' that first extracts the elements (old 'v' element & its specific 'inits') within the new 'v'. — rickhg12hs, Jul 03 '14 at 10:44
so like an [v, inits] kind of an array (if properly done)? That could work — bdeonovic, Jul 03 '14 at 11:04
@bdeonovic could you please post exactly what you did as an answer so this could be marked as answered? Thanks! — Chris Rackauckas, Aug 10 '16 at 06:57

score 1 · Accepted Answer · answered Aug 10 '16 at 15:35

Suppose you have some input v that you want to run pmap on. Each element of v also has an associated vector of inits which are stored in the columns of the matrix inits.

inits = reshape(randn(20), 5,4)
v = collect(1:4)

The key is to write your function to take in a tuple of your input and your inits

f = function(args::Tuple{Int64, Vector{Float64}})
  x, inits = args
  return sum(x * inits)
end

and then zip together your input and inits

args = collect(zip(v, [inits[:,j] for j in 1:4]))
pmap(f, args)

julia: pmap passing in vectors with indexes

1 Answers1