Julia pmap- writing to array

Question

Can you please point me to a code using pmap that iterates over columns of an array writing to it complex structures that are not bit type objects? I understand that SharedArray has the limitation of requiring bit-type elements.

Thanks,

Answer 1

A SharedArray is just a block of memory shared by all processes running on the same local node within a Julia cluster. Since you share memory you need to know the size of its block and hence the requirement for the bit type.

I can think of two options:

• use a library designed for sending objects between workers in a Julia cluster - ParallelDataTransfer.jl does this job quite well
• try to serialize the shared object into memory using the SharedArray based on bytes. In this case you will need unfortunately to manage yourself the memory layout of the object.

For the second example the code could look something like this:

using Distributed
addprocs(2)
@everywhere using Serialization, SharedArrays
s = SharedVector{UInt8}(1_000_0000);
@everywhere struct Dat           
       a::Int                           
       b::String                        
end                                      
d = Dat(3,"hello")               

Now this object can be written to the shared memory:

io = IOBuffer(s,write=true)
serialize(io, d)

And it can be read by remote workers:

julia> @sync @distributed for i in 1:2
           io = IOBuffer(s)
           dat2 = deserialize(io)
           println(myid, dat2)
       end
      From worker 2:    myidDat(3, "hello")
      From worker 3:    myidDat(3, "hello")
Task (done) @0x0000000078079140