Is strided numpy array shared across multiple processes?

Question

Let`s assume that we created a numpy array with views on another array using stride tricks:

import numpy as np
from numpy.lib import stride_tricks
x = np.arange(20).reshape([4, 5])
arr = stride_tricks.as_strided(x, shape=(3, 2, 5),strides=(20, 20, 4))

We can confirm that this new array is indeed a view:

assert not arr.flags['OWNDATA']
# True

Question:

If I pass arr as an argument into multiprocessing.Process() will arr be copied into each process ? Will x be copied ? Please explain why.

I'd suggest experimenting with a small example. I haven't used multiprocessing much, so can't say what happens to views. While an `as_strided` view does not increase memory use, many operations using it do. I recently helped a poster who found that `std` caused a memory error because it subtracts the mean from the view. — hpaulj, Jan 30 '21 at 23:15
Yes, each argument is *pickled* then deserialized in the new process. Whether `x` is copied depends. — juanpa.arrivillaga, Jan 31 '21 at 00:06
@juanpa.arrivillaga how could you deserialize without making a brand new copy? — Mark Ransom, Jan 31 '21 at 01:34
@MarkRansom oh the arguments are always copied. `x` is a global variable. It depends on how/if `x` is used and how the process is created (e.g. fork vs spawn) — juanpa.arrivillaga, Jan 31 '21 at 01:46

hpaulj · Accepted Answer · 2021-01-31T01:55:15.940

If the sharing is via pickle serialization, then clearly the view (how ever generated) will produce a copy:

In [298]: x = np.arange(10)
In [299]: y = x.reshape(2,5)
In [300]: import pickle
In [301]: B = pickle.dumps(y)
In [302]: Y = pickle.loads(B)
In [303]: Y
Out[303]: 
array([[0, 1, 2, 3, 4],
       [5, 6, 7, 8, 9]])
In [304]: y.__array_interface__['data']
Out[304]: (43176224, False)
In [305]: x.__array_interface__['data']
Out[305]: (43176224, False)
In [306]: Y.__array_interface__['data']
Out[306]: (59035584, False)

For what it's worth the pickle of a numpy array is actually performed by np.save.

Passing x and making the view in each process might be better.

"Passing `x` and making the view in each process might be better." - that's not going to avoid a copy. — user2357112, Feb 02 '21 at 03:50

Is strided numpy array shared across multiple processes?

1 Answers1