2

I have got two dask arrays i.e., a and b. I get dot product of a and b as below

>>>z2 = da.from_array(a.dot(b),chunks=1)
>>> z2
dask.array<from-ar..., shape=(3, 3), dtype=int32, chunksize=(1, 1)>

But when i do 

sigmoid(z2)

Shell stops working. I can't even kill it. Sigmoid is given as below:

def sigmoid(z):
        return 1/(1+np.exp(-z))
Kavan
  • 331
  • 1
  • 4
  • 13

2 Answers2

2

Got it... I tried and it worked! 

ans = z2.map_blocks(sigmoid)
Kavan
  • 331
  • 1
  • 4
  • 13
2

When working with Dask Arrays, it is normally best to use functions provided in dask.array. The problem with using NumPy functions directly is they will pull of the data from the Dask Array into memory, which could be the cause of the shell freezing that you experienced. The functions provided in dask.array are designed to avoid this by lazily chaining computations until you wish to evaluate them. In this case, it would be better to use da.exp instead of np.exp. Provided an example of this below.

Have provided a modified version of your code to demonstrate how this would be done. In the example I have called .compute(), which also pulls the full result into memory. It is possible that this could also cause issues for you if your data is very large. Hence I have demonstrated taking a small slice of the data before calling compute to keep the result small and memory friendly. If your data is large and you wish to keep the full result, would recommend storing it to disk instead.

Hope this helps.

In [1]: import dask.array as da

In [2]: def sigmoid(z):
   ...:     return 1 / (1 + da.exp(-z))
   ...: 

In [3]: d = da.random.uniform(-6, 6, (100, 110), chunks=(10, 11))

In [4]: ds = sigmoid(d)

In [5]: ds[:5, :6].compute()
Out[5]: 
array([[ 0.0067856 ,  0.31701817,  0.43301395,  0.23188129,  0.01530903,
         0.34420555],
       [ 0.24473798,  0.99594466,  0.9942868 ,  0.9947099 ,  0.98266004,
         0.99717379],
       [ 0.92617922,  0.17548207,  0.98363658,  0.01764361,  0.74843615,
         0.04628735],
       [ 0.99155315,  0.99447542,  0.99483032,  0.00380505,  0.0435369 ,
         0.01208241],
       [ 0.99640952,  0.99703901,  0.69332886,  0.97541982,  0.05356214,
         0.1869447 ]])
jakirkham
  • 685
  • 5
  • 18