3

I want to do the following with Dask:

  1. Load a matrix from a HDF5 file
  2. Parallelize the calculation of each entry

Here is my code:

def blocked_func(x):
    return np.random.random()

with h5py.File(file_path) as f:
    d = f['/data']
    arr = da.from_array(d, chunks=(chunks_row, chunks_col))

    arr2 = arr.map_blocks(blocked_func, dtype='float32').compute()

But the code throws the following error:

File ".../remote_fr_thinkpad/test_big_data.py", line 43, in <module>
    arr2 = arr.map_blocks(blocked_func, dtype='float32').compute()
  File ".../anaconda3/lib/python3.7/site-packages/dask/base.py", line 156, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File ".../anaconda3/lib/python3.7/site-packages/dask/base.py", line 399, in compute
    return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
  File ".../anaconda3/lib/python3.7/site-packages/dask/base.py", line 399, in <listcomp>
    return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
  File ".../anaconda3/lib/python3.7/site-packages/dask/array/core.py", line 779, in finalize
    return concatenate3(results)
  File ".../anaconda3/lib/python3.7/site-packages/dask/array/core.py", line 3497, in concatenate3
    chunks = chunks_from_arrays(arrays)
  File ".../anaconda3/lib/python3.7/site-packages/dask/array/core.py", line 3327, in chunks_from_arrays
    result.append(tuple([shape(deepfirst(a))[dim] for a in arrays]))
  File ".../anaconda3/lib/python3.7/site-packages/dask/array/core.py", line 3327, in <listcomp>
    result.append(tuple([shape(deepfirst(a))[dim] for a in arrays]))
IndexError: tuple index out of range

I googled around and also tried dask's gu_func, but that threw the same error.

Thanks for your help.

Andy R
  • 1,339
  • 10
  • 20

1 Answers1

1

map_block expects blocked_func to return an array of the same shape of its input (chunks_row, chunks_col), while it actually just returns a float.

Try either with

1) a function which preserves shape, e.g:

def blocked_func(x):
    return x*2

or

2) tell map_blocks that the shape of the output will be different:

arr2 = arr.map_blocks(blocked_func, chunks=(1,1), dtype='float32').compute()

but keep the dimensionality of the input array in blocked_func, e.g.:

def blocked_func(x):
    return np.random.random()[None,None]
    # or like this
    # return np.array([1,1])
malbert
  • 308
  • 1
  • 7