I want to do the following with Dask:
- Load a matrix from a HDF5 file
- Parallelize the calculation of each entry
Here is my code:
def blocked_func(x):
return np.random.random()
with h5py.File(file_path) as f:
d = f['/data']
arr = da.from_array(d, chunks=(chunks_row, chunks_col))
arr2 = arr.map_blocks(blocked_func, dtype='float32').compute()
But the code throws the following error:
File ".../remote_fr_thinkpad/test_big_data.py", line 43, in <module>
arr2 = arr.map_blocks(blocked_func, dtype='float32').compute()
File ".../anaconda3/lib/python3.7/site-packages/dask/base.py", line 156, in compute
(result,) = compute(self, traverse=False, **kwargs)
File ".../anaconda3/lib/python3.7/site-packages/dask/base.py", line 399, in compute
return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
File ".../anaconda3/lib/python3.7/site-packages/dask/base.py", line 399, in <listcomp>
return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
File ".../anaconda3/lib/python3.7/site-packages/dask/array/core.py", line 779, in finalize
return concatenate3(results)
File ".../anaconda3/lib/python3.7/site-packages/dask/array/core.py", line 3497, in concatenate3
chunks = chunks_from_arrays(arrays)
File ".../anaconda3/lib/python3.7/site-packages/dask/array/core.py", line 3327, in chunks_from_arrays
result.append(tuple([shape(deepfirst(a))[dim] for a in arrays]))
File ".../anaconda3/lib/python3.7/site-packages/dask/array/core.py", line 3327, in <listcomp>
result.append(tuple([shape(deepfirst(a))[dim] for a in arrays]))
IndexError: tuple index out of range
I googled around and also tried dask's gu_func, but that threw the same error.
Thanks for your help.