14

Does any one have an idea for updating hdf5 datasets from h5py? Assuming we create a dataset like:

import h5py
import numpy
f = h5py.File('myfile.hdf5')
dset = f.create_dataset('mydataset', data=numpy.ones((2,2),"=i4"))
new_dset_value=numpy.zeros((3,3),"=i4")

Is it possible to extend the dset to a 3x3 numpy array?

George Monet
  • 317
  • 2
  • 3
  • 7

1 Answers1

16

You need to create the dataset with the "extendable" property. It's not possible to change this after the initial creation of the dataset. To do this, you need to use the "maxshape" keyword. A value of None in the maxshape tuple means that that dimension can be of unlimited size. So, if f is an HDF5 file:

dset = f.create_dataset('mydataset', (2,2), maxshape=(None,3))

creates a dataset of size (2,2), which may be extended indefinitely along the first dimension and to 3 along the second. Now, you can extend the dataset with resize:

dset.resize((3,3))
dset[:,:] = np.zeros((3,3),"=i4")

The first dimension can be increased as much as you like:

dset.resize((10,3))
Yossarian
  • 5,226
  • 1
  • 37
  • 59
  • 6
    resize also accepts an axis argument so you only have to specify the new size for the axis you're extending, rather than all of them: dset.resize(10, axis=0) – Joseph Sheedy Nov 11 '15 at 20:50
  • That's very useful @velotron, it could be added to the accepted answer because for multidimensional arrays it is easier to do so. – Guillem Cucurull May 12 '16 at 07:52
  • does this example overwrite what was previously in the file? or does it just append the new contents? – user798719 May 14 '17 at 09:51
  • You 'append' to the current data while you increase or decrease the size. The indexes of the data do not change when you resize a dataset. http://docs.h5py.org/en/latest/faq.html – Aquiles Carattino Mar 12 '18 at 08:14
  • As I was searching for it, If you want to resize a 1-dimentional array, you have to pass (None, ) to maxshape, don't forget to write , to make it a tuple. Also do it in resize method or use axie=0 as Joseph mentioned. – Arsham Arya Dec 14 '21 at 11:56