0

For a deep learning project, I reorganize data from hdf5 files into a big training_data.hfd5 containing batch0...batch_max When running it in local on my laptop it works, but the process is a bit long (much data), so I tried to run it on big-CPU servers I have access to. Now I have this error :

Traceback (most recent call last):
  File "prepare_data.py", line 88, in <module>
    main(readargs())
  File "prepare_data.py", line 82, in main
    prepare_data('', args)
  File "prepare_data.py", line 72, in prepare_data
    args.preprocessdatadir, args.datadir, stdout=None)
  File "/home/bizeul/Stage/GCNN-1/GCNN_Icube/data_prepare_cube/group_batch.py", line 113, in group_batchs
    curr_batch_idx, count_events = organizer.iter_data()
  File "/home/bizeul/Stage/GCNN-1/GCNN_Icube/data_prepare_cube/group_batch.py", line 93, in iter_data
    self.next_batch(batchsize)
  File "/home/bizeul/Stage/GCNN-1/GCNN_Icube/data_prepare_cube/group_batch.py", line 65, in next_batch
    self.next_event(fileout)
  File "/home/bizeul/Stage/GCNN-1/GCNN_Icube/data_prepare_cube/group_batch.py", line 82, in next_event
    fileout[self.group_name + '/label'][self.event_idx] = label
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/bizeul/virtual_cpu/build/h5py/h5py/_objects.c:2840)
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/bizeul/virtual_cpu/build/h5py/h5py/_objects.c:2798)
  File "/home/bizeul/virtual_cpu/lib/python2.7/site-packages/h5py/_hl/dataset.py", line 630, in __setitem__
    self.id.write(mspace, fspace, val, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/bizeul/virtual_cpu/build/h5py/h5py/_objects.c:2840)
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/bizeul/virtual_cpu/build/h5py/h5py/_objects.c:2798)
  File "h5py/h5d.pyx", line 221, in h5py.h5d.DatasetID.write (/home/bizeul/virtual_cpu/build/h5py/h5py/h5d.c:3680)
  File "h5py/_proxy.pyx", line 132, in h5py._proxy.dset_rw (/home/bizeul/virtual_cpu/build/h5py/h5py/_proxy.c:2022)
  File "h5py/_proxy.pyx", line 93, in h5py._proxy.H5PY_H5Dwrite (/home/bizeul/virtual_cpu/build/h5py/h5py/_proxy.c:1732)
IOError: Can't write data (No appropriate function for conversion path)

What do you think is the problem ?

Gericault
  • 219
  • 1
  • 3
  • 8
  • It may help to know what kind of data you are writing. Also check the `h5py` and `hdf5` versions. For example you might be using a feature/type that's present in a new version on your machine, but absent on an older version. – hpaulj Jun 13 '17 at 23:13
  • checked it ! (same) type of data is pretty simple, it's oragnized in various groups and datasets but in the end it's only float/int/string, sometimes organized in numpy array – Gericault Jun 14 '17 at 00:25

1 Answers1

0

Had the same issue. I was trying to create a dataset with set of images and labels. But didn't resize it. After resizing all the images to same size, i was able to run the code and create dataset.

img = cv2.resize(img,(500,500))

Resize after you read the image

Shreehari
  • 334
  • 4
  • 10