0

It seems that I get an error when format=table but no error with format=fixed. Here is the command. What's weird is that it still seems to load the data. I just have to figure out a way to move past this. And it would give me peace of mind to not have any error. The dataframe is preprocessed, types set within the columns.

The command I run is:

hdf = pd.HDFStore('path-to-file')

hdf.put('df',df,format='table')

The error I get is:

HDF5ExtError: HDF5 error back trace

  File "../../../src/H5Dio.c", line 182, in H5Dread
    can't read data
  File "../../../src/H5Dio.c", line 550, in H5D__read
    can't read data
  File "../../../src/H5Dchunk.c", line 1837, in H5D__chunk_read
    unable to read raw data chunk
  File "../../../src/H5Dchunk.c", line 2863, in H5D__chunk_lock
    unable to read raw data chunk
  File "../../../src/H5Fio.c", line 113, in H5F_block_read
    read through metadata accumulator failed
  File "../../../src/H5Faccum.c", line 258, in H5F_accum_read
    driver read request failed
  File "../../../src/H5FDint.c", line 142, in H5FD_read
    driver read request failed
  File "../../../src/H5FDsec2.c", line 725, in H5FD_sec2_read
    file read failed: time = Mon Nov 17 01:25:35 2014
user3659451
  • 1,913
  • 9
  • 30
  • 43
  • http://stackoverflow.com/questions/26963654/hdfstore-start-stop-not-working?noredirect=1#comment42467626_26963654 – user3659451 Nov 17 '14 at 03:44
  • pls read the extensive docs: http://pandas.pydata.org/pandas-docs/stable/io.html#hdf5-pytables; sounds like you want append which is not possible with a fixed store - not enough code to tell what you are actually doing – Jeff Nov 17 '14 at 03:49
  • fascinating... I defined a method I pulled from another SO page: def chunker(seq, size): """given an array with a __getitem__ attribute and size of chunks return an iterator of array""" return (seq[pos:pos + size] for pos in range(0, len(seq), size)) – user3659451 Nov 17 '14 at 04:05
  • Then when I run this with `hdf.append` rather than `hdf.put` it works. Not only that, when I do the whole thing with hdf.append it seems to work, but only sometimes.... I think I'd rather just treat this in chunks. I will move on, and hope to optimize this later. – user3659451 Nov 17 '14 at 04:07

0 Answers0