0

I have a bunch of binary data in N-byte chunks, where each chunk corresponds exactly to one row of a PyTables table.

Right now I am parsing each chunk into fields, writing them to the various fields in the table row, and appending them to the table.

But this seems a little silly since PyTables is going to convert my structured data back into a flat binary form for inclusion in an HDF5 file.

If I need to optimize the CPU time necessary to do this (my data comes in large bursts), is there a more efficient way to load the data into PyTables directly?

Jason S
  • 184,598
  • 164
  • 608
  • 970

1 Answers1

0

PyTables does not currently expose a 'raw' dump mechanism like you describe. However, you can fake it by using UInt8Atom and UInt8Col. You would do something like:

import tables as tb
f = tb.open_file('my_file.h5', 'w')
mytable = f.create_table('/', 'mytable', {'mycol': tb.UInt8Col(shape=(N,))})
mytable.append(myrow)
f.close()

This would likely get you the fastest I/O performance. However, you will miss out on the meaning of the various fields that are part of this binary chunk.

Arguably, raw dumping of the chunks/rows is not what you want to do anyway, which is why it is not explicitly supported. Internally HDF5 and PyTables handle many kinds of conversion for you. This includes but is not limited to things like endianness and thet platform specific feature. By managing the data types for you the resultant HDF5 file and data set cross platform. When you dump raw bytes in the manner you describe you short-circuit one of the main advantages of using HDF5/PyTables. If you do short-circuit, you have a high probability that the resulting file will look like garbage on anything but the original system that produced it.

So in summary, you should be converting the chunks to the appropriate data types in memory and then writing out. Yes this takes more processing power, time, etc. So in addition to being the right thing to do it will ultimately save you huge headaches down the road.

Anthony Scopatz
  • 3,265
  • 2
  • 15
  • 14
  • I don't want the table *declared* as raw data (I like HDF5's structure), I just want to submit it as raw data. – Jason S Oct 12 '13 at 18:36
  • As I mentioned in my answer, you can't currently do this in PyTables and have the HDF5 structure. You might be able to use numpy's ndarray buffer interface, but then numpy will be type casting so it isn't clear that there is a performance advantage here. Pull requests are welcome for this feature. – Anthony Scopatz Oct 12 '13 at 19:45