0

I created a chunked array by:

import tables
FILTERS = tables.Filters(complib='lzo', complevel=1)
h5file = tables.openFile('file.h5', mode='w', filters=FILTERS)
x = h5file.createCArray(h5file.root,'chunk_array',tables.Float64Atom(),
                        shape=(256, 256, 256, 6, 6),
                        chunkshape = (256, 256, 256, 1, 1))

fill x by some value

h5file.close()

But when I read this file, pytables takes a huge time:

FILTERS = tables.Filters(complib='lzo', complevel=1)
E5F = tables.open_file('file.h5', mode='r', filters=FILTERS)
carray = E5F.root.chunk_array[0, 0, 0]

It's take... 22 seconds!

Did I do something wrong? How to speed up the reading performance in this case?

user2863620
  • 635
  • 3
  • 10
  • 17
  • I choice a chunkshape = (256, 256, 256, 1, 1) because of the way filling x. Something likes that: x[:, :, :, 0, 0] = numpy array... and it's faster than "None" hundred of times. – user2863620 Apr 17 '14 at 08:03

1 Answers1

0

Try changing the chunkshape to chunking the outermost axis rather than the innermost ones first. For example, chunkshape=(1, 256, 256, 6, 6). Doing this executes your second script in 0.474 sec on my machine.

Anthony Scopatz
  • 3,265
  • 2
  • 15
  • 14
  • 1
    I choice a chunkshape = (256, 256, 256, 1, 1) because of the way filling x. Something likes that: x[:, :, :, 0, 0] = numpy array... Now the file is already written, could I change the chunkshape of this file when reading? I don't see any arg* concerning chunkshape in the read method. – user2863620 Apr 17 '14 at 08:05