Multiple files or single files into HDFStore

Question

I am converting 100 csv files into dataframes and storing them in an HDFStore.

What are the pros and cons of

a - storing the csv file as 100 different HDFStore files?

b - storing all the csv files as separate items in a single HDFStore?

Other than performance issues, I am asking the question as I am having stability issues and my HDFStore files often get corrupted. So, for me, there is a risk associated with a single HDFStore. However, I am wondering if there are benefits to having a single store.

See also: http://stackoverflow.com/questions/25130511/how-do-i-read-write-to-a-subgroup-withing-a-hdf5store — Frames Catherine White, Sep 01 '14 at 09:50

score 1 · Answer 1 · answered Sep 01 '14 at 13:57

These are the differences:

multiple files

when using multiple files you can only corrupt a single file when writing (eg you have a power failure when writing)
you can parallelize writing with multiple files (note - never, ever try to parallelize with a single file a this will corrupt it!!!)

single file

grouping if logical sets

IMHO the advantages of multiple files outweigh using a single file as you can easily replicate the grouping properties by using sub directories

Multiple files or single files into HDFStore

1 Answers1