In pandas, one can keep data in a Panel of multiple dataframes. That Panel can even be 4D or of higher dimensions. When saving data, one can save to HDF5 format, which allows an internal directory structure, with nested directories. Thus, it is completely possible to store multiple dataframes under a single directory, similar to how one could keep multiple dataframes in a single Panel.
When should one use a Panel vs. nested directories in an HDF5?
I'm writing and reading many small tables/dataframes of the same size, generated with different set of parameters. I have been organizing them with Panels and Panel4Ds, then saving the whole Panel to a table in an HDF5 file. When reading back the data, I generally want to eventually use all of the dataframes, but at any one time I'm actually only using one of the dataframes. For my current purposes it seems none of this matters; the dataframes are small, so reading in the whole Panel is not troublesome. But if anyone else who doesn't use pandas tries to read the HDF5 file, they will have more trouble accessing and understanding the data. What other factors should I be considering?
