Replace hdf5 with sqlite

Question

As hdf5 cannot handle well new data in a same file (files become bigger). What would be the drawback to replace it with sqlite + sql akchemey ?

Answer 1

The HDF5 library will track free space and try to shrink the file if possible while it’s open. If the persistent file free space tracking setting is enabled, that will carry over through close->reopen cycles also. Running h5repack on the file will eliminate the wasted space also.