Is there a way to save each HDF5 data set as a .csv column?

Question

I'm struggling with a H5 file to extract and save data as a multi column csv. as shown in the picture the structure of h5 file consisted of main groups (Genotypes, Positions, and taxa). The main group, Genotypes contains more than 1500 subgroups (genotype partial names) and each subgroup contains sub-sun groups (complete name of genotypes).There are about 1 million data sets (named calls) -each one is laid in one sub-sub group - which i need them to be written - each one - in a separate column. The problem is that when i use h5py (group.get function) i have to use the path of any calls. I extracted the all paths containing "calls" at the end of path but I cant reach all 1 million calls to get them into a csv file. could anybody help me to extracts "calls" which are 8bit integer i\as a separate columns in a csv file. By running the code in first answer I get this error:

1. Traceback (most recent call last): File "path/file.py", line 32, in h5r.visititems(dump_calls2csv) #NOTE: function name is NOT a string! File "path/file.py", line 565, in visititems return h5o.visit(self.id, proxy) File "h5py_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
   File "h5py\h5o.pyx", line 355, in h5py.h5o.visit File "h5py\defs.pyx", line 1641, in h5py.defs.H5Ovisit_by_name File "h5py\h5o.pyx", line 302, in h5py.h5o.cb_obj_simple File "path/file.py", line 564, in proxy return func(name, self[name]) File "path/file.py", line 10, in dump_calls2csv np.savetxt(csvfname, arr, fmt='%5d', delimiter=',') File "<array_function internals>", line 6, in savetxt File "path/file.py", line 1377, in savetxt open(fname, 'wt').close() OSError: [Errno 22] Invalid argument: 'Genotypes_ArgentineFlintyComposite-C(1)-37-B-B-B2-1-B25-B2-B?-1-B:100000977_calls.csv

Answer 1

16-May-2020 Update:

• Added a second example that reads and exports using Pytables (aka tables) using .walk_nodes(). I prefer this method over h5py .visititems()
• For clarity, I separated the code that creates the example file from the 2 examples that read and export the CSV data.

Enclosed below are 2 simple examples that show how to recursively loop on all top level objects. For completeness, the code to create the test file is at the end of this post.

Example 1: with h5py
This example uses the .visititems() method with a callable function (dump_calls2csv).
Summary of this procedure:
1) Checks for dataset objects with calls in the name.
2) When it finds a matching object it does the following:
a) reads the data into a Numpy array,
b) creates a unique file name (using string substitution on the H5 group/dataset path name to insure uniqueness),
c) writes the data to the file with numpy.savetxt().

import h5py
import numpy as np

def dump_calls2csv(name, node):    

    if isinstance(node, h5py.Dataset) and 'calls' in node.name :
       print ('visiting object:', node.name, ', exporting data to CSV')
       csvfname = node.name[1:].replace('/','_') +'.csv'
       arr = node[:]
       np.savetxt(csvfname, arr, fmt='%5d', delimiter=',')

##########################    

with h5py.File('SO_61725716.h5', 'r') as h5r :        
    h5r.visititems(dump_calls2csv) #NOTE: function name is NOT a string!

If you want to get fancy, you can replace arr in np.savetxt() with node[:].
Also, you you want headers in your CSV, extract and reference the dtype field names from the dataset (I did not create any in this example).

Example 2: with PyTables (tables)
This example uses the .walk_nodes() method with a filter: classname='Leaf'. In PyTables, a leaf can be any of the storage classes (Arrays and Table).
The procedure is similar to the method above. walk_nodes() simplifies the process to find datasets and does NOT require a call to a separate function.

import tables as tb
import numpy as np

with tb.File('SO_61725716.h5', 'r') as h5r :     
    for node in h5r.walk_nodes('/',classname='Leaf') :         
       print ('visiting object:', node._v_pathname, 'export data to CSV')
       csvfname = node._v_pathname[1:].replace('/','_') +'.csv'
       np.savetxt(csvfname, node.read(), fmt='%d', delimiter=',')

For completeness, use the code below to create the test file used in the examples.

import h5py
import numpy as np

ngrps = 2
nsgrps = 3
nds = 4
nrows = 10
ncols = 2

with h5py.File('SO_61725716.h5', 'w') as h5w :    
    for gcnt in range(ngrps):
        grp1 = h5w.create_group('Group_'+str(gcnt))
        for scnt in range(nsgrps):
            grp2 = grp1.create_group('SubGroup_'+str(scnt))
            for dcnt in range(nds):
                i_arr = np.random.randint(1,100, (nrows,ncols) )
                ds = grp2.create_dataset('calls_'+str(dcnt), data=i_arr)