2

I am concerned with, I guess, a relatively simple problem.

I currently conduct a simulation where I track time, position and orientation of one particle for a given number of simulation steps.

The task is: Simply write this data to a h5-file on the fly.

So far, I did this using Jupyter. With the h5py-package it is very simple to create a data set of predefined structure (columns x rows) via

outfile = h5py.File("outfile.h5", "w")

dset = outfile.create_dataset("dsetname", (number_of_lines, number_of_columns))

and then write the data line by line for each simulation time step to the data set with

dset[time_step] = np.array([t, x, phi])

Now, I moved to C++, implemented the simulation there and would like to store the data in the same way I used to do it with Python.

However, from basic examples like this, one would have to store the whole set of data in an array during the simulation run and then write its content to the h5 file afterwards.

This is not very elegant. As I did with Python, I would like to just write the data line by line to the HDF5 data set on the fly – and not store the (sometimes up to several GB) large amounts of data in an array.

Unfortunately, so far, I did not find a way how I can implement the procedure I used with Python into C++.

Has anybody ever encountered a similar problem and could show me a way how to solve this it?

Thank you!

Best, Sven

Svennard
  • 71
  • 1
  • 8
  • maybe this helps : [Writing & Appending arrays of float to the only dataset in hdf5 file in C++](https://stackoverflow.com/questions/15379399/writing-appending-arrays-of-float-to-the-only-dataset-in-hdf5-file-in-c) – Thomas Jul 26 '18 at 17:32
  • There is also the option to use extendible datasets: https://support.hdfgroup.org/HDF5/Tutor/extend.html – Tom de Geus Jul 27 '18 at 05:51
  • Perfect! Thank you so much for pointing me into the right direction. It was the "count" and "offset" arguments one can put into the "selectHyperslab" method, which is exactly what I needed. There, was does not even need to extend or append the data set. – Svennard Jul 27 '18 at 12:17

0 Answers0