Save Data in C++, Load from Python - Recommended Data Formats

Question

I have a ROS/CPP simulator that saves large amounts of data to a rosbag (around 90 MB). I want to read this data frequently from Python and since reading rosbags is slow and cumbersome, I currently have another python script that reads the rosbag and saves the relevant contents to a HDF5 file.

It would be nice though to be able to just save the data from the simulator directly (in C++) and then read it from my scripts (in Python). So I was wondering which data format I should use. It should be:

Fast to load from Python
Be compact (so ideally a binary of some sort)
Be easy to use

You might be wondering why I don't just save to HDF5 from my C++ simulator, but it just doesn't seem to be easy. There is basically nothing on forums such as Stackoverflow and the HDF5 Group website is opaque, seems to have some complicated licensing and very poor examples. I just want something quick and dirty that I can get running this afternoon.

score 2 · Accepted Answer · answered Aug 30 '19 at 18:44

2

You may want to have a look at HDFql as it is a high-level language (similar to SQL) to manage HDF5 files. Amongst others, HDFql supports C++ and Python. There are some examples that illustrates how to use HDFql in these languages here.

answered Aug 30 '19 at 18:44

SOG

876
6
10

Hi @SOG, I have been trying to get HDFql running on my system, but been running into problem after problem (as described here: https://stackoverflow.com/questions/57780098/linking-external-hdfql-library-in-cmake). Perhaps you would consider dropping by on that post and giving some advice? Many thanks! – Mr Squid Sep 04 '19 at 02:00
1

@MrSquid just dropped a comment in the suggested post. – SOG Sep 04 '19 at 07:30

score 0 · Answer 2 · answered Aug 30 '19 at 00:36

0

I see two solutions that can be useful for your problem :

LV: Length Value that you can store directly in binary into a file.
JSON: This does not add many data more than you need, and there are many libraries in Python or C++ that can simplify you the work

answered Aug 30 '19 at 00:36

cocool97

1,201
1
10
22

JSON is going to be too bulky for me. If I save the data as JSON it winds up being substantially bigger than HDF, my files are pretty huge as it is... – Mr Squid Aug 30 '19 at 01:49
I see... The best thing would be LV for you so ! – cocool97 Aug 30 '19 at 06:23

score 0 · Answer 3 · answered Aug 30 '19 at 01:02

0

Protocol Buffers is an option with language bindings in C++ and Python, though it might be more time investment than quick/dirty running this afternoon.

answered Aug 30 '19 at 01:02

Doug Richardson

10,483
6
51
77

Save Data in C++, Load from Python - Recommended Data Formats

3 Answers3