How do you save objects with variable length members to a binary file usefully?

Question

I'm working with c++ and I am writing a budget program (I'm aware many are available--it's just a learning project).

I want to save what I call a book object that contains other objects such as 'pages'. Pages also contain cashflows and entries. The issue is that there can be any amount of entries or cashflows.

I have found a lot of information on saving data to text files but that is not what I want to do.

I have tried looking into using the boost library as I've been told serialization might be the solution to this problem. I'm not entirely sure which functions is boost are going to help or even what the proper ways are to use boost.

Most examples of binary files that I have seen are with objects that have fixed size members. For example, a point might contain an x value and a y value that are both doubles. This will always be the case so it is simple to just use sizeOf(Point).

So, I'm either looking for direct answers to this question or useful links to information on how to solve my problem. But please make sure you links are specific to the question.

I've also posted the same question on cplusplus

Boost is not a library, it is a _collection_ of libraries. Boost.Serialization is only one of many; there are probably many libraries you could make use of if you're using C++03. — ildjarn, Jun 09 '12 at 18:32
@ildjarn: That might be true technically but making the distinction improves nothing in this context. — user541686, Jun 12 '12 at 06:16
@Mehrdad : It could improve the clarity of future questions of the OP (and other readers). Think outside the box. ;-] — ildjarn, Jun 12 '12 at 06:25
@ildjarn: I was -- that's why I said it doesn't matter for this question. ;) — user541686, Jun 12 '12 at 06:25
@Mehrdad : This context != this question. This context == SO. — ildjarn, Jun 12 '12 at 06:26
@ildjarn: I think you forgot to wrap an `assert()` around that. :P — user541686, Jun 12 '12 at 06:29

score 3 · Accepted Answer · edited May 23 '17 at 10:34

In general, there are two methods to store variable length records:

Store size integer first, followed by the data.
Store the data, append a sentinel character (or value) at the end.

C-style strings use the 2nd option.
For option one, the number contains the size of the data.

Optional Fields

If your considering relational database design for optional fields, you would have one table with the known or fixed records and another table containing an option field with the record ID.

A simpler route may be to go to something similar to XML: field labels.
Split your object into two sections: static fields and optional fields. The static field section would be followed by an optional field section. The optional field section would contain the field name, followed by the field data. Read in the field name then the value.

I suggest you review your design to see if optional fields can be eliminated. Also, for complex fields, have them read in their own data.

Storing Binary Data

If the data is shared between platforms, consider using ASCII or textual representation.

Read up upon Endianess and also bit sizes. For example one platform could store its binary representation least significant byte first and use 32 bits (4 bytes). The receiving platform, 64-bit, most significant byte first, would have problems reading the data directly and would need to convert; thus losing any benefit from binary storage.

Similarly, floating point doesn't fare well in binary either. There is also the loss of precision when converting between floating point formats.

When using optional fields in binary, one would use a sentinel byte or number for the field ID rather than a textual name.

Also, data in textual format is much easier to debug than data in binary format.

Consider using a Database

See At what point is it worth using a database?

So for example: suppose I want to write a book object call it b get its size and write that to file but then when reading back how would you distinguish between the members? For example some would be predictable because they would be fixed, but some would not. Or do you have to write each member along with its size? Then you would know that you have to read n members (with a size before each) before you can reconstruct the object. Does that seem right? — user838293, Jun 09 '12 at 20:11
If you have a book object with fields of variable lengths, each field would write its size, then content. — Thomas Matthews, Jun 10 '12 at 04:16
When using a database does this involve installing extra software to get the program to work? The only work I've done with databases has been with PHP and mysql. It does seem to me that this has to be an incredibly common issue: creating some type of non-readable file format. Do most programs use a database or do they write their serializing code. This topic isn't typically covered in c++ intro books so where are people learning these concepts. Sorry many questions. — user838293, Jun 12 '12 at 23:53
The issue is that binary data is platform dependent. To get around this, many protocols (data format definitions) specify the bit width of variables and their endianness. The program on the receiving platform (or the one reading the data) may need to perform extra instructions to convert the protocol to native formats. Only textual representations are truely portable. — Thomas Matthews, Jun 13 '12 at 14:29
A budget program is a good candidate for using databases. If you want to learn about databases, and want to construct your own, go ahead. Read up on tables, indices, B-Trees, linking records without using memory pointers. — Thomas Matthews, Jun 13 '12 at 14:32

score 0 · Answer 2 · edited Jun 09 '12 at 18:33

0

The boost::serialization documentation is here.

boost::serialization handles user-written classes as well as STL containers: std::deque, std::list, etc.

edited Jun 09 '12 at 18:33

ildjarn

62,044
9
127
211

answered Jun 09 '12 at 18:27

slaphappy

6,894
3
34
59

How do you save objects with variable length members to a binary file usefully?

2 Answers2

Optional Fields

Storing Binary Data

Consider using a Database