Binary Serialization of multiple data types

Question

I'm trying to serialize a struct in C++ in Visual Studio with multiple data types into binary file and de-serialize them all at once. But facing problem with memory allocation with strings while reading the data back. I know what the problem is and that there are other open source libraries which I can use but I do not want to use them unless it is really necessary and I also know that I can write/read data types one by one but that method is too long for struct containing large number of data types. I want to perform write/read operations in one go without using any open source library. Here is a struct for example:

struct Frame {
    bool isPass{ true };
    uint64_t address{ 0 };
    uint32_t age{ 0 };
    float marks{ 0.0 };
    std::string userName;
};

is there any way to perform write/read operation in one go in binary format? Thankyou

I would strongly suggest you to use the open source library (boost is excellent) serialization is full of nuances and pitfalls. As soon as you are not dealing with a trivial type you are in trouble. Furthermore, even for trivial type you will face compatibility issue with big and small edian. — Alessandro Teruzzi, Aug 19 '21 at 14:06
so protobuf or boost::serialization are my only options ,thankyou guys... — Rav, Aug 19 '21 at 14:27

score 1 · Answer 1 · answered Aug 19 '21 at 14:12

Not using existing libraries is NEVER good. Still,...

You could, for example, create a create pure virtual class like

class Serializable
{
public:
    virtual std::vector<char> serialize() = 0;
}

Then:

You implement it for all your own classes that you have
You implement serialization methods for all STL and PoD types that you use (std::strings, PoD types and structs with only PoD types) inside of some static class. Basically, during serialization you can put there something like [size][type][data ~ [size][type][data][size][type][data]].
Then, when you process a class for serialization, you create a buffer, first put a size into it, then type identifier, then put all bytes from all members serialized by those you have implemented in 1) and 2)
When you read anything from such an array, you do the same backwards: read N bytes from an array (first field), determine its actual type (second field), read all members, deserialize all stuff included.

The process is recursive.

But... man, its really a bad idea. Use protobuf, or boost::serialization. Or anything else - there's a lot of serialization libraries on the internet. Read these precious comments under your question. People are right.

score 0 · Answer 2 · answered Aug 19 '21 at 14:41

Assuming you have some other mechanism to keep track of your frame size rewrite your struct as:

struct Frame {
    bool isPass{ true };
    uint8_t pad1[3]{};
    uint32_t age{ 0 };
    uint64_t address{ 0 };
    double marks{ 0.0 };
    char userName[];
};

If we have a pointer Frame* frame. We can write this using write(fd, frame, frame_size). (frame_size > sizeof(frame)).

Assuming you have read the frame into a buffer, you can access the data using: auto frame = reinterpret<const Frame*>(buf) The length of userName will therefore be frame_size - sizeof(Frame). You can now access the elements through your struct.

The is very C like and the approach is limited to only one variable length element at the end of the array.

This is also limited to a single element per file. If it is always the same struct, you always know the type. But if there are strings in it, you need to remember object size as well to deserialize correctly. — Leontyev Georgiy, Aug 19 '21 at 16:48

Binary Serialization of multiple data types

2 Answers2