Binary "bulk" serialization of bitfields

Question

I have objects with lots of boolean properties, so I am using bitfields to pack the properties more compact. I also want to be able to serialize and deserializa those properties in a compact way, e.g. not field by field but by serializing and deserializing the 64 bit uint that holds the fields. This way is not only much faster (e.g. avoid all the shifting and stuff) but 8 times more memory efficient.

However, I read that the standard provides no guarantee that bitfield implementation will be uniform across different platforms. Can I expect that "bulk" binary serialization of the bitfield container will produce uniform results across platforms? Or perhaps it will be safer to go for manual shifting and masking when working with the properties so that bulk serialization and deserialization is possible?

what kind of bitfield are you using? `std::bitset`? – MatthiasB Jul 29 '14 at 14:35 — MatthiasB, Jul 29 '14 at 14:35
@MatthiasB - no plain old C bit fields. – Jul 29 '14 at 14:47 — , Jul 29 '14 at 14:47

score 1 · Answer 1 · answered Jul 29 '14 at 14:47

You can have a look at std::bitset:

It provides well defined functions to cast your bits into an unsigned long long, and create a bitset from a stored unsigned long long. It is defined that the first bit in a bitset is the least significant digit of the ullong representation.

so you can have something like:

 std::bitset<N> bits;
 unsigned long long val = bits.to_ullong();
 // serialize your ullong value
 // load ullong from serialized data
 unsigned long long val2 = ...;
 std::bitset<N> newBits(val2);

So, as long as your serialization can load/store unsigned long long correctly, you are good to go.

The only problem is when you have a bitfield which is too big for a unsigned long long. In this case, the standard provides no simple way to extract the bitfield.

Paul Thorpe · Answer 2 · 2014-07-30T15:18:56.897

One possibility would be to use ASN.1 to handle this via a BIT STRING. It precisely defines the serialization in a way that is independent of local representation. This allows it to be consistent across platforms regardless of whether the local platform is big-endian or little-endian. You can play with a free online ASN.1 compiler and encoder/decoder at http://asn1-playground.oss.com to see the resulting serialization.

ASN.1 allows you also to give a "name" to each bit so that you can easily set or check each named bit in a bit string.

score 0 · Answer 3 · answered Jul 29 '14 at 14:40

The variation in endian-ness of platforms suggest that any such serialization would be non-portable. Based on that I would say that you cannot expect bulk binary serialization of the bitfield container to be uniform across platforms.

A solution would have to account for the bit ordering and correct depending on the platform.

Goblinhack · Answer 4 · 2019-09-07T13:28:19.640

this might help you, it's a little example of various serialization types. I added bitset and raw bit values, that can be used like the below.

(all examples at https://github.com/goblinhack/simple-c-plus-plus-serializer)

class BitsetClass {
public:
    std::bitset<1> a;
    std::bitset<2> b;
    std::bitset<3> c;

    unsigned int d:1; // need c++20 for default initializers for bitfields
    unsigned int e:2;
    unsigned int f:3;
    BitsetClass(void) { d = 0; e = 0; f = 0; }

    friend std::ostream& operator<<(std::ostream &out,
                                    Bits<const class BitsetClass & > const m
    {
        out << bits(my.t.a);
        out << bits(my.t.b);
        out << bits(my.t.c);

        std::bitset<6> s(my.t.d | my.t.e << 1 | my.t.f << 3);
        out << bits(s);

        return (out);
    }

    friend std::istream& operator>>(std::istream &in,
                                    Bits<class BitsetClass &> my)
    {
        std::bitset<1> a;
        in >> bits(a);
        my.t.a = a;

        in >> bits(my.t.b);
        in >> bits(my.t.c);
        std::bitset<6> s;
        in >> bits(s);

        unsigned long raw_bits = static_cast<unsigned long>(s.to_ulong());
        my.t.d = raw_bits & 0b000001;
        my.t.e = (raw_bits & 0b000110) >> 1;
        my.t.f = (raw_bits & 0b111000) >> 3;

        return (in);
    }
};

Binary "bulk" serialization of bitfields

4 Answers4