Clean way to make portable endian-correct file-reading / writing code in C++

Question

I want to write some C++ code that can read and write from files in an endian-correct way. More exactly, I want to be able to read a particular type of file, whose endianness I can easily detect (its magic number being reversed or not).

But how would I then go about cleanly reading the file correctly? I've read the following article which gives a helpful idea:

http://www.gamedev.net/page/resources/_/technical/game-programming/writing-endian-independent-code-in-c-r2091

The idea there is to make a class which has some function pointers to the endianness-correct read() function desired. But in my experience, function pointers are slow, especially when you have to call them so frequently as in this case. Another alternative is to have an

if (file_was_detected_big_endian) { read_bigendian(); } else { read_littleendian(); }

for every single read_x_bit_int() function I have but this too seems inefficient.

I'm using Boost so I have all its magnificence to help me. In particular, there's the endian sublibrary:

http://www.boost.org/doc/libs/develop/libs/endian/doc/buffers.html

Though I'm not sure how I can cleanly use this code to do what I want. I'd love to have some code where I can read say 16 bytes directly into the pointer of a struct that represents a part of the file while automatically correcting for endianness. I could of course write this code myself but I have a feeling a solid solution must already exist.

I think all the code I have will be manually padded and guarded against alignment issues.

Thanks!

The traditional way is to store the data in a well-known/fixed endian-ness (e.g. big-endian) and then use e.g. htonl()/htons() during writes and ntohl()/ntohs() during reads to handle the byte-swapping. In any case, your file writes and file reads are likely to be I/O bound, so CPU-cycle efficiency isn't likely to be a big concern. — Jeremy Friesner, May 26 '15 at 20:14
Unfortunately, it's not my format, and I cannot guarantee the endianness in the file itself. But I see what you mean if the endianness is known! — bombax, May 26 '15 at 20:16
In that case you can just write your own versions of hton*() and ntoh*() that check the state of file_was_detected_big_endian and do the right thing. I wouldn't worry about the overhead of the if() statement that is involved, since modern CPUs have branch prediction to minimize that. — Jeremy Friesner, May 26 '15 at 21:17

score 4 · Answer 1 · answered May 27 '15 at 19:40

So the virtual-function method that dasblinkenlight proposes will probably suffice - especially since I/O will probably be the dominant eater of time. However, if you do find that your read function is eating up a lot of cpu time, you can get rid of the virtual function dispatch by templatizing your file reader.

Here's some pseudocode demonstrating this:

Basically, create two reader classes, one for each endianness:

class LittleReader {
  public:
  LittleReader(std::istream& is) : m_is(is) {}
  char read_char() {//read byte from m_is}
  int read_int32() {//read 32-bit int and convert;}
  float read_float()....
  private:
  std::istream& m_is;
};

class BigReader {
  public:
  BigReader(std::istream& is): m_is(is){}
  char read_char(){...}
  int read_int32(){..}
  float read_float(){...}
  private:
  std::istream& m_is;
}

Separate out the main part of your reading logic (except for the magic number bit) into a function template that takes an instance of one of the above classes as an argument:

template <class Reader>
void read_endian(Reader &rdr){
  field1 = rdr.read_int32();
  field2 = rdr.read_float();
  // process rest of data file
  ...
}

Essentially, the compiler will create two implementations of your read_endian function - one for each endianness. Since there is no dynamic dispatch, the compiler can also inline all the calls to read_int32, read_float, etc.

Finally, in your main reader function, look at the magic number to determine which kind of reader to instantiate:

void read_file(std::istream& is){
  int magic(read_magic_no(is));
  if (magic == MAGIC_BIG_ENDIAN)
     read_endian(BigReader(is));
  else
     read_endian(LittleReader(is));
}

This technique gives you flexibility without incurring any virtual dispatch overhead, at the cost of increased (binary) code size. It can be very useful where you have extremely tight loops and you need to squeeze every last drop of performance.

If there is an existing library doing that will be great. – Nick Allen Jul 27 '20 at 02:45 — Nick Allen, Jul 27 '20 at 02:45

score 3 · Accepted Answer · edited May 23 '17 at 11:59

There are two approaches to this problem:

Write your files in an endianness-agnostic way, and
Add a marker, and read files in an endianness-aware way.

The first approach requires more work on writing, while the second approach makes writing "overhead-free".

Both approaches can be implemented without function pointers: the need for them has greatly diminished in C++ due to virtual functions^*.

Implementing both approaches is similar: you need to make an abstract base class for serializing primitive data types, make an instance of that class that reads the correct endianness, and call its virtual member functions for reading and writing:

struct PrimitiveSerializer {
    virtual void serializeInt(ostream& out, const int val) = 0;
    virtual void serializeChar(ostream& out, const char val) = 0;
    virtual void serializeString(ostream& out, const std::string& val) = 0;
    ...
    virtual int deserializeInt(istream& in) = 0;
    virtual char deserializeChar(istream& in) = 0;
    virtual std::string deserializeString(istream& in) = 0;
};
struct BigEndianSerializer : public PrimitiveSerializer {
    ...
};
struct LittleEndianSerializer : public PrimitiveSerializer {
    ...
};

Depending on the approach, the decision which subclass to use is made differently. If you use the first approach (i.e. writing endianness-agnostic files) then you would instantiate the serializer that matches your system's endianness. If you take the second approach, you would read the magic number from the file, and choose the subclass that matches the endianness of your file.

In addition, the first approach can be implemented using hton / ntoh functions.

^* Function pointers are not "slow" by themselves, although they make it easier to write inefficient code.

Ah yes, I've been writing C code for so long it's easy to forget you can use polymorphism like this! Thank you! It does raise a question however; do virtual functions like this require more overhead or does the compiler (often) do some magic tricks to minimize it? — bombax, May 26 '15 at 20:29
@bombax Virtual functions carry very little overhead on modern CPU. Pretty much the only thing the compiler cannot do is to inline them to speed things up eve further, but it takes a lot of optimization to make this kind of an overhead be relevant. In particular, there is no chance of this happening in code that reads and writes files. — Sergey Kalinichenko, May 26 '15 at 20:32
Perfect, thanks. I think I will use Boost's conversion functions for reading: http://www.boost.org/doc/libs/develop/libs/endian/doc/conversion.html I really like option 2 because it makes writing very easy and I don't have to worry about the endianness of the written file. — bombax, May 26 '15 at 20:36

bombax · Answer 3 · 2016-02-21T22:32:23.087

I've written a small .h and .cpp that can now handle (probably) all endianness problems. While I have adapted the functions for my own application, they might help someone.

endian_bis.h:

/**
 * endian_bis.h - endian-gnostic binary input stream functions
 * Copyright (C) 2015
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License along
 * with this program; if not, write to the Free Software Foundation, Inc.,
 * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 */

#pragma once

#include <cstdint>
#include <istream>

class BinaryInputStream {

public:
    inline int8_t   read_int8(std::istream &in)   { char buf[1]; in.read(buf, 1); return read_int8(buf, 0);   }
    inline int16_t  read_int16(std::istream &in)  { char buf[2]; in.read(buf, 2); return read_int16(buf, 0);  }
    inline int32_t  read_int32(std::istream &in)  { char buf[4]; in.read(buf, 4); return read_int32(buf, 0);  }
    inline int64_t  read_int64(std::istream &in)  { char buf[8]; in.read(buf, 8); return read_int64(buf, 0);  }
    inline uint8_t  read_uint8(std::istream &in)  { char buf[1]; in.read(buf, 1); return read_uint8(buf, 0);  }
    inline uint16_t read_uint16(std::istream &in) { char buf[2]; in.read(buf, 2); return read_uint16(buf, 0); }
    inline uint32_t read_uint32(std::istream &in) { char buf[4]; in.read(buf, 4); return read_uint32(buf, 0); }
    inline uint64_t read_uint64(std::istream &in) { char buf[8]; in.read(buf, 8); return read_uint64(buf, 0); }
    inline float    read_float(std::istream &in)  { char buf[4]; in.read(buf, 4); return read_float(buf, 0);  }
    inline double   read_double(std::istream &in)  { char buf[8]; in.read(buf, 8); return read_double(buf, 0); }

    inline int8_t    read_int8(char buf[], int off)  { return (int8_t)buf[off]; }
    inline uint8_t   read_uint8(char buf[], int off) { return (uint8_t)buf[off]; }
    virtual int16_t  read_int16(char buf[], int off)   = 0;
    virtual int32_t  read_int32(char buf[], int off)   = 0;
    virtual int64_t  read_int64(char buf[], int off)   = 0;
    virtual uint16_t read_uint16(char buf[], int off)  = 0;
    virtual uint32_t read_uint32(char buf[], int off)  = 0;
    virtual uint64_t read_uint64(char buf[], int off)  = 0;
    virtual float    read_float(char buf[], int off)   = 0;
    virtual double   read_double(char buf[], int off)  = 0;

    static BinaryInputStream *endianCorrectStream(int streamIsBigEndian);
    static BinaryInputStream *endianCorrectStream(std::istream &in,
                                                  uint32_t expectedBigEndianMagic,
                                                  uint32_t expectedLittleEndianMagic);

};

endian_bis.cpp:

/**
 * endian_bis.cpp - endian-gnostic binary input stream functions
 * Copyright (C) 2015 Jonah Schreiber (jonah.schreiber@gmail.com)
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License along
 * with this program; if not, write to the Free Software Foundation, Inc.,
 * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 */

#include "endian_bis.h"

#include <cstring>

/*
 * Delegated functions
 */

static inline int16_t  read_be_int16(char buf[], int off) {
    return (int16_t)(((buf[off]   & 0xff) << 8) |
                     ((buf[off+1] & 0xff)));
}

static inline int32_t  read_be_int32(char buf[], int off) {
    return (int32_t)(((buf[off]   & 0xff) << 24) |
                     ((buf[off+1] & 0xff) << 16) |
                     ((buf[off+2] & 0xff) << 8)  |
                     ((buf[off+3] & 0xff)));
}

template<int> static inline int64_t read_be_int64(char buf[], int off); // template indicates default word size (size_t)
template<> inline int64_t read_be_int64<4>(char buf[], int off) {
    return (((int64_t)(((buf[off]   & 0xff) << 24) |
                       ((buf[off+1] & 0xff) << 16) |
                       ((buf[off+2] & 0xff) << 8)  |
                       ((buf[off+3] & 0xff)))
                      ) << 32) | (
             (int64_t)(((buf[off+4] & 0xff) << 24) |
                       ((buf[off+5] & 0xff) << 16) |
                       ((buf[off+6] & 0xff) << 8)  |
                       ((buf[off+7] & 0xff))));
}

static inline uint16_t read_be_uint16(char buf[], int off) {
    return (uint16_t)(((buf[off]   & 0xff) << 8) |
                      ((buf[off+1] & 0xff)));
}

static inline uint32_t read_be_uint32(char buf[], int off) {
    return (uint32_t)(((buf[off]   & 0xff) << 24) |
                      ((buf[off+1] & 0xff) << 16) |
                      ((buf[off+2] & 0xff) << 8)  |
                      ((buf[off+3] & 0xff)));
}

template<int> static inline uint64_t read_be_uint64(char buf[], int off); // template indicates default word size (size_t)
template<> inline uint64_t read_be_uint64<4>(char buf[], int off) {
    return (((uint64_t)(((buf[off]   & 0xff) << 24) |
                        ((buf[off+1] & 0xff) << 16) |
                        ((buf[off+2] & 0xff) << 8)  |
                        ((buf[off+3] & 0xff)))
                       ) << 32) | (
             (uint64_t)(((buf[off+4] & 0xff) << 24) |
                        ((buf[off+5] & 0xff) << 16) |
                        ((buf[off+6] & 0xff) << 8)  |
                        ((buf[off+7] & 0xff))));
}

inline static int16_t  read_le_int16(char buf[], int off) {
    return (int16_t)(((buf[off+1] & 0xff) << 8) |
                     ((buf[off]   & 0xff)));
}

inline static int32_t  read_le_int32(char buf[], int off) {
    return (int32_t)(((buf[off+3] & 0xff) << 24) |
                     ((buf[off+2] & 0xff) << 16) |
                     ((buf[off+1] & 0xff) << 8)  |
                     ((buf[off]   & 0xff)));
}

template<int> static inline int64_t read_le_int64(char buf[], int off); // template indicates default word size (size_t)
template<> inline int64_t read_le_int64<4>(char buf[], int off) {
    return (((int64_t)(((buf[off+7] & 0xff) << 24) |
                       ((buf[off+6] & 0xff) << 16) |
                       ((buf[off+5] & 0xff) << 8)  |
                       ((buf[off+4] & 0xff)))
                      ) << 32) | (
             (int64_t)(((buf[off+3] & 0xff) << 24) |
                       ((buf[off+2] & 0xff) << 16) |
                       ((buf[off+1] & 0xff) << 8)  |
                       ((buf[off]   & 0xff))));
}

inline static uint16_t read_le_uint16(char buf[], int off) {
    return (uint16_t)(((buf[off+1] & 0xff) << 8) |
                      ((buf[off]   & 0xff)));
}

inline static uint32_t read_le_uint32(char buf[], int off) {
    return (uint32_t)(((buf[off+3] & 0xff) << 24) |
                      ((buf[off+2] & 0xff) << 16) |
                      ((buf[off+1] & 0xff) << 8)  |
                      ((buf[off]   & 0xff)));
}

template<int> static inline uint64_t read_le_uint64(char buf[], int off); // template indicates default word size (size_t)
template<> inline uint64_t read_le_uint64<4>(char buf[], int off) {
    return (((uint64_t)(((buf[off+7] & 0xff) << 24) |
                        ((buf[off+6] & 0xff) << 16) |
                        ((buf[off+5] & 0xff)<< 8)  |
                        ((buf[off+4] & 0xff)))
                      ) << 32) | (
             (uint64_t)(((buf[off+3] & 0xff) << 24) |
                        ((buf[off+2] & 0xff) << 16) |
                        ((buf[off+1] & 0xff) << 8)  |
                        ((buf[off]   & 0xff))));
}

/* WARNING: UNTESTED FOR 64 BIT ARCHITECTURES; FILL IN 3 MORE METHODS LIKE THIS TO TEST
   THE CORRECT FUNCTION WILL BE SELECTED AUTOMATICALLY AT COMPILE TIME
template<> inline uint64_t read_uint64_branch<8>(char buf[], int off) {
    return (int64_t)((buf[off]   << 56) |
                     (buf[off+1] << 48) |
                     (buf[off+2] << 40) |
                     (buf[off+3] << 32) |
                     (buf[off+4] << 24) |
                     (buf[off+5] << 16) |
                     (buf[off+6] << 8)  |
                     (buf[off+7]));
}*/

inline static float  read_matching_float(char buf[], int off) {
    float f;
    memcpy(&f, &buf[off], 4);
    return f;
}

inline static float  read_mismatched_float(char buf[], int off) {
    float f;
    char buf2[4] = {buf[3], buf[2], buf[1], buf[0]};
    memcpy(&f, buf2, 4);
    return f;
}

inline static double  read_matching_double(char buf[], int off) {
    double d;
    memcpy(&d, &buf[off], 8);
    return d;
}

inline static double  read_mismatched_double(char buf[], int off) {
    double d;
    char buf2[8] = {buf[7], buf[6], buf[5], buf[4], buf[3], buf[2], buf[1], buf[0]};
    memcpy(&d, buf2, 4);
    return d;
}


/*
 * Types (singleton instantiations)
 */

/*
 * Big-endian stream, Big-endian runtime
 */
static class : public BinaryInputStream {

public:
    int16_t  read_int16(char buf[], int off)  { return read_be_int16(buf, off); }
    int32_t  read_int32(char buf[], int off)  { return read_be_int32(buf, off); }
    int64_t  read_int64(char buf[], int off)  { return read_be_int64<sizeof(size_t)>(buf, off); }
    uint16_t read_uint16(char buf[], int off) { return read_be_uint16(buf, off); }
    uint32_t read_uint32(char buf[], int off) { return read_be_uint32(buf, off); }
    uint64_t read_uint64(char buf[], int off) { return read_be_uint64<sizeof(size_t)>(buf, off); }
    float    read_float(char buf[], int off)  { return read_matching_float(buf, off); }
    double   read_double(char buf[], int off) { return read_matching_double(buf, off); }
} beStreamBeRuntime;

/*
 * Big-endian stream, Little-endian runtime
 */
static class : public BinaryInputStream {

public:
    int16_t  read_int16(char buf[], int off)  { return read_be_int16(buf, off); }
    int32_t  read_int32(char buf[], int off)  { return read_be_int32(buf, off); }
    int64_t  read_int64(char buf[], int off)  { return read_be_int64<sizeof(size_t)>(buf, off); }
    uint16_t read_uint16(char buf[], int off) { return read_be_uint16(buf, off); }
    uint32_t read_uint32(char buf[], int off) { return read_be_uint32(buf, off); }
    uint64_t read_uint64(char buf[], int off) { return read_be_uint64<sizeof(size_t)>(buf, off); }
    float    read_float(char buf[], int off)  { return read_mismatched_float(buf, off); }
    double   read_double(char buf[], int off) { return read_mismatched_double(buf, off); }
} beStreamLeRuntime;

/*
 * Little-endian stream, Big-endian runtime
 */
static class : public BinaryInputStream {

public:
    int16_t  read_int16(char buf[], int off)  { return read_le_int16(buf, off); }
    int32_t  read_int32(char buf[], int off)  { return read_le_int32(buf, off); }
    int64_t  read_int64(char buf[], int off)  { return read_le_int64<sizeof(size_t)>(buf, off); }
    uint16_t read_uint16(char buf[], int off) { return read_le_uint16(buf, off); }
    uint32_t read_uint32(char buf[], int off) { return read_le_uint32(buf, off); }
    uint64_t read_uint64(char buf[], int off) { return read_le_uint64<sizeof(size_t)>(buf, off); }
    float    read_float(char buf[], int off)  { return read_mismatched_float(buf, off); }
    double   read_double(char buf[], int off) { return read_mismatched_double(buf, off); }
} leStreamBeRuntime;

/*
 * Little-endian stream, Little-endian runtime
 */
static class : public BinaryInputStream {

public:
    int16_t  read_int16(char buf[], int off)  { return read_le_int16(buf, off); }
    int32_t  read_int32(char buf[], int off)  { return read_le_int32(buf, off); }
    int64_t  read_int64(char buf[], int off)  { return read_le_int64<sizeof(size_t)>(buf, off); }
    uint16_t read_uint16(char buf[], int off) { return read_le_uint16(buf, off); }
    uint32_t read_uint32(char buf[], int off) { return read_le_uint32(buf, off); }
    uint64_t read_uint64(char buf[], int off) { return read_le_uint64<sizeof(size_t)>(buf, off); }
    float    read_float(char buf[], int off)  { return read_matching_float(buf, off); }
    double   read_double(char buf[], int off) { return read_matching_double(buf, off); }
} leStreamLeRuntime;

/*
 * "Factory" singleton methods (plus helper)
 */

static inline int isRuntimeBigEndian() {
    union { int32_t i; int8_t c[4]; } bint = {0x01020304};
    return bint.c[0] == 1;
}

BinaryInputStream *BinaryInputStream::endianCorrectStream(int streamIsBigEndian) {
    if (streamIsBigEndian) {
        if (isRuntimeBigEndian()) {
            return &beStreamBeRuntime;
        } else {
            return &beStreamLeRuntime;
        }
    } else {
        if (isRuntimeBigEndian()) {
            return &leStreamBeRuntime;
        } else {
            return &leStreamLeRuntime;
        }
    }
}

BinaryInputStream *BinaryInputStream::endianCorrectStream(std::istream &in,
                                                          uint32_t expectedBigEndianMagic,
                                                          uint32_t expectedLittleEndianMagic) {
    uint32_t magic = ((BinaryInputStream*)&beStreamBeRuntime)->read_uint32(in);
    if (magic == expectedBigEndianMagic) {
        if (isRuntimeBigEndian()) {
            return &beStreamBeRuntime;
        } else {
            return &beStreamLeRuntime;
        }
    } else if (magic == expectedLittleEndianMagic) {
        if (isRuntimeBigEndian()) {
            return &leStreamBeRuntime;
        } else {
            return &leStreamLeRuntime;
        }
    } else {
        return 0; /* not expected magic number */
    }
}

Suggested use:

BinaryInputStream *bis = BinaryInputStream::endianCorrectStream(in, 0x01020304, 0x04030201);

if (bis == 0) {
    cerr << "error: infile is not an Acme EarthQUAKEZ file" << endl;
    return 1;
}

in.ignore(4);
int32_t number = bis->read_int32(in);
...

(a) to implement `read_?int64_branch<8>` you need to cast to `?int64_t` before you shift e.g. `((int64_t)(buf[off]) << 56) | ... ` and (b) your signed-int read functions, `read_float` and `read_double` definitely aren't portable since you assume all computers represent the same numbers with the same bit patterns. — beerboy, May 28 '15 at 03:14
Are you sure about this? I thought a shift of anything caused an automatic cast to `int` which would be 64-bit. — bombax, May 28 '15 at 23:47

Clean way to make portable endian-correct file-reading / writing code in C++

3 Answers3