Testing C++ code for endian-independence

Question

How can I test or check C++ code for endian-independence? It's already implemented, I would just like to verify that it works on both little- and big-endian platforms.

I could write unit tests and run them on the target platforms, but I don't have the hardware. Perhaps emulators?

Are there compile time checks that can be done?

Answer 1

If you have access to an x86-based Mac then you can take advantage of the fact that Mac OS X has PowerPC emulation built in as well as developer tool support for both x86 (little endian) and PowerPC (big endian). This enables you to compile and run a big and little endian executable on the same platform, e.g.

$ gcc -arch i386 foo.c -o foo_x86 # build little endian x86 executable
$ gcc -arch ppc foo.c -o foo_ppc  # build big endian PowerPC executable

Having built both big endian and little endian executables you can then run whatever unit tests you have available on both, which will catch some classes of endianness-related problems, and you can also compare any data generated by the executables (files, network packets, whatever) - this should obviously match.

Answer 2

You can set up an execution environment in the opposite endianness using qemu. For example if you have access to little-endian amd64 or i386 hardware, you can set up qemu to emulate a PowerPC Linux platform, run your code there.

Answer 3

I read a story that used Flint (Flexible Lint) to diagnose this kind of errors.

Don't know the specifics anymore, but let me google the story back for you:

http://www.datacenterworks.com/stories/flint.html

An Example: Diagnosing Endianness Errors

On a recent engagement, we were porting code from an old Sequent to a SPARC, and after the specific pointer issues we discussed in the Story of Thud and Blunder, we needed to look for other null pointer issues and also endian-ness errors.

Answer 4

I would suggest adapting a coding technique that avoids the problem all together.

First, you have to understand in which situation an endianess problem occurs. Then either find an endianess-agnostic way to write this, or isolate the code.

For example, a typical problem where endianess issues can occur is when you use memory accesses or unions to pick out parts of a larger value. Concretely, avoid:

long x;
...
char second_byte = *(((char *)&x) + 1);

Instead, write:

long x;
...
char second_byte = (char)(x >> 8)

Concatenation, this is one of my favorites, as many people tend to think that you can only do this using strange tricks. Don't do this:

union uu
{
  long x;
  unsigned short s[2];
};
union uu u;
u.s[0] = low;
u.s[1] = high;
long res = u.x;       

Instead write:

long res = (((unsigned long)high) << 16) | low

Answer 5

I could write unit tests and run them on the target platforms, but I don't have the hardware.

You can setup your design so that unit tests are easy to run independent of actually having hardware. You can do this using dependency injection. I can abstract away things like hardware interfaces by providing a base interface class that the code I'm testing talks to.

class IHw
{
public:
    virtual void SendMsg1(const char* msg, size_t size) = 0;
    virtual void RcvMsg2(Msg2Callback* callback) = 0;
     ...
};

Then I can have the concrete implementation that actually talks to hardware:

class CHw : public IHw
{
public:
    void SendMsg1(const char* msg, size_t size);
    void RcvMsg2(Msg2Callback* callback);
};

And I can make a test stub version:

class CTestHw : public IHw
{
public:
    void SendMsg1(const char* msg, size_t);
    void RcvMsg2(Msg2Callback* callback);
};

Then my real code can us the concrete Hw, but I can simulate it in test code with CTestHw.

class CSomeClassThatUsesHw
{
public:
   void MyCallback(const char* msg, size_t size)
   {
       // process msg 2
   }
   void DoSomethingToHw()
   {
       hw->SendMsg1();
       hw->RcvMsg2(&MyCallback);
   }
private:
    IHw* hw; 
}

Answer 6

I personally use Travis to test my software hosted on github and it supports running on multiple architectures [1], including s390x which is big endian.

I just had to add this to my .travis.yml:

arch:
  - amd64
  - s390x  # Big endian arch

It's probably not the only CI proposing this, but that's the one I was already using. I run both unit tests and integrated test on both systems which gives me some reasonable confidence that it works fine no matter the endianness.

It's no silver bullet though, I'd like to have an easy way to test it manually too just to ensure there's no hidden error (e.g I'm using SDL, colors could be wrong. I'm using screenshot to validate the output but the code for taking screenshot could have errors compensating the display problem, so the tests could pass with the display being wrong).

[1] https://blog.travis-ci.com/2019-11-12-multi-cpu-architecture-ibm-power-ibm-z

Answer 7

IMO, the only answer that comes close to being correct is Martin's. There are no endianness concerns to address if you aren't communicating with other applications in binary or reading/writing binary files. What happens in a little endian machine stays in a little endian machine if all of the persistent data are in the form of a stream of characters (e.g. packets are ASCII, input files are ASCII, output files are ASCII).

I'm making this an answer rather than a comment to Martin's answer because I am proposing you consider doing something different from what Martin proposed. Given that the dominant machine architecture is little endian while network order is big endian, many advantages arise if you can avoid byte swapping altogether. The solution is to make your application able to deal with wrong-endian inputs. Make the communications protocol start with some kind of machine identity packet. With this info at hand, your program can know whether it has to byte swap subsequent incoming packets or leave them as-is. The same concept applies if the header of your binary files has some indicator that lets you determine the endianness of those files. With this kind of architecture at hand, your application(s) can write in native format and can know how to deal with inputs that are not in native format.

Well, almost. There are other problems with binary exchange / binary files. One such problem is floating point data. The IEEE floating point standard doesn't say anything about how floating point data are stored. It says nothing regarding byte order, nothing about whether the significand comes before or after the exponent, nothing about the storage bit order of the as-stored exponent and significand. This means you can have two different machines of the same endianness that both follow the IEEE standard and you can still have problems communicating floating point data as binary.

Another problem, not so prevalent today, is that endianness is not binary. There are other options than big and little. Fortunately, the days of computers that stored things in 2143 order (as opposed to 1234 or 4321 order) are pretty much behind us, unless you deal with embedded systems.

Bottom line: If you are dealing with a near-homogenous set of computers, with only one or two oddballs (but not too odd), you might want to think of avoiding network order. If the domain has machines of multiple architectures, some of them very odd, you might have to resort to the lingua franca of network order. (But do beware that this lingua franca does not completely resolve the floating point problem.)