How to convert byte array to integral types (int, long, short, etc.) 'endian safely'?

Question

template <class T>
T readData(size_t position)
{
    byte rawData[sizeof(T)] = { 0, };

    // some logic that write data into rawData

    return *((T*)rawData);
}

Now I'm developing cross-platform game engine. but I heard that casting is absolutly dangerous because of endian difference. How can I convert rawData to type T endian safely without using conditions about endianness?

How is `byte` defined? [It shouldn't be standard type](http://stackoverflow.com/questions/20024690/is-there-byte-data-type-in-c). — MikeCAT, Feb 28 '16 at 14:49
Not only is it not endian-safe, it's also not alignment-safe or integer-size-safe. [Here's Rob PIke's solution](http://commandcenter.blogspot.com/2012/04/byte-order-fallacy.html). — Raymond Chen, Feb 28 '16 at 14:50
Do you know the endianness of the "raw" data? Do you know the endianness of the host platform? Where does the raw data come from? — Some programmer dude, Feb 28 '16 at 14:50
@MikeCAT sorry about that, byte is same with [code]unsigned char[/code] — sophia, Feb 28 '16 at 14:51
@RaymondChen thanks for your advice I take care of that problems — sophia, Feb 28 '16 at 14:52
@JoachimPileborg that "raw" data is from a file. and I want to implement 'readData' function Platform-Independently. — sophia, Feb 28 '16 at 14:54
You should know first how data is stored in the file: its byte order and size — Lol4t0, Feb 28 '16 at 14:58
A simple answer is to always encode integers using big-endian, and decode using Rob Pike's solution pointed to by @RaymondChen. — President James K. Polk, Feb 28 '16 at 14:58
@Lol4t0 It's just an .wav file and I want to read header to get some information of audio format. — sophia, Feb 28 '16 at 15:05
@JamesKPolk Thanks to answer, I'll check about Rob Pike's solution too. — sophia, Feb 28 '16 at 15:05
@JoachimPileborg It's just an .wav file, And I think raw data written using little-endian format. but I haven't no idea that can I read data without endian-safety. — sophia, Feb 28 '16 at 15:08
According to [this reference of the WAV file format](http://soundfile.sapp.org/doc/WaveFormat/) "The default byte ordering assumed for WAVE data files is little-endian. Files written using the big-endian byte ordering scheme have the identifier RIFX instead of RIFF. " So all you need to do to learn the byte order of the header and meta-data chunks is to read the first four bytes. This is of course unrelated to the actual audio data byte order. But there are hundreds if not thousands of libraries to read audio files, why not use one of them instead of reinventing the wheel? — Some programmer dude, Feb 28 '16 at 15:14
If your program does not communicate with other machines that potentially has a different endianess, then you don't really need to care. Endianess matters when you do network communication between machines, and serialize data to a file on one machine and to be read by another. — user3528438, Feb 28 '16 at 15:18
@JoachimPileborg Maybe It can be hear like stupid, I'll use only important libraries to study. (OpenAL, DirectSound, etc.) and so many licences are made me hard to develop free. — sophia, Feb 28 '16 at 15:43
[SDL_sound](https://www.icculus.org/SDL_sound/) is an old library library, but still works very well. It can handle most audio file formats, decode them, and can give you the raw data to be used by another library to play it. SDL_sound is licensed under the [GNU Lesser General Public License](https://en.wikipedia.org/wiki/GNU_Lesser_General_Public_License) which allows you to use it basically any way you see fit, including both free, open source *and* proprietary applications. — Some programmer dude, Feb 28 '16 at 16:46
There are many other libraries available to load audio files, using even less stringent or strict licenses (like one of the BSD and variants like Apache). — Some programmer dude, Feb 28 '16 at 16:49
@JoachimPileborg I didn't know about that library. I'll check that out. Thanks to your help! — sophia, Feb 28 '16 at 20:56

score 0 · Answer 1 · answered Feb 28 '16 at 17:13

0

There's no need for you to care. Unless your rawData comes from a different system (network stream, external peripheral, ...). As you develop a game engine i presume that is not the case.

Yes, you can do twisted things like writing data byte by byte and then reading them as integer but that is a design problem. You should avoid that rather than spending to much time worrying about endianness.

answered Feb 28 '16 at 17:13

Peter Silon

457
2
11

I really want you to know how thankful I am, I'll remember your advice. thanks! – sophia Feb 28 '16 at 20:53

score 0 · Accepted Answer · edited Apr 12 '17 at 07:31

You must know the endianness or the source data. Data is usually big-endian when being transferred over a network. Then you need to determine if your system is a little-endian or a big-endian machine. If the endianness of the data and the system is not the same, just reverse the bytes, and then use it.

You can determine the endianness of your system as follows:

int is_little_endian() {
  short a = 1;
  return *((char*)&a) & 1;
}

Convert from little/big endian to system endian and vice versa using these macros:

#define LITTLE_X_SYSTEM(dst_type, src) if(!is_little_endian()) memrev((src), 1 , sizeof(dst))
#define BIG_X_SYSTEM(dst_type, src) if(is_little_endian()) memrev((src), 1, sizeof(dst))

You can use it like this:

template <class T>
T readData(size_t position)
{
    byte rawData[sizeof(T)] = { 0, };
    // assuming source data is big endian
    BIG_X_SYSTEM(T, rawData);
    return *((T*)rawData);
}

This answer gives some more insight into endianness.

thanks for your good answer. I'll try that solution. Thank you! — sophia, Feb 28 '16 at 20:54

How to convert byte array to integral types (int, long, short, etc.) 'endian safely'?

2 Answers2