Initialize huge uint8_t array statically with reasonable compilation time

Question

I would like to statically initialize huge (megabytes) uint8_t array.

At the beginning I tried this:

constexpr uint8_t arr[HUGE_SIZE] = { 0, 255, ... };

Unfortunatelly, compilation time of above is very long (no optimization - around 30 seconds, optimizations on - above hour).

I found out that compilation time can be reduced to negligible (in both optimization off and on cases) if we use c style string initialization:

constexpr uint8_t arr[HUGE_SIZE + 1] = "\x00\xFF\x...";

Is this good approach in C++? Should I use some string literal to make types of both sides of above assignment equal?

Do you **really** need such a big array? Maybe you should think about the algorithm. — cadaniluk, Dec 28 '15 at 11:26
@mkk What is the array for, and do you actually need an array filled with memory at compile time? — Poriferous, Dec 28 '15 at 11:28
I am surprised, that optimizations make a difference in the first case. Can somebody explain that? — Ctx, Dec 28 '15 at 11:28
@Ctx: Probably because the compiler generates code to filll the array, which is dumped in some special function - this function then becomes enormous, and at least gcc and clang [more specifically llvm] do not like large functions due to some O(N^x) complexity. — Mats Petersson, Dec 28 '15 at 11:32
Is `constexpr` always static or is it more likely to be optimized away than just using `static`? Will the compile time with `constexpr` vary compared to the compile time of `static`? — ilent2, Dec 28 '15 at 11:33
What version of what compiler is giving you this hour+ compile-time? — Marc Glisse, Dec 28 '15 at 11:38
@Poriferous: this array stands for static dictionary (to be precise dictionary data); it would be great to don't load it from a file and don't keep it in free store, just have it in some compiled object — mkk, Dec 28 '15 at 11:40
@ilent2: when I wrote static, I meant initialized during compilation, not running; I guess compilation time does not vary when switching from constexpr to static, if you want, I can do a test today — mkk, Dec 28 '15 at 11:42
Note that it is possible to embed an arbitrary binary file in an object file (see many posts on this website), if your compiler can't be convinced to finish in a reasonable time. — Marc Glisse, Dec 28 '15 at 11:43
@MarcGlisse: I am using standard compiler provided in Command Line Tools for the newest version of XCode. I'll let you know precise version number in couple of hours (I don't have access for that machine in this moment) — mkk, Dec 28 '15 at 11:46
@FUZxxl: stackoverflow did not let me to notify you with MarcGlisse, please take a look at comment addressed to him — mkk, Dec 28 '15 at 11:47
@mkk The assembler included by XCode supports the syntax I mention as far as I know. — fuz, Dec 28 '15 at 11:49
Marking it as `constrexpr` suggests to the compiler "try to replace uses of this inline with the according value", which causes quite some compile overhead. How about using plain "const" instead? — Ulrich Eckhardt, Dec 28 '15 at 11:49
@UlrichEckhardt If you can back this up with facts, consider writing an answer. — fuz, Dec 28 '15 at 11:52
@UlrichEckhardt: as far as I remember, I tried const too but it did not fixed compilation time issue. — mkk, Dec 28 '15 at 11:54
Just to make it more explicit: Why did you mark this as `constexpr` at all? Do you understand the difference between `const` and `constexpr`? Did you actually put this into a header file? How often is that header file included? That said, please extract a minimal example that at least answers the above questions. Also, please clarify what `HUGE_SIZE` really is. — Ulrich Eckhardt, Dec 28 '15 at 11:59
@UlrichEckhardt: I used constexpr just to have a guarantee that compiler won't produce code of initialization of this array in runtime. Will const give that same guarantee? This code is placed in a source file, not a header. The lowest size of HUGE_SIZE where I observed long compilation time was around 200 000 (I did not tested lower values). — mkk, Dec 28 '15 at 12:18
I didn't manage to reproduce the huge compile-time difference between -O0 and -O3 with a file that has just the array (and a trivial function that returns the address of the array, so it is not eliminated). — Marc Glisse, Dec 28 '15 at 13:46
@mkk Is the array a global variable? Or is it local? If it's local, is it automatic or static? — fuz, Dec 28 '15 at 14:26

fuz · Accepted Answer · 2016-01-22T16:57:24.463

4

If the array is really large, consider using a utility to directly generate an object file from the array. For example, with the GNU assembler you can do something like this:

    .section .rodata # or .data, as needed
    .globl arr
arr:
    .incbin "arr.bin" # assuming arr.bin is a file that contains the data
    .size arr,.-arr

Then assemble this file with the GNU assembler and link it to your program. To use this data elsewhere in your program, just declare it as extern "C":

extern "C" const uint8_t arr[];

edited Jan 22 '16 at 16:57

answered Dec 28 '15 at 11:44

fuz

88,405
25
200
352

That would be a solution. Won't C++ programmers' world be unhappy with this approach? (-; – mkk Dec 28 '15 at 11:50
@mkk Well, with this solution you are operating outside of the realm of the C++ programming language. This applies to any other approach, too, though. In a portable open source application, I would take care to add a solution for Windows, too. – fuz Dec 28 '15 at 11:51
You're right. I am wondering if embedding it in cpp file (like I did in question) gives some guarantees, that are lost in case of your approach. For instance endianness could be corrupted when developping for various devices (so we are required to provide version of arr.bin for big-endian and little-endian), am I right? – mkk Dec 28 '15 at 12:13
@mkk Endianness does not matter if you use `uint8_t` for your element type—a single byte has no endianess. The `.incbin` directive literally dumps the file named after it into the object file without changing anything. Endianess is not affected. – fuz Dec 28 '15 at 12:19
Why not use plain C for this? – chqrlie Dec 28 '15 at 13:59
@chqrlie If OPs C++ compiler is horribly slow with this in C++, it's likely that it's just as slow with plain C. – fuz Dec 28 '15 at 14:25
@FUZxxl: not even likely, worth a try anyway. – chqrlie Dec 28 '15 at 16:20

score 1 · Answer 2 · answered Dec 28 '15 at 18:28

Found that compile time for large arrays does improve a bit if the array is broken down into smaller chunks. Yet the string approach is still significantly faster. With such a scheme, the true array could union this array of arrays.

Posting the below as an example of how to test OP's problem without explicitly coding million byte source files. As this is not much of an answer but a resource for investigation, marking this community wiki.

#include <iostream>
using namespace std;

#include <cstdint>

#define METHOD 5

#if METHOD == 1
// 1 byte blocks 28 secs
#define ZZ16 65, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255,
#define ZZ256 ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16
#define ZZ4K  ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256
#define ARR constexpr uint8_t arr[]
#define COUT cout << arr << endl

#elif METHOD == 2
// 16 byte blocks 16 secs
#define ZZ16 {66, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255},
#define ZZ256 ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16
#define ZZ4K  ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256
#define ARR constexpr uint8_t arr[][16]
#define COUT cout << arr[0] << endl

#elif METHOD == 3
// 256 byte blocks 16 secs
#define ZZ16 67, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255,
#define ZZ256 {ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16},
#define ZZ4K  ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256
#define ARR constexpr uint8_t arr[][256]
#define COUT cout << arr[0] << endl

#elif METHOD == 4
// 4K byte blocks 13 secs
#define ZZ16 68, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255,
#define ZZ256 ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16
#define ZZ4K  {ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256},
#define ARR constexpr uint8_t arr[][4096]
#define COUT cout << arr[0] << endl

#elif METHOD == 5
// String 4 sec
#define ZZ16 "\x45\xFF\x00\xFF\x00\xFF\x00\xFF\x00\xFF\x00\xFF\x00\xFF\x00\xFF"
#define ZZ256 ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16
#define ZZ4K  ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256
#define ARR constexpr uint8_t arr[]
#define COUT cout << arr << endl
#endif

#define ZZ64K ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K
#define ZZ1M  ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K
#define ZZ16M ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M

// 3 million bytes
ARR = {
    ZZ1M ZZ1M ZZ1M
};

int main() {
  cout << "!!!Hello World!!!" << endl;
  COUT;
  cout << sizeof(arr) << endl;
  return 0;
}

score 0 · Answer 3 · answered Dec 28 '15 at 11:48

0

Are you intending to recompile the file in which the array is defined very often? If not, you could put the definition of the array into a separate .cpp file with a forward declaration in a .h file. Thus you'll face compilation overhead only when the array changes.

answered Dec 28 '15 at 11:48

Oleg Andriyanov

5,069
1
22
36

During developing process array could change even once per day, so losing hour or more to compile it is not a case. Anyway thanks for answer. (-: – mkk Dec 28 '15 at 12:20

chqrlie · Answer 4 · 2015-12-28T14:06:59.607

0

Move the array definition to a separate C file and compile it as such. C++ can refer to external global data from C object modules.

If gcc takes too long to compile it, use tcc .

edited Dec 28 '15 at 14:06

answered Dec 28 '15 at 14:01

chqrlie

131,814
10
121
189

`constexpr uint8_t arr[HUGE_SIZE] = { 0, 255, ... };` does not compile in C. If you are suggesting different code, perhaps posting to clarify your answer. – chux - Reinstate Monica Dec 28 '15 at 16:01

Initialize huge uint8_t array statically with reasonable compilation time

4 Answers4