initialization list of C++ containers of static const causes stack overflow

Question

I have a static const variable of a std::vector as such:

std::vector<std::pair<GUID, std::array<double, 13>>>

I've also tried(vectors in theory take less memory):

std::unordered_map<GUID, std::array<double, 13>, HashGUID >
std::map<GUID, std::array<double, 13>, GUIDComparer >

I initialize it at the start of my program, with about 5400 items using the initializer list. I know that seems a bit large, but it's nothing out of the ordinary. Its a temp/intermediate solution.

However, it keeps throwing:

0xC00000FD: Stack overflow (parameters: 0x00000000, 0x00052000).

If I keep the list smaller then about 4000, it seems to work, but my full list of 5400, just doesn't. Any ideas why?

EDIT:

here is how I'm initializing (4000 or so of these lines work great. 5400+, no go):

static const std::vector<std::pair<GUID, std::array<double, 13>>> engVals={
    {{0x58341899, 0x8844, 0x3333, { 0x04, 0x04, 0x04, 0x04, 0x04, 0x04, 0x04, 0x04}}, {11, 101, 1.50, 3.50, 225.0, 850.0, 125.0,0.55, 19, 175, 565, 1.2, 0.44}},
    {{0x67633448, 0x8103, 0x3333, { 0x04, 0x04, 0x04, 0x04, 0x04, 0x04, 0x04, 0x04}}, {11, 102, 1.50, 3.50, 475, 1300.0, 275.0, 0.55, 19, 175, 565, 1.2, 0.44}},
    {{0x94422980, 0x6497, 0x3333, { 0x04, 0x04, 0x04, 0x04, 0x04, 0x04, 0x04, 0x04}}, {11, 103, 1.50, 3.50, 875.0, 1600.0, 500.0, 0.55, 19, 175, 565, 1.4, 0.51}},
...
};

EDIT 2 Forgot to specify. I'm using vs2013. This code is actually in a library compiled as a dll.

I need a initialization list because the above initialization will be generated by another app, so i need a one-liner way of initializing this container.

Do big initializer lists overflow the stack? I hadn't considered that before. — Neil Kirk, Jul 15 '14 at 21:41
Can you show us what kind of initialization is done with `initializer_list`s? Maybe you can use another kind of initialization. — dyp, Jul 15 '14 at 21:44
Thanks all for the quick reply. Updated my initializer list. not familiar with "ulimit" — mateuscb, Jul 15 '14 at 21:48
Wait... if it's `static` and `const` (size fixed and known at compile-time), then why is it a vector? Why not an `array`? — dyp, Jul 15 '14 at 21:50
@dyp, valid. I really wanted a map. the const is less important. But good point. — mateuscb, Jul 15 '14 at 21:50
When you change it to an `array`, you wouldn't be using `initializer_list`s any more, so the error might disappear. — dyp, Jul 15 '14 at 21:51
13*4 + 1*8=60 bytes per element, *5.4k = 324k of data. As an aside, why store in a `vector` and not an `array`? — Yakk - Adam Nevraumont, Jul 15 '14 at 21:57
No, a double is 8 bytes and a GUID 16 bytes. So 120 bytes per item. 4000 items (±470K) are ok, 5400 items (±630K) are not. So that looks like a 512K stack limit. That seems low, but not implausible so. — pdw, Jul 15 '14 at 22:05
I just tried g++4.10, and it can initialize a global `static const vector` of this kind can even with an `initializer_list` containing 12,000 elements. The resulting executable (at `-O0`) is rather large, of course, 1.4 MB. — dyp, Jul 15 '14 at 22:07
So, the reason I'm using an initialization list, is because this file will be generated. So, i need some sort of initialiation list. as for the size (16bytes (GUID) + 104bytes(double[13]) * 5400) = 648000 bytes. less than 1 Megabyte, doesn't seem like a lot. — mateuscb, Jul 15 '14 at 22:07
@pdw just saw your comment about 512K stack size (didn't refresh). Does seem low, but seems like an odd coincidence. — mateuscb, Jul 15 '14 at 22:11
*"using vs2013"* Ah, almost thought so. Since it doesn't have support for `constexpr`, I suspect that it needs to construct the `initializer_list` at runtime (initialization time), therefore requiring stack space. — dyp, Jul 15 '14 at 22:19
that did it. Stack size on my EXE project (not the dll ofcourse) Bumped it to 4MB and 40K (http://msdn.microsoft.com/en-us/library/8cxs58a6.aspx) Thanks all. Who should get credit for the answer? @dyp and @pdw? answer away! — mateuscb, Jul 15 '14 at 22:27
@mateuscb Permanently increasing stack size for the whole run of your program, for a one-off use case, is not efficient. — Neil Kirk, Jul 15 '14 at 23:08
@NeilKirk makes sense, what are downsides of larger stack size? — mateuscb, Jul 15 '14 at 23:22
@mateuscb More memory usage. If you only need 512kb (reasonable for a typical program) and you use 4mb, then you have 3.5mb unused for the rest of your program. Doesn't seem like much, but what if every program on your computer did that, and you had many open at the same time? — Neil Kirk, Jul 15 '14 at 23:25
@mateuscb: You keep saying it has to be in an initialization list because it's generated by another program, but that doesn't hold water. Simply have the other program generate to a binary file, and load the vector from that. Bypasses the problem entirely. http://ideone.com/MrVOEe — Mooing Duck, Jul 15 '14 at 23:33
@MooingDuck the values need to be in compiled, can't be easily editable by a user. Agree that I don't need initialization list, but would just really like it. — mateuscb, Jul 15 '14 at 23:49
@mateuscb: An external binary file is not much easier to edit than is the executable itself. Also has the side effect that you can update them separately, faster compile times, etc. — Mooing Duck, Jul 15 '14 at 23:50
@MooingDuck I'm not very familiar with how that works. Will def look into it. — mateuscb, Jul 16 '14 at 00:12
@MooingDuck I ended up going with a variation of your answer. Worked like a charm! Please add it as answer so I can give you credit, and of course choose an answer. No ones likes those unanswered questions ;) Thx! — mateuscb, Jul 17 '14 at 23:41

score 1 · Answer 1 · answered Jul 15 '14 at 21:52

1

Try @dyp suggestion of using std::array first.

If it doesn't work, you could try the following work around. I just made a simple example.

std::vector<int> CreateVector()
{
    std::vector<int> temp;
    temp.reserve(3);
    temp.push_back(1);
    temp.push_back(2);
    temp.push_back(3);
    return temp;
}

static const std::vector<int> data = CreateVector();

answered Jul 15 '14 at 21:52

Neil Kirk

21,327
9
53
91

thanks, but I really needed a solution that still used initialization list. I had not specified that in my question. – mateuscb Jul 15 '14 at 22:28
2

@mateuscb: I remain unconvinced that you need an initialization list – Mooing Duck Jul 15 '14 at 23:46

Stian Svedenborg · Answer 2 · 2014-07-15T23:35:16.293

Well the issue here I believe is the initializer list itself, as that is being pushed on the stack as you are calling the constructor to the outer vector.

What I suggest is moving back to plain old arrays and making that a file static or global constant.

struct GUI_xxx {
    GUID guid;
    double values[13];
};

const GUI_xxx engVal[] = {
     {{0x58341899, 0x8844, 0x3333, { 0x04, 0x04, 0x04, 0x04, 0x04, 0x04, 0x04, 0x04}}, {11, 101, 1.50, 3.50, 225.0, 850.0, 125.0,0.55, 19, 175, 565, 1.2, 0.44}},
     {{0x67633448, 0x8103, 0x3333, { 0x04, 0x04, 0x04, 0x04, 0x04, 0x04, 0x04, 0x04}}, {11, 102, 1.50, 3.50, 475, 1300.0, 275.0, 0.55, 19, 175, 565, 1.2, 0.44}},
     {{0x94422980, 0x6497, 0x3333, { 0x04, 0x04, 0x04, 0x04, 0x04, 0x04, 0x04, 0x04}}, {11, 103, 1.50, 3.50, 875.0, 1600.0, 500.0, 0.55, 19, 175, 565, 1.4, 0.51}},
     ...  
};

You will technically still be using the initializer list syntax, but (assuming GUID is simple enough) this may be optimized into a constant table in the global constant portion of your library. If you run into similar problems still, you could even try creating this constant as an external C constant in a separate .c file. (Assuming GUID is POD.)

really wanted that map functionality. I increased the stack size. Seemed to work for now. — mateuscb, Jul 15 '14 at 23:21

score 1 · Accepted Answer · answered Jul 17 '14 at 23:51

You keep saying it has to be in an initialization list because it's generated by another program, but that doesn't hold water. Simply have the other program generate to a binary file, and load the vector from that. This bypasses the problem entirely.

std::istream& operator>>(std::istream& in, std::pair<GUID, std::array<double, 13>>& var)
{return in.read((char*)&var, sizeof(var));}

std::vector<std::pair<GUID, std::array<double, 13>>> loadGUIDs()
{
    std::vector<std::pair<GUID, std::array<double, 13>>> ret;
    ret.reserve(5400); //or sizeof file / 120b or whatever
    std::ifstream infile("GUIDs.bin");
    std::istream_iterator<std::pair<GUID, std::array<double, 13>>> begin(infile),end;
    ret.assign(begin,end);
    return ret;
}
static const std::vector<std::pair<GUID, std::array<double, 13>>> engVals = loadGUIDs();

http://ideone.com/MrVOEe

As for worries about user editing, an external binary file is not much easier to edit than is the executable itself. Also has the side effect that you can update them separately, faster compile times, etc.

I made a small modification to this answer, instead of storing the file in the open, i added it as resource. It doesn't make it any harder, but at least its encapsulated within the EXE. — mateuscb, Jul 18 '14 at 17:18
@mateuscb: Actually, now that I've slept on it, you should be reading this into a `boost::flatmap` or `std::unordered_map` rather than a `std::vector`. Uses significantly more memory, but then you can do `std::array& data = engVals[myGUID];` in O(1) time. — Mooing Duck, Jul 18 '14 at 17:23
Exactly, I used the std::unordered_map! thanks again for all the help! — mateuscb, Jul 18 '14 at 20:33

initialization list of C++ containers of static const causes stack overflow

3 Answers3