Is gcc compile time proportional to number of executions or lines of code?

Question

I am using gcc 4.8.5 to compile a c++98 code. My c++ code statically initializes unordered_map of unoredred_maps with ~20,000 total key-value pairs, and overloaded function which will take ~450 different types. This program will be executed on a continuous stream of data, and for every block of data, overloaded function will return an output.

The problem is, gcc takes too long to compile due to initializing ~20,000 key-value pairs.

The nested unordered_map has a structure of map< DATATYPE, map< key, value >>, and only one of the overloaded function gets called for each data input. In other words, I do not need to statically initialize the entire nested map, but I can instead dynamically define map<key, value> for the corresponding datatype when needed. For example, I can check for the definition of a map and when it is undefined, I can later populate it in run time. This will result a map with ~45 average key-value pairs.

However, I know that dynamic initialization will require longer code. For a simple execution described above (statically initializing entire map), will other method such as dynamic initialization significantly reduce time? My understanding is, whatever alternative I take, I still need to write a code to populate entire key-value pairs. Also, overhead and actual computation that goes behind populating an unordered_map (hashmap) should not differ asymptotically in most cases, and should not show significant difference than running same number of loops to increment a value.

For reference, I am writing a python script that reads in multiple json files to print out the c++ code, which then gets compiled using gcc. I am not reading json directly from c++ so whatever I do, c++ source will need to insert key-value one by one because it will not have access to json file.

// below is someEXE.cpp, which is a result from python script. 
// Every line is inside python's print"" (using python 2.7) 
// so that it can write complete c++ that should  compile.

someEXE.cpp

// example of an overloaded function among ~450
// takes in pointer to data and exampleMap created above
void exampleFunction(DIFFERENT_TYPE1*data, 
std::unorderd_map<std::string, std::unordered_map<std::string, std::string>> exampleMap) {
   printf("this is in specific format: %s", exampleMap["DATATYPE1"] 
   [std::to_string(data->member_variable)].c_str();
   //... more print functions below (~25 per datatype)
};

int main() {

   // current definition of the unordered_map (total ~20,000 pairs)
   std::unordered_map<std::string, std::unordered_map<std::string, 
   std::string>> exampleMap = {
       {"DATATYPE1", {{"KEY1", "VAL1"}, {"KEY2", "VAL2"}, /*...*/}}
   };

   // create below test function for all ~450 types
   // when I run the program, code will printf values to screen
   DIFFERENT_TYPE1 testObj = {0};
   DIFFERENT_TYPE1 *testObjPointer = &testObj;
   exampleFunction(testObjPointer, exampleMap);

   return 0;
}

EDIT: My initial question was "Is CMAKE compile time proportional to...". Changed the term "CMAKE" with actual compiler name, gcc 4.8.5 with the help from the comments.

Answer 1

With the further code you posted, and Jonathan Wakely's answer on the specific issue with your compiler, I can make a suggestion.

When writing my own codegen, if possible, I prefer generating plain old data and leaving logic and behaviour in non-generated code. This way you get a small(er) pure C++ code in data-driven style, and a separate block of dumb and easy-to-generate data in declarative style.

For example, directly code this

// GeneratedData.h
namespace GeneratedData {
  struct Element {
    const char *type;
    const char *key;
    const char *val;
  };

  Element const *rawElements();
  size_t rawElementCount();
}

and this

// main.cpp
#include "GeneratedData.h"

#include <string>
#include <unordered_map>

using Map = std::unordered_map<std::string, std::string>;
using TypeMap = std::unordered_map<std::string, Map>;

TypeMap buildMap(GeneratedData::Element const *el, size_t count)
{
  TypeMap map;
  for (; count; ++el, --count) {
    // build the whole thing here
  }
}
// rest of main can call buildMap once, and keep the big map.
// NB. don't pass it around by value!

and finally generate the big dumb file

// GeneratedData.cpp
#include "GeneratedData.h"

namespace {
  GeneratedData::Element const array[] = {
    // generated elements here
  };
}

namespace GeneratedData {
  Element const *rawElements { return array; }
  size_t rawElementCount() { return sizeof(array)/sizeof(array[0]); }
}

if you really want to, you can separate even that logic from your codegen by just #includeing it in the middle, but it's probably not necessary here.

---

Original answer

Is CMAKE

CMake.

... compile time

CMake configures a build system which then invokes your compiler. You haven't told us which build system it is configuring for you, but you could probably run it manually for the problematic object file(s), and see how much of the overhead is really CMake's.

... proportional to number of executions or lines of code?

No.

There is some overhead per-execution. Each executed compiler process has some overhead per line of code, but probably much more overhead per enabled optimization, and some optimizations may scale with cyclomatic complexity or other metrics.

statically initializes unordered_map of unoredred_maps with ~20,000 total key-value pairs

You should try to hide your giant initialization as much as possible - you haven't shown any code, but if it's only visible in one translation unit, only one object file will take a very long time to compile.

You could also probably use a codegen tool like gperf to build a perfect hash.

I can't give you a lot more detail without seeing at least a fragment of your actual code and some hint as to how your files and translation units are layed out.

Answer 2

Older versions of GCC take a very long time to compile large initializer-lists like this:

unordered_map<string, unordered_map<string, string>> exampleMap = {
    {"DATATYPE1", {{"KEY1", "VAL1"}, {"KEY2", "VAL2"}, /*...*/}}
};

The problem is that every new element in the initializer-list causes more code to be added to the block being compiled, and it gets bigger and bigger, needing to allocate more and more memory for the compiler's AST. Recent versions have been changed to process the initializer-list differently, although some problems still remain. Since you're using GCC 4.8.5 the recent improvements won't help you anyway.

However, I know that dynamic initialization will require longer code. For a simple execution described above (statically initializing entire map), will other method such as dynamic initialization significantly reduce time?

Splitting the large initializer-list into separate statements that insert elements one-by-one will definitely reduce the compile time when using older versions of GCC. Each statement can be compiled very quickly that way, instead of having to compile a single huge initialization that requires allocating more and more memory for each element.