Why nlohmann does not release memory

Question

I am using nlohmann library to parse JSON codes. We have two problems:
1- Why nlohmann use huge memory to parse data
2- After call parser in a function locally like below code it does not release memory. My JSON data size is about 8MB and the parser use more then 50MB for parsing. I parsed this JSON data 10 times and memory usage goes up to 600MB and after the function is finished memory did not released.

#include "nlohmann/json.hpp"

    #define REPEAT 10

        void GenerateNlohmann() {
          std::string filePath{FILE_ADDRESS};
          std::ifstream iFile(filePath.c_str(), std::ios::in);
          std::string data{};
          if (iFile.is_open()) {
            data = std::string((std::istreambuf_iterator<char>(iFile)),
                               std::istreambuf_iterator<char>()); // About 8MB size
            iFile.close();
          }
          if (!data.empty()) {
            nlohmann::json json = nlohmann::json::parse(data); // Use memory about 50MB
            std::vector<nlohmann::json> jsons{};
            for (int i = 0; i < REPEAT; ++i) {
              nlohmann::json j = nlohmann::json::parse(data);
              jsons.emplace_back(j);
            }
            while (!jsons.empty()) {
              jsons.pop_back();
            }
          }
        }

            int main() {
              GenerateNlohmann();

        // Now memory usage is about 600MB
              std::cout << "Input a numberto exit" << std::endl;
              int i;
              std::cin >> i;

              return 0;
            }

I checked it in system monitoring. I wrote a comment in main function. I am using Ubuntu. — mohammad-nazari, Jan 28 '20 at 14:20
I think you can't really measure something by looking at it. Use some profiling tools — RoQuOTriX, Jan 28 '20 at 14:22
We use it in a big project and after some days memory goes up.I am developing codes in clion and used Valgrind and also Profiler. They did not show any memory leak in this library. but we check other sections unit by unit. — mohammad-nazari, Jan 28 '20 at 14:26
Which version are you using? I use 3.5 and 3.7 and both release memory properly. The `parse` method is really bad for memory usage, but it does release in the end. — ChrisMM, Jan 28 '20 at 14:44
*I parsed this JSON data 10 times and memory usage goes up to 600MB and after the function is finished memory did not released.* -- Do not use system monitoring. Just call this function (or functions) many times. If the memory does not increase above 600MB, then it is the allocator that is holding onto the memory, and not a fault of nlohman library. If the memory went up 600MB, then 1200MB, then 1800MB, etc. on each iteration, *then* you have an issue the library itself. — PaulMcKenzie, Jan 28 '20 at 14:55
PaulMcKenzie, your correct Paul. I called function many times in a loop and memory did not goes up more then 600MB. Maybe it is a memory paging policy in linux. I did this test in windows and memory was released immediately. But Why? I test repidjson library that is OK. — mohammad-nazari, Feb 01 '20 at 07:12

score 0 · Accepted Answer · answered Feb 10 '20 at 07:27

I had reproduce successful in ubuntu. My json data size about 1.4 KB but I parsed this json data for many times. Here are my test result:

Before Run:

KiB Mem :  8167476 total,  5461204 free,   284120 used,  2422152 buff/cache

1000 times:
KiB Mem :  8167476 total,  5456600 free,   288724 used,  2422152 buff/cache

10000 times:
KiB Mem :  8167476 total,  5405916 free,   339376 used,  2422184 buff/cache

100000 times:
KiB Mem :  8167476 total,  4893176 free,   852104 used,  2422196 buff/cache

After input the int (After run)
KiB Mem :  8167476 total,  5462208 free,   283116 used,  2422152 buff/cache

There is indeed a problem, but this is an optimization by the allocator (probably glibc in my case) and unrelated to the library.

If I add malloc_trim(0) in my code:

while (!jsons.empty()) {
      jsons.pop_back();
    }

+   malloc_trim(0);
  }

I will find everythink will be OK. In windows we can not reproduce because what we use is not glibc, I think.

Other Test: I had wrote other program to malloc many small memory with glibc, and the problem will still alive.My program is unrelated to the library, it just malloc and free many small memory.

Anyway, the probleam is unrelated to the library. If we add malloc_trim(0) into the library, there will be many calls during the parsering, which will reduce performance.So the better solution is add malloc_trim(0) in your code.

Why nlohmann does not release memory

1 Answers1