10

I'm trying to create an application which reloads a shared library multiple times. But at some point in time, dlmopen fails with error

/usr/lib/libc.so.6: cannot allocate memory in static TLS block

Here is the minimal code reproducing this issue:

#include <dlfcn.h>
#include <cstdio>
#include <vector>

int main() {
  for (int i = 0; i < 100; ++i) {
    void *lib_so = dlmopen(LM_ID_NEWLM, "lib.so", RTLD_LAZY | RTLD_LOCAL);
    if (lib_so == NULL) {
      printf("Iteration %i loading failed: %s\n", i, dlerror());
      return 1;
    }
    dlclose(lib_so);
  }

  return 0;
}

And empty lib.cpp, compiled with 

g++ -rdynamic -ldl -Wl,-R . -o test main.cpp
g++ -fPIC -shared lib.cpp -o lib.so

Update

It seems that it crashes even with one thread. The question is: how can I force a library unload or a destruction of unused namespaces created with LM_ID_NEWLM?

TriskalJM
  • 2,393
  • 1
  • 19
  • 20
Ilya Lukyanov
  • 103
  • 2
  • 7
  • Possible answers may be found here: http://stackoverflow.com/questions/14892101/cannot-load-any-more-object-with-static-tls. I suspect your lib.so is allocating TLS data, and since you are loading it 10x per thread, may be exhausting the storage for a thread. – Nicholas Smith Jun 09 '16 at 00:10
  • TLS = [Thread Local Storage](https://en.wikipedia.org/wiki/Thread-local_storage), & it's possible there's too much in ThreadLocal object. [A link](http://stackoverflow.com/questions/6021273/how-to-allocate-thread-local-storage) describing allocating TLS. `test_thread` shouldn't need thread_id passed as argument. [thread_id](http://en.cppreference.com/w/cpp/thread/thread/id)'s is not type `int`, & each `id` is accessible by `thread.get_id`. Shorter way to start threads: `for(int i=0; i<10;i++) threads.push_back(thread(test_thread));` & joining them... `for(auto& thread: threads) thread.join();` – NonCreature0714 Jun 09 '16 at 00:42
  • 1
    @qexyn lib.so in this example compiled from empty file, so only `dlmopen` allocating TLS data. I've seen that question before, but since lib.so compiled with `-fPIC` it already using `-ftls-model=global-dynamic`. Also, looks like `dlclose` does not actually deallocates memory and next `dlmopen` creates new object that leads to leak. – Ilya Lukyanov Jun 09 '16 at 06:16
  • Could you obtain the same by using `dlopen` and adding the attributes `RTLD_NOW | RTLD_GROUP | RTLD_LOCAL`. – Jens Munk Aug 10 '16 at 20:24
  • @JensMunk, it's Linux, so `dlopen` don't have `RTLD_GROUP` flag. I can't use `dlopen` because I need to load every object in new namespace. – Ilya Lukyanov Aug 17 '16 at 11:16
  • 1
    You can't have many namespaces - it mentions `The glibc implementation supports a maximum of 16 namespaces` in the man page on my system, and there's no facility for cleaning up the link-map namespace. You're better off recycling the same namespace for the library by unloading and re-loading the library only, rather than trying to keep on creating new namespaces (but this is all speculation). You can use `dlinfo` to get the link map id to keep using it for subsequent dlmopen calls for the same library; you just need to keep closing before reopening it. – Anya Shenanigans Jan 10 '17 at 14:05
  • @Petesh Oh, I see. I suppose this should be accepted answer. – Ilya Lukyanov Jan 11 '17 at 07:03

2 Answers2

3

There is a built-in limit to the number of link map namespaces available to a process. This is rather poorly documented in the comment:

The glibc implementation supports a maximum of 16 namespaces

in the man page.

Once you create a link map namespace, there is no support for 'erasing' it via any APIs. This is just the way it's designed, and there's no real way to get around that without editing the glibc source and adding some hooks.

Using namespaces for reloading of a library is not actually reloading the library - you're simply loading a new copy of the library. This is one of the use cases of the namespaces - if you tried to dlopen the same library multiple times, you would get the same handle to the same library; however if you load the second instance in a different namespace, you won't get the same handle. If you want to accomplish reloading, you need to unload the library using dlclose, which will unload the library once the last remaining reference to the library has been released.

If you want to attempt to 'force unload' a library, then you could try issuing multiple dlclose calls until it unloads; however if you don't know what the library has done (e.g. spawned threads) there may be no way of preventing a crash in that case.

Anya Shenanigans
  • 91,618
  • 3
  • 107
  • 122
0

Older glibc versions might have some bugs related to this:

https://bugzilla.redhat.com/show_bug.cgi?id=89692 https://sourceware.org/bugzilla/show_bug.cgi?id=14898

What version are you using? Try using a newer glibc version, your code works pretty fine on my computer (glibc 2.23).

Michael Spector
  • 36,723
  • 6
  • 60
  • 88