11

We've been building a large open source software on a variety of platforms (Linux, Windows, Mac OS X, 32-bit and 64-bit) for several years without troubles. Lately however, the Mac OS X build (64-bit) stopped working correctly and started to crash randomly. It more or less coincided with an update of Mac OS X on our build machine from 10.7 to 10.8.2 (but the compiler toolchain didn't change, it's still llvm-gcc 4.2.1).

Our application is made of a couple of dynamic (shared) libraries and many executables using them. One of the shared library overrides the new and delete operators for a variety of reasons. On Mac OS X (and Linux), all symbols are exported by default, including our overloaded new and delete operators. The crashes on Mac OS X seem related to some memory being allocated with one memory subsystem (not ours) then freed through our own (and incompatible) delete implementation.

The sanest solution seems to be preventing the overloaded operators from being visible to the users of the shared library. This can be accomplished in two ways: marking the operators with __attribute__((visibility("hidden"))), or using the -unexported_symbols_list linker command line option to prevent some symbols from being exported. The first solution unfortunately doesn't work: gcc emits warnings saying that the operators have been declared differently (in <new>) and thus the attributes will be ignored. From my readings in various places, the second solution seems to be the right one to this problem. However for some reason we can't make it work.

When linking the shared library, we're passing the -Wl,-unexported_symbols_list unexported_symbols_list.txt option to g++, which in turns should be passed to ld. The unexported_symbols_list.txt file contains the following list of symbols:

__ZdaPv
__ZdaPvRKSt9nothrow_t
__ZdlPv
__ZdlPvRKSt9nothrow_t
__ZdlPvS_
__Znam
__ZnamRKSt9nothrow_t
__Znwm
__ZnwmPv
__ZnwmRKSt9nothrow_t

These are all the variations of new and delete that we override and want to be hidden. We found these symbols by doing nm libappleseed.dylib then unmangling the symbol names using c++filt.

Here's the command line generated by CMake to link libappeseed.dylib:

/usr/bin/g++  -g -Werror -dynamiclib -Wl,-headerpad_max_install_names -framework Cocoa -lcurl    -Werror -Wl,-unexported_symbols_list -Wl,unexported_symbols_list.txt -o ../mac-gcc4/appleseed/libappleseed.dylib [...]

Unfortunately, despite all our efforts it appears that the symbols remain (as nm shows).

Any idea what we are doing wrong? Is there another approach that we could try?

UPDATE Dec. 19, 2012:

Our problem and the supposed solution are well covered in this technical note from Apple: http://developer.apple.com/library/mac/#technotes/tn2185/_index.html (section "Overriding new/delete").

Pointers to relevant source code:

Fragment of nm's output after building libappleseed.dylib with -fvisibility=hidden and running strip -x libappleseed.dylib:

...
00000000002a41b0 T __ZdaPv
00000000002a41f0 T __ZdaPvRKSt9nothrow_t
00000000002a4190 T __ZdlPv
00000000002a41d0 T __ZdlPvRKSt9nothrow_t
00000000002a4060 T __Znam
00000000002a4130 T __ZnamRKSt9nothrow_t
00000000002a3ff0 T __Znwm
00000000002a40d0 T __ZnwmRKSt9nothrow_t
...
François Beaune
  • 4,270
  • 7
  • 41
  • 65
  • Can you drop the relevant lines from nm which shows that the symbols are still there? – alexp Dec 19 '12 at 17:47
  • Sure, the lines are in the update above. Not only are the offending symbols still there, but the application continues to crash randomly. – François Beaune Dec 19 '12 at 21:30
  • You've undoubtedly looked at this already, but I notice a comma separating `-unexported_symbols_list` from the name of the file in the g++ command line - that's not breaking it? – Dan Nissenbaum Dec 21 '12 at 13:49
  • @DanNissenbaum Each linker argument must be prefixed by `-Wl,`. As far as I can see there is no extra comma. Thanks for having a look though. – François Beaune Dec 24 '12 at 13:33

2 Answers2

7

You should be building with -fvisibility=hidden and then export only what you want. Have a read here:

http://gcc.gnu.org/wiki/Visibility

It also explains -fvisibility-inlines-hidden. Many large libraries (Qt, for example) make use of this. The benefits are quite substantial.

Nikos C.
  • 50,738
  • 9
  • 71
  • 96
  • This is something we'd like to do eventually. I was under the impression that gcc's visibility syntax was incompatible with Visual C++'s one (i.e. visibility attributes were used in different places in declarations) but it appears from this document that I was wrong (or the situation improved since last time I checked). Thanks for pointing this out, we'll probably go this route eventually. **That's a rather important change to apply and test though, so we'd still be interested in a quick interim solution**. – François Beaune Dec 10 '12 at 10:16
  • 1
    @FrançoisBeaune I suppose using `-unexported_symbol symbol` multiple times instead of using a file doesn't work either? (For example `-unexported_symbol __ZdaPv -unexported_symbol __ZdaPvRKSt9nothrow_t [...]`) – Nikos C. Dec 10 '12 at 10:29
  • So we tried using `-unexported_symbol symbol` (by passing e.g. `-Wl,-unexported_symbol -Wl,__ZdaPv` to g++) but to no avail. The arguments are properly passed down to ld, but nm shows that the symbols remain in the generated library file. Our conclusions so far are that: either the linker is not ld (g++ actually invokes `collect2` instead of `ld`), ld on OS X doesn't really support these flags, or we're making a stupid mistake (but we tried many different things all leading to the same conclusions). Any idea/hint would be very much appreciated. We'll offer a bounty as soon as we're allowed. – François Beaune Dec 10 '12 at 23:46
  • @FrançoisBeaune Also try: `-Xlinker -unexported_symbol -Xlinker __ZdaPv -Xlinker -unexported_symbol -Xlinker __ZdaPvRKSt9nothrow_t ...` ld on OS X does support the `-unexported_symbol` flag. Another thing to check is whether you need to strip the resulting library in order for those symbols to be removed. Also check the flags `strip` supports and whether you need some specific ones for stripping libraries. – Nikos C. Dec 11 '12 at 00:13
  • Thanks, we'll try that as well, although it appears to be a synonym to `-Wl,`: http://stackoverflow.com/questions/7221141/any-difference-between-wl-option-and-xlinker-option-syntax-for-gcc. Also we have already checked that the options are correctly passed to the linker by enabling verbosity on g++. We'll also investigate whether stripping makes a difference (I hope not). Thanks for the help and suggestions. – François Beaune Dec 11 '12 at 08:05
  • An update on this (yet unresolved) issue: we're now building with `-fvisibility=hidden` but the problem persists: according to `nm`, the `new` and `delete` operators are still visible. Stripping the shared library doesn't seem to have any effect. Bounty opened. – François Beaune Dec 19 '12 at 16:40
  • It looks like `strip -x libappleseed.dylib` had an effect (the symbol list went from 5 MB to 2 MB), however it appears that the new/delete operators are still there, as indicated by `nm`: for instance `nm`'s output shows `00000000002a41b0 T __ZdaPv` where the T stands for "text section symbol". The application still crashes. – François Beaune Dec 19 '12 at 17:37
  • We still didn't figure out the problem, but we eventually decided to no longer override the `new` and `delete` operators since memory blocks are natively 16-byte-aligned on Mac OS X and Linux anyway (our primary reason to provide our own allocation functions). @NikosC I'm attributing you the bounty for your help and many suggestions, thanks again. – François Beaune Dec 24 '12 at 13:36
  • @FrançoisBeaune Thanks. But I'm sorry to hear the problem is still unsolved. I suppose some form of paid support from Apple (if they even offer that) is out of the question? – Nikos C. Dec 25 '12 at 05:15
  • @FrançoisBeaune I just found a detail in the docs that I didn't notice before: "Note that due to ISO C++ specification requirements, operator new and operator delete must always be of default visibility." Source: http://gcc.gnu.org/onlinedocs/gcc-4.7.2/gcc/Code-Gen-Options.html – Nikos C. Dec 25 '12 at 05:36
0

You may take a look at symbol maps/versioning (--version-script ld option)

http://accu.org/index.php/journals/1372

alexp
  • 811
  • 4
  • 10
  • Thanks. That aspect of things should already be taken care of with gcc's `__attribute__((visibility("default")))` attributes. – François Beaune Dec 19 '12 at 17:29
  • It seems --version-script is (still) not supported clang/llvm to this date. – codesniffer Mar 13 '20 at 21:09
  • @codesniffer can you please confirm that --version-script is not suppoted in xcode clang cpp build, tnx – Boki Mar 20 '20 at 19:19
  • @Boki I get build errors for --version-script on 10.10, 10.12, and 10.15, and I don't see it in the man page. It shouldn't matter, but I'm using only XCode CLT. ld version on my 10.15 system is LLVM version 11.0.0, (clang-1100.0.33.17). Is it working for you? – codesniffer Mar 21 '20 at 03:11
  • @codesnifer Ok, thanks, I am just about planning to refactor some code for build with XCode 11.3 on 10.15, cause I experienced issues/crashes with dlclose when executing app on HighSierra or Mohave(no issues on Catalina). So, for beginning, my idea was to reduce/control exports using version scripts. I used to think about, when I saw your comment here, but, I think I noticed earlier, some options in BuildSettings for linker map scripts or similar, I am not sure, I will recheck during weekend anyway. If I find something useful I will keep you informed. – Boki Mar 21 '20 at 20:38