0

Background

I have read countless GitHub project threads and everything I can find on StackOverflow about this problem, though so far no luck. I have a Mac 10.14 box running with the stock CommandLineTools and/or Xcode. I'm trying to "port" a Python wrapper library I have written around an older C and C++ library using CTypes in Python3. It works well already on Ubuntu Linux. However, there is no end to the problems I have been coming across since moving to a Mac platform. There just doesn't seem to be an easy answer to fixing what I'm trying to do on the broken Mac OS platform right now -- at least for the uninitiated Linux person like myself.

I have one question right off the bat before I describe how I'm compiling the dylib I'm trying to load up with CTypes: Do I now need to sign my dylib somehow before I can use it on Mac 10.14? Otherwise, I guess my question boils down to how the $%^@ (and that is truncated speak for what I do mean right now) can I deal with shared / dynamic libraries on Mac with a Python C extension interface?

My preference is to not even touch Xcode and just use the stock Mac tools that come with the system out of the box. My solution must work on the command line without defering to some auto configuration magic that Xcode will give you in GUI form. Really, this is all fairly painless for what it is under Linux.

Compilation and linking

The scenario is actually more complicated than I can describe. I will just sketch what seems to me to be the relevant parts of the solution, and then let you all experts who have gotten this working before tell me the obvious missteps in between.

I'm compiling the older C/C++ source code library as a static archive first using the following gcc (read clang) options on Mac (some of them get ignored):

-O0 -march=native -force_cpusubtype_ALL -fPIC -I../include -fPIC -m64 \
-fvisibility=default -Wno-error -lc++ -lc++abi

Then I'm compiling and linking with a combination of 

-Wl,-all_load $(LIBGOMPSTATIC).a $(LIBGMPSTATIC) -Wl,-noall_load \
-ldl -lpthread -lc++ -lc++abi

and 

 -dynamiclib -install_name $(MODULENAME) \
 -current_version 1.0.0 -compatibility_version 1.0

to generate the dylib output.

For comparison, on Linux, the analogs to these flags that work are approximately

-Wl,-export-dynamic -Wl,--no-undefined -shared -fPIC \
                        -fvisibility=default -Wl,-Bsymbolic
-Wl,-Bstatic -Wl,--whole-archive $(LIBGOMPSTATIC).a $(LIBGMPSTATIC) -Wl,--no-whole-archive \
                       -Wl,-Bdynamic -Wl,--no-as-needed -ldl -lpthread

and

-Wl,-soname,$(MODULENAME)

The dylib output

The above procedure gives me a dylib file that I can scan with nm to see the symbols I am trying to import with CTypes. This is a good start. When I try to run the test python script to test my CTypes wrapper library, I get a SEGFAULT immediately. Since gdb is apparently useless on Mac these days (sorry), I used the stock llvm to load up a brew-installed python3 with extra debugging symbols:

lldb /usr/local/Cellar/python-dbg\@3.7/3.7.6_13/bin/python3
(lldb) run myscript.py
Process 75435 launched: '/usr/local/Cellar/python-dbg@3.7/3.7.6_13/bin/python3' (x86_64)
Process 75435 stopped
* thread #2, stop reason = exec
    frame #0: 0x0000000100005000 dyld`_dyld_start
dyld`_dyld_start:
->  0x100005000 <+0>: popq   %rdi
    0x100005001 <+1>: pushq  $0x0
    0x100005003 <+3>: movq   %rsp, %rbp
    0x100005006 <+6>: andq   $-0x10, %rsp
Target 0: (Python) stopped.
(lldb) bt
* thread #2, stop reason = exec
  * frame #0: 0x0000000100005000 dyld`_dyld_start
(lldb) c

... redacted path information ...

File "/usr/local/Cellar/python-dbg@3.7/3.7.6_13/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 442, in LoadLibrary
    return self._dlltype(name)
  File "/usr/local/Cellar/python-dbg@3.7/3.7.6_13/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(GTFoldPython.dylib, 6): image not found
Process 75435 exited with status = 1 (0x00000001)

I do have each of the environment variables PYTHONAPPSDIR=/usr/local/Cellar/python-dbg@3.7/3.7.6_13, PYTHONPATH, LD_LIBRARY_PATH, DYLD_LIBRARY_PATH set to correct paths.

So the question is what do I try next to get this working? Many, many hours of thanks in advance!

mds
  • 129
  • 4
  • The function ``ctypes.PyDLL("mylib.so")`` works for me on Linux, though not on Mac. What at least sometimes gets it to start is calling ``ctypes.LoadLibrary("dylib.dylib")``. I can post more of my Python wrapper code in CTypes if that will help. – mds Feb 08 '20 at 06:16
  • What is `GTFoldPython.dylib`? It can't seem to find that dylib. – l'L'l Feb 08 '20 at 08:00
  • The dylib ``GTFoldPython.dylib`` is the dynamic library I created (called it ``mylib.dylib`` in the comment above). It is in the path of the env vars I have in the post. – mds Feb 08 '20 at 08:31
  • Do you know if I need to sign the dylib to open it and use it on Mojave++? I had to go through this process of creating a certificate for software signing to use ``gdb`` (presumably ``lldb`` is auto configured like this on install). That's one possibility that I'm not clear on. Another is the weird path variables, but this works just fine on Linux. – mds Feb 08 '20 at 08:33
  • Again, it looks to me like the dylib is getting compiled and linked correctly. I can see all the symbols I want in it with ``nm``. It's just that loading it with the standard CTypes construction in Python3 is so painful on Mac! – mds Feb 08 '20 at 08:37
  • It's probably one of two things, either your app can't access the location of where that `dylib` is located because of `SIP` (system integrity protection), or it's referenced in a relative location which it's unable to resolve. You could try disabling `SIP` (not recommended) to see if it works to determine if that's it, or use an absolute path to the dylib (not a great solution), or relink the dylib using `install_name_tool`. It's difficult to know what's happening without seeing more about the way it's actually linked. If you can post the output of `otool -l ` that would be helpful. – l'L'l Feb 08 '20 at 08:50
  • Trying an absolute path like this might work`ctypes.CDLL('GTFoldPython.dylib') => ctypes.CDLL('/path/to/GTFoldPython.dylib')`. Where is `GTFoldPython.dylib` residing? – l'L'l Feb 08 '20 at 08:54
  • The absolute paths to it (which is in a local subdirectory) are included in the env variables. I adjusted the env's with ``greadlink -f dir``. I just found this idea that suggests a ``DYLD_FALLBACK_LIBRARY_PATH``: https://stackoverflow.com/questions/3146274/is-it-ok-to-use-dyld-library-path-on-mac-os-x-and-whats-the-dynamic-library-s – mds Feb 08 '20 at 09:05
  • And the fallback path doesn't work... – mds Feb 08 '20 at 09:07
  • Yeah, I didn't think that would if the other `env` variables aren't working. Try the other suggestions, and check the path with `otool -l` as mentioned. – l'L'l Feb 08 '20 at 09:08
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/207450/discussion-between-mds-and-lll). – mds Feb 08 '20 at 09:09
  • Take a good look at [\[SO\]: How do I ask a good question?](https://stackoverflow.com/help/how-to-ask) or [\[SO\]: How to create a Minimal, Reproducible Example (reprex (mcve))](https://stackoverflow.com/help/mcve) then edit the question accordingly. Stop posting info in comments, edit the question instead.. Add the code you're using and the compile flags (not parts of them) and relevant info on the *.dylib* (e.g. `otool -L`). – CristiFati Feb 08 '20 at 09:27

1 Answers1

1

In my case, the issue turned out to be two things.

The first is that I was running my python script with a different version of python than the C extensions were linked with. For example, the following is the output of my python3-config --ldflags command:

-L/usr/local/Cellar/python-dbg@3.7/3.7.6_13/Frameworks/Python.framework/Versions/3.7/lib/python3.7/config-3.7dm-darwin -lpython3.7dm -ldl -framework CoreFoundation

So running it with /usr/local/Cellar/python-dbg@3.7/3.7.6_13/bin/python3 caused errors for me. This can be resolved by running the script with /usr/local/Cellar/python-dbg@3.7/3.7.6_13/bin/python3.7dm. Not an obvious fix given that brew installs each with an only slightly modified tap formula.

Second, in my C code, I am frequently writing to an extern'ed character buffer that lives on the stack. When this happens, the default clang stack protection mechanisms throw a SIGABRT at the script. To avoid this, you can recompile by passing the following flags to both the compiler and linker (might be more disabling than is actually needed):

-fno-stack-check -fno-stack-protector -no-pie -D_FORTIFY_SOURCE=0

With these two fixes in place my script runs. And still crashes for other reasons related to multithreading with Python in C. However, this is to be expected, but still has yet to show up in my testing on Linux.

Thanks to @l'L'l for helping me to work through this.

mds
  • 129
  • 4