waf check_python_headers fails with python installed with pyenv

Question

I'm trying to build a project using the waf build tool. The build fails in the configure step when checking the python headers.

I narrowed the problem down to this minimal example of my wscript:

def options(ctx):
        ctx.load('compiler_c')

def configure(ctx):
    ctx.load('compiler_c python')
    ctx.check_python_version((2,7,0))
    if int(ctx.env.PYTHON_VERSION[0]) == 2:
        print ('→ Configuring for python2')
    else:
        print ('→ Configuring for python3')
    ctx.check_python_headers(features='pyext')

I use python versions (3.6.12 & 3.7.9) that I installed with pyenv on Ubuntu 20.04.

I run that like this and it fails with an error that I cannot make sense of:

$ python waf configure
Setting top to                           : /home/myuser/waf-test
Setting out to                           : /home/myuser/waf-test/build
Checking for 'gcc' (C compiler)          : /usr/bin/gcc
Checking for program 'python'            : /home/myuser/.pyenv/versions/3.7.9/bin/python
Checking for python version >= 2.7.0     : 3.7.9
→ Configuring for python3
python-config                            : /home/myuser/.pyenv/versions/3.7.9/bin/python-config
Asking python-config for pyext '--cflags --libs --ldflags' flags : yes
Testing pyext configuration                                      : Could not build python extensions
The configuration failed
(complete log in /home/myuser/waf-test/build/config.log)

$ tail -n 20 build/config.log
[1/2] Compiling build/.conf_check_d3b505aa7ab58576b6d76a2fc3091b1f/test.c
['/usr/bin/gcc', '-fPIC', '-g', '-fwrapv', '-O3', '-I../../../../.pyenv/versions/3.7.9/include/python3.7m', '-DPYTHONDIR="/usr/local/lib/python3.7/site-packages"', '-DPYTHONARCHDIR="/usr/local/lib/python3.7/site-packages"', '-DNDEBUG', '../test.c', '-c', '-o/home/myuser/waf-test/build/.conf_check_d3b505aa7ab58576b6d76a2fc3091b1f/testbuild/test.c.1.o']
[2/2] Linking build/.conf_check_d3b505aa7ab58576b6d76a2fc3091b1f/testbuild/testprog.cpython-37m-x86_64-linux-gnu.so
['/usr/bin/gcc', '-shared', 'test.c.1.o', '-o/home/myuser/waf-test/build/.conf_check_d3b505aa7ab58576b6d76a2fc3091b1f/testbuild/testprog.cpython-37m-x86_64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/home/myuser/.pyenv/versions/3.7.9/lib/python3.7/config-3.7m-x86_64-linux-gnu', '-L/home/myuser/.pyenv/versions/3.7.9/lib', '-lpython3.7m', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lpython3.7m', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm']
err: /usr/bin/ld: /home/myuser/.pyenv/versions/3.7.9/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(pylifecycle.o): relocation R_X86_64_PC32 against symbol `Py_VerboseFlag' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
from /home/myuser/waf-test: Test does not build: Traceback (most recent call last):
  File "/home/myuser/waf-test/.waf3-2.0.20-36f5354d605298f6a89c09e0c7ef6c1d/waflib/Configure.py", line 335, in run_build
    bld.compile()
  File "/home/myuser/waf-test/.waf3-2.0.20-36f5354d605298f6a89c09e0c7ef6c1d/waflib/Build.py", line 176, in compile
    raise Errors.BuildError(self.producer.error)
waflib.Errors.BuildError: Build failed
 -> task in 'testprog' failed with exit status 1 (run with -v to display more information)
Could not build python extensions
from /home/myuser/waf-test: The configuration failed

Note that this problem does not appear with python 3.8.6 that was also installed with pyenv. I can configure without errors.

My questions: How can it be that the python versions in pyenv differ in a way that waf checks fail in one version and not in another? Since I simply install the versions with pyenv install 3.6.12, what do I need to change in the install so my build succeeds?

I am not sure, that this is a problem coming from waf - the error message come from the library ``can not be used when making a shared object; recompile with -fPIC``. Have you tried compiling in command line based on waf -v with an actived pyenv? — user69453, Nov 06 '20 at 17:52
@user69453 thanks for your comment. I'm not completely sure if I understand what you mean. I can see that the problem is in the linking step, but I don't understand what makes it fail. The problem I describe in my question is completely reproducable in an activated pyenv with the code that I show. `waf -v` does not give any more information than I already provide. — b3000, Nov 09 '20 at 18:01

neuro · Answer 1 · 2020-11-13T08:26:32.040

Well, the error is that, when creating the test program to check if pyext is correctly set, the compiler complains about the fact that pylifecycle.o included in libpython3.7m.a is non relocatable ie non compiled with the -fPIC flag.

As the code fragment generated by waf is compiled with -fPIC, it seems that the problem is that pylifecycle.o in libpython3.7m.a is not compiled with -fPIC, which makes sense as it is in a static library. Maybe the problem is that it uses the .a archive lib instead of the shared object one ie the .so.

Check if you have the shared object version of libpython ie libpython3.7m.so.

waf check_python_headers fails with python installed with pyenv

1 Answers1