How to tell which specific compiler will be invoked for a Python C extension: GCC or Clang?

Question

I have a Python C++ extension that requires the following compilation flags when compiled using Clang on OS X:

CPPFLAGS='-std=c++11 -stdlib=libc++ -mmacosx-version-min=10.8'
LDFLAGS='-lc++'

Detecting OS X in my setup.py is easy enough. I can do this:

if sys.prefix == 'darwin':
    compile_args.append(['-mmacosx-version-min=10.8', '-stdlib=libc++'])
    link_args.append('-lc++')

(See here for full context)

However, on GCC this compilation flag is invalid. So, compilation will fail if someone will try to use GCC on OS X if I write the setup.py this way.

GCC and Clang support different compiler flags. So, I need to know which compiler will be invoked, so I can send different flags. What is the right way to detect the compiler in the setup.py?

Edit 1:
Note that no Python exception is raised for compilation errors:

$ python setup.py build_ext --inplace
running build_ext
building 'spacy.strings' extension
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -c spacy/strings.cpp -o build/temp.linux-x86_64-2.7/spacy/strings.o -O3 -mmacosx-version-min=10.8 -stdlib=libc++
gcc: error: unrecognized command line option ‘-mmacosx-version-min=10.8’
gcc: error: unrecognized command line option ‘-stdlib=libc++’
error: command 'gcc' failed with exit status 1
$

Answer 1

I hit upon your question as I need the same kind of switch. Besides, in my case, sys.prefix is not great as the flags are for clang regardless of the platform.

I am not sure it is perfect but here is what works best for me. So, I check if a CC variable is set; if not, I check what I guess is where distutils looks at.

Any better solution welcome!

import os
import distutils

try:
    if os.environ['CC'] == "clang":
        clang = True
except KeyError:
    clang = False

if clang or distutils.sysconfig_get_config_vars()['CC'] == 'clang':
    try:
        _ = os.environ['CFLAGS']
    except KeyError:
        os.environ['CFLAGS'] = ""
    os.environ['CFLAGS'] += " -Wno-unused-function"
    os.environ['CFLAGS'] += " -Wno-int-conversion"
    os.environ['CFLAGS'] += " -Wno-incompatible-pointer-types

Note for the grumpy guys: I would have loved to use the extra_compile_args option, but it puts the flags at the wrong position in the clang compilation command.

Answer 2

Add the following code to your setup.py. It explicitly detects which flags are accepted by the compiler and then only those are added.

# check whether compiler supports a flag
def has_flag(compiler, flagname):
    import tempfile
    from distutils.errors import CompileError
    with tempfile.NamedTemporaryFile('w', suffix='.cpp') as f:
        f.write('int main (int argc, char **argv) { return 0; }')
        try:
            compiler.compile([f.name], extra_postargs=[flagname])
        except CompileError:
            return False
    return True


# filter flags, returns list of accepted flags
def flag_filter(compiler, *flags):
    result = []
    for flag in flags:
        if has_flag(compiler, flag):
            result.append(flag)
    return result


class BuildExt(build_ext):
    # these flags are not checked and always added
    compile_flags = {"msvc": ['/EHsc'], "unix": ["-std=c++11"]}

    def build_extensions(self):
        ct = self.compiler.compiler_type
        opts = self.compile_flags.get(ct, [])
        if ct == 'unix':
            # only add flags which pass the flag_filter
            opts += flag_filter(self.compiler,
                                '-fvisibility=hidden', '-stdlib=libc++', '-std=c++14')
        for ext in self.extensions:
            ext.extra_compile_args = opts
        build_ext.build_extensions(self)

setup(
   cmdclass=dict(build_ext=BuildExt),
   # other options...
)

The has_flag method was taken from this pybind11 example. https://github.com/pybind/python_example

Answer 3

Here is a cross-compiler and cross-platform solution:

from setuptools import setup
from setuptools.command.build_ext import build_ext


class build_ext_ex(build_ext):

    extra_args = {
        'extension_name': {
            'clang': (
                ['-std=c++11', '-stdlib=libc++', '-mmacosx-version-min=10.8'],
                ['-lc++']
            )
        }
    }

    def build_extensions(self):
        # only Unix compilers and their ports have `compiler_so`
        compiler_cmd = getattr(self.compiler, 'compiler_so', None)
        # account for absolute path and Windows version
        if compiler_cmd is not None and 'clang' in compiler_cmd[0]:
            self.cname = 'clang'
        else:
            self.cname = self.compiler.compiler_type

        build_ext.build_extensions(self)

    def build_extension(self, ext):
        extra_args = self.extra_args.get(ext.name)
        if extra_args is not None:
            extra_args = extra_args.get(self.cname)
            if extra_args is not None:
                ext.extra_compile_args = extra_args[0]
                ext.extra_link_args    = extra_args[1]

        build_ext.build_extension(self, ext)


setup(
    ...
    cmdclass = {'build_ext': build_ext_ex},
    ...
)

... and a list of supported compiler types (as returned by setup.py build_ext --help-compiler):

--compiler=bcpp     Borland C++ Compiler
--compiler=cygwin   Cygwin port of GNU C Compiler for Win32
--compiler=mingw32  Mingw32 port of GNU C Compiler for Win32
--compiler=msvc     Microsoft Visual C++
--compiler=unix     standard UNIX-style compiler

If you will face the same problem as @xoolive, you could override build_extensions(self) only and append the options to the end of self.compiler.compiler_so and self.compiler.linker_so respectively.