Can't import cpplist into Cython?

Question

I'm trying to figure out how to work lists/arrays in Cython, which seems impossibly complex, so I would prefer to just use C++ lists as I saw some people use on SO. However when I've ran their code, I'm getting a gcc+ compile error in ipynb. Cython data structures are infuriating.

When ran alone in a cell I get this error, I've tried importing with and without the %%cython magic call and both error...

'''

%%cython

from libcpp.list cimport list as cpplist 

'''
def main(int t):

cdef cpplist[int] temp

for x in range(t):
    if x> 0:
        temp.push_back(x)

cdef int N = temp.size()
cdef list OutputList = N*[0]

for i in range(N):
    OutputList[i] = temp.front()
    temp.pop_front()

return OutputList
'''

'''

 ---------------------------------------------------------------------------
DistutilsExecError                        Traceback (most recent call last)
/anaconda3/lib/python3.6/distutils/unixccompiler.py in _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts)
    117             self.spawn(compiler_so + cc_args + [src, '-o', obj] +
--> 118                        extra_postargs)
    119         except DistutilsExecError as msg:

/anaconda3/lib/python3.6/distutils/ccompiler.py in spawn(self, cmd)
    908     def spawn(self, cmd):
--> 909         spawn(cmd, dry_run=self.dry_run)
    910 

/anaconda3/lib/python3.6/distutils/spawn.py in spawn(cmd, search_path, verbose, dry_run)
     35     if os.name == 'posix':
---> 36         _spawn_posix(cmd, search_path, dry_run=dry_run)
     37     elif os.name == 'nt':

/anaconda3/lib/python3.6/distutils/spawn.py in _spawn_posix(cmd, search_path, verbose, dry_run)
    158                           "command %r failed with exit status %d"
--> 159                           % (cmd, exit_status))
    160             elif os.WIFSTOPPED(status):

DistutilsExecError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

CompileError                              Traceback (most recent call last)
<ipython-input-6-70891eecfa66> in <module>()
----> 1 get_ipython().run_cell_magic('cython', '--cplus', '\n# distutils: language = c++\nfor i in range(10):\n    print(i)\n    \n\n\n#from libc.math cimport log\nfrom libcpp.list cimport list as cpplist')

/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2165             magic_arg_s = self.var_expand(line, stack_depth)
   2166             with self.builtin_trap:
-> 2167                 result = fn(magic_arg_s, cell)
   2168             return result
   2169 

<decorator-gen-127> in cython(self, line, cell)

/anaconda3/lib/python3.6/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188 
    189         if callable(arg):

/anaconda3/lib/python3.6/site-packages/Cython/Build/IpythonMagic.py in cython(self, line, cell)
    327 
    328         self._build_extension(extension, lib_dir, pgo_step_name='use' if args.pgo else None,
--> 329                               quiet=args.quiet)
    330 
    331         module = imp.load_dynamic(module_name, module_path)

/anaconda3/lib/python3.6/site-packages/Cython/Build/IpythonMagic.py in _build_extension(self, extension, lib_dir, temp_dir, pgo_step_name, quiet)
    437             if not quiet:
    438                 old_threshold = distutils.log.set_threshold(distutils.log.DEBUG)
--> 439             build_extension.run()
    440         finally:
    441             if not quiet and old_threshold is not None:

/anaconda3/lib/python3.6/distutils/command/build_ext.py in run(self)
    337 
    338         # Now actually compile and link everything.
--> 339         self.build_extensions()
    340 
    341     def check_extensions_list(self, extensions):

/anaconda3/lib/python3.6/distutils/command/build_ext.py in build_extensions(self)
    446             self._build_extensions_parallel()
    447         else:
--> 448             self._build_extensions_serial()
    449 
    450     def _build_extensions_parallel(self):

/anaconda3/lib/python3.6/distutils/command/build_ext.py in _build_extensions_serial(self)
    471         for ext in self.extensions:
    472             with self._filter_build_errors(ext):
--> 473                 self.build_extension(ext)
    474 
    475     @contextlib.contextmanager

/anaconda3/lib/python3.6/distutils/command/build_ext.py in build_extension(self, ext)
    531                                          debug=self.debug,
    532                                          extra_postargs=extra_args,
--> 533                                          depends=ext.depends)
    534 
    535         # XXX outdated variable, kept here in case third-part code

/anaconda3/lib/python3.6/distutils/ccompiler.py in compile(self, sources, output_dir, macros, include_dirs, debug, extra_preargs, extra_postargs, depends)
    572             except KeyError:
    573                 continue
--> 574             self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
    575 
    576         # Return *all* object filenames, not just the ones we just built.

/anaconda3/lib/python3.6/distutils/unixccompiler.py in _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts)
    118                        extra_postargs)
    119         except DistutilsExecError as msg:
--> 120             raise CompileError(msg)
    121 
    122     def create_static_lib(self, objects, output_libname,

CompileError: command 'gcc' failed with exit status 1

When ran alone in a cell I get this error, I've tried importing with and without the %%cython magic call and both error...

in Cython, I get the object of type 'None' has no length (the ONLY error message in Cython language)

or

invalid syntax

Please advise, Cython has me ready to rip my hair out 2 days in.

EDITS: I've tried using:

%%cython --cplus
#distutils: language = c++

same error message. Also, JUST RUNNING '%%cython --cplus' GIVES ME AN ERROR MESSAGE, SAME ONE? With anything in the cell, a simple print or anything. Something is wrong with my cpp extension I think... how do I resolve?

In terminal (using runipy -- don't know how else to run ipynb in terminal aside from compiling via a setup.py and distutils Build)

zacharys-mbp:Cython zoakes$ runipy CSTL.ipynb
08/08/2019 08:47:47 PM INFO: Reading notebook CSTL.ipynb
08/08/2019 08:47:49 PM INFO: Running cell:
%load_ext cython 


08/08/2019 08:47:49 PM INFO: Cell returned
08/08/2019 08:47:49 PM INFO: Running cell:
%%cython 

#distutils: language = c++
from libcpp.list cimport list as cpplist

    warning: include path for stdlibc++ headers not found; pass '-    stdlib=libc++' on
      the command line to use the libc++ standard library instead
      [-Wstdlibcxx-not-found]
/Users/zoakes/.ipython/cython/_cython_magic_5a0764b273da2aafc5775e4dd20b1249.cpp:592:10: fatal error: 
      'ios' file not found
#include "ios"
         ^~~~~
1 warning and 1 error generated.
08/08/2019 08:47:50 PM INFO: Cell raised uncaught exception: 
---------------------------------------------------------------------------
DistutilsExecError                        Traceback (most recent call last)
/anaconda3/lib/python3.6/distutils/unixccompiler.py in _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts)
    117             self.spawn(compiler_so + cc_args + [src, '-o', obj] +
--> 118                        extra_postargs)
    119         except DistutilsExecError as msg:

/anaconda3/lib/python3.6/distutils/ccompiler.py in spawn(self, cmd)
    908     def spawn(self, cmd):
--> 909         spawn(cmd, dry_run=self.dry_run)
    910 

/anaconda3/lib/python3.6/distutils/spawn.py in spawn(cmd, search_path, verbose, dry_run)
     35     if os.name == 'posix':
---> 36         _spawn_posix(cmd, search_path, dry_run=dry_run)
     37     elif os.name == 'nt':

/anaconda3/lib/python3.6/distutils/spawn.py in _spawn_posix(cmd, search_path, verbose, dry_run)
    158                           "command %r failed with exit status %d"
--> 159                           % (cmd, exit_status))
    160             elif os.WIFSTOPPED(status):

DistutilsExecError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

CompileError                              Traceback (most recent call last)
<ipython-input-2-e4f283bb7389> in <module>()
----> 1 get_ipython().run_cell_magic('cython', '', '\n#distutils: language = c++\nfrom libcpp.list cimport list as cpplist')

/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2165             magic_arg_s = self.var_expand(line, stack_depth)
   2166             with self.builtin_trap:
-> 2167                 result = fn(magic_arg_s, cell)
   2168             return result
   2169 

<decorator-gen-127> in cython(self, line, cell)

/anaconda3/lib/python3.6/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188 
    189         if callable(arg):

/anaconda3/lib/python3.6/site-packages/Cython/Build/IpythonMagic.py in cython(self, line, cell)
    327 
    328         self._build_extension(extension, lib_dir, pgo_step_name='use' if args.pgo else None,
--> 329                               quiet=args.quiet)
    330 
    331         module = imp.load_dynamic(module_name, module_path)

/anaconda3/lib/python3.6/site-packages/Cython/Build/IpythonMagic.py in _build_extension(self, extension, lib_dir, temp_dir, pgo_step_name, quiet)
    437             if not quiet:
    438                 old_threshold = distutils.log.set_threshold(distutils.log.DEBUG)
--> 439             build_extension.run()
    440         finally:
    441             if not quiet and old_threshold is not None:

/anaconda3/lib/python3.6/distutils/command/build_ext.py in run(self)
    337 
    338         # Now actually compile and link everything.
--> 339         self.build_extensions()
    340 
    341     def check_extensions_list(self, extensions):

/anaconda3/lib/python3.6/distutils/command/build_ext.py in build_extensions(self)
    446             self._build_extensions_parallel()
    447         else:
--> 448             self._build_extensions_serial()
    449 
    450     def _build_extensions_parallel(self):

/anaconda3/lib/python3.6/distutils/command/build_ext.py in _build_extensions_serial(self)
    471         for ext in self.extensions:
    472             with self._filter_build_errors(ext):
--> 473                 self.build_extension(ext)
    474 
    475     @contextlib.contextmanager

/anaconda3/lib/python3.6/distutils/command/build_ext.py in build_extension(self, ext)
    531                                          debug=self.debug,
    532                                          extra_postargs=extra_args,
--> 533                                          depends=ext.depends)
    534 
    535         # XXX outdated variable, kept here in case third-part code

/anaconda3/lib/python3.6/distutils/ccompiler.py in compile(self, sources, output_dir, macros, include_dirs, debug, extra_preargs, extra_postargs, depends)
    572             except KeyError:
    573                 continue
--> 574             self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
    575 
    576         # Return *all* object filenames, not just the ones we just built.

/anaconda3/lib/python3.6/distutils/unixccompiler.py in _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts)
    118                        extra_postargs)
    119         except DistutilsExecError as msg:
--> 120             raise CompileError(msg)
    121 
    122     def create_static_lib(self, objects, output_libname,

CompileError: command 'gcc' failed with exit status 1
08/08/2019 08:47:50 PM INFO: Shutdown kernel
08/08/2019 08:47:50 PM WARNING: Exiting with nonzero exit status

I don't know anything about cython really, but is it possible to see the compiler error? I tried running your code through Cython (via the command line) on my system, and I got an error about a missing include file. It seems like other people get the same error: https://github.com/cython/cython/issues/2694 — Daniel McLaury, Aug 06 '19 at 02:15
Did you try to run ‘%%cython?’ and read the output in order to find out how c++ can be activated for a cell? — ead, Aug 06 '19 at 02:58
Possible duplicate of [Wrapping C++ Standard library with Cython in ipython](https://stackoverflow.com/questions/34060663/wrapping-c-standard-library-with-cython-in-ipython) — DavidW, Aug 06 '19 at 06:45
For reference, the more useful gcc error message appears in the terminal that you ran jupyter from instead of the jupyter notebook, (which really isn't helpful). — DavidW, Aug 06 '19 at 06:46
%%cython magic is running fine, is it possible I need to specify as c++ rather than c? I didn't run in terminal, I'm using the Ipy magic method, no other formal compiler aside from the automated one associated with magic method. I'll include full error — Zach Oakes, Aug 07 '19 at 17:47
How did you start your ipython notebook? Didn't you execute `ipython notebook` or similar from a terminal? — ead, Aug 07 '19 at 18:40
If I copy your code, and either change `%%cython` to `%%cython --cplus` or add `#distutils: language = c++` at the start it compiles correctly for me. As we have suggested, start the notebook from a terminal and then look for the gcc error message in the terminal. — DavidW, Aug 08 '19 at 16:33
How do I start the notebook from terminal? I navigated to the ipynb and entered python3 filename.ipynb and nothing happened, still in the folder but no gcc error, no build message, nothing. I'm looking at documentation for ipynb and don't see anything for running ipynb in terminal. I can compile it with distutils and setup.py in terminal, but that's not the issue I'm having. — Zach Oakes, Aug 08 '19 at 19:28
Using runipy (don't know how else to use terminal for ipynb, aside from compiling with a setup.py and distutils) -- This is my error message: — Zach Oakes, Aug 09 '19 at 01:54

score 1 · Accepted Answer · answered Aug 09 '19 at 07:00

I think this is a Mac issue, which limits my ability to help. However, the key error message seems to be:

warning: include path for stdlibc++ headers not found; pass '-stdlib=libc++' on
      the command line to use the libc++ standard library instead
      [-Wstdlibcxx-not-found]

If you search for (part of) this message it looks like it's related to XCode. At some point Apple switched the compiler from GCC to Clang, and this changed which implementation of the C++ standard library it uses.

I think the best solution is to install "stdlibc++" on XCode. Unfortunately I have no idea how you'd practically do this.

The second best solution involves adding the suggested command line argument for Cython - I think this is second-best because it's using a slightly mismatched implementation of the C++ standard library.

%%cython --compile-args=-stdlib=libc++ --link-args=-stdlib=libc++

I'm not sure if it needs to be in both compile and link args or just compile args.

I was able to get it to work with: %%cython --compile-args=-stdlib=libc++ --link-args=-stdlib=libc++ --cplus #from libc.math cimport log from libcpp.list cimport list as cpplist THANK YOU! Do you think uninstall of Xcode/Reinstall would help? (I think maybe it has to do with Atom, as that uses a Clang version for running C++) — Zach Oakes, Aug 10 '19 at 17:36
To be honest I've never used a Mac so I'd be guessing about the XCode reinstall - it might help, but it might also cause more problems. Glad you got it to work though! — DavidW, Aug 10 '19 at 18:11

Can't import cpplist into Cython?

1 Answers1