Am I doing cythonization right?

Question

I am trying to get cython to work on a project.

What I do is the following:

Create a file called make_cython.py

import distutils.core
import Cython.Build
distutils.core.setup(ext_modules = Cython.Build.cythonize("main.py"))
distutils.core.setup(ext_modules = Cython.Build.cythonize("helper.py"))

Run python3 make_cython.py build_ext --inplace

Run for both cythnoized files

gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing       -Ipython/anaconda3/include/python3.9 -o main2.so main.c
gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing       -Ipython/anaconda3/include/python3.9 -o helper2.so helper.c

Change all import calls that call main.py and helper.py in my scripts to import main2 and import helper2.

Now, I have two questions:

if helper.py and main.py themselves have import for other .py files, do I need to cythonize them too, and then change the imports, and then cythonize the new file, or is it all done recursively?
I get the error when trying to run files that use main2.so (import main2) and helper2.so (import helper2):
```
ImportError: helper2.so: undefined symbol: Py_EnterRecursiveCall
```

what did I miss in the compilation?

score 1 · Accepted Answer · edited Oct 10 '22 at 19:30

1

No.

A few simple points:

In setup.py you can run multiple Cython files in a single call to cythonize (see https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#multiple-cython-files-in-a-package). There's a few different ways of doing it - I think cythonize(["main.py", "helper.py"]) will probably work.
python3 make_cython.py build_ext --inplace should generate the .so files, so you don't need to do your step 2.
Don't rename anything! The Python (and Cython) import mechanism is very name dependent. If you try to import main2.<something>.so it'll look for a C function called PyInit_main2. That will not be the C function that Cython generates if the file was called main.py originally.
Python will also import .so files ahead of .py files, so again, no need to rename.

if helper.py and main.py themselves have import for other .py files, do I need to cythonize them too, and then change the imports, and then cythonize the new file, or is it all done recursively?

It is not done recursively. Cython can import .py files happily so you don't need to Cythonize it all.

I get the error when trying to run files [...]

You'll probably find this goes away if you just use the files that setup.py generates. But it's because you didn't link libpython.

edited Oct 10 '22 at 19:30

wim

338,267
99
616
750

answered Oct 10 '22 at 19:27

DavidW

29,336
6
55
86

@wim, thanks, you say "It is not done recursively. Cython can import .py files happily so you don't need to Cythonize it all.", but I do want to have "all" files cythonized. My question is whether I need to cythonize them in the order of the dependency tree (from bottom to top), or whether it doesn't matter. I assume becasuse it is .so, the .so itself would look for other .so in runtime. – kloop Oct 10 '22 at 21:21
It doesn't matter what order you do them in. – DavidW Oct 10 '22 at 21:31
thanks. so if the build .py creates main.cpython-39-x86_64-linux-gnu.so, I need to do import main or import main.cpython-39-x86_64-linux-gnu? It sounds a bit strange to do the first, because then there is no way to revert back to .py without deleting the .so (?), but then the second is also counter-intuitive. – kloop Oct 10 '22 at 21:45
`import main`. The `cpython-39-x86_64-linux-gnu` is an internal tag that means you could have a .so files for different versions of Cython in the same directory. You can't easily revert back to the .py without deleting the .so. – DavidW Oct 11 '22 at 05:35
Comment above should read "different versions of *Python* in the same directory" – DavidW Oct 11 '22 at 17:51

Am I doing cythonization right?

1 Answers1