ImportError: File.so undefined symbol: PyInterpreterState_GetID

Question

When Cythonizing a file and running it via import, an error is presented that claims there is a problem getting the python interpreter state.

Removing multiprocessing code like multiprocessing.start(); switching to Cython's own prange().

Googling what the error means, and there is not a single result that has talked about this since I first came across it (2 years ago).

Cython 0.29.6 Python 3.7

build using

pyhton3 setup.py build_ext --inplace

Where the setup.py contains

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

ext_modules = [Extension("Logic", ["Logic.pyx"], extra_compile_args=['-Ofast'],)]
setup(name="Logic", ext_modules=cythonize("Logic.pyx"))

And the problem code... There is a small list of errors expected, but the interpreter state was not one of them.

import secrets
import os
import time
import Cython

cdef list race = ["Asian", "Black", "White"]
cdef list hair = ["Brown", "Brown", "Brown", "Brown", "Black", "Black", "Black", "Blond", "Blond", "Red"]
cdef list eyes = ["Brown", "Brown", "Brown", "Blue", "Blue", "Blue", "Green", "Green", "Grey", "Hazel"]
cdef list gender = ["Female", "Male", "Female", "Female", "Female", "Female", "Male", "Male", "Male", "Male", "Female", "Female", "Female", "Female", "Male", "Male", "Male", "Male", "Female", "Female", "Female", "Female", "Male", "Male", "Male", "Male", "Female", "Female", "Female", "Female", "Male", "Male", "Male", "Male", "Female", "Female", "Female", "Female", "Male", "Male", "Male", "Male", "Female", "Female", "Female", "Female", "Male", "Male", "Male", "Male", "Female", "Female", "Female", "Female", "Male", "Male", "Male", "Male", "Female", "Female", "Female", "Female", "Male", "Male", "Male", "Male", "Female", "Female", "Female", "Female", "Male", "Male", "Male", "Male", "Female", "Female", "Female", "Female", "Male", "Male", "Male", "Male", "Female", "Female", "Female", "Female", "Male", "Male", "Male", "Male", "Female", "Female", "Female", "Female", "Male", "Male", "Male", "Male", "X", "XXY"]
cdef long age_limit_elder = 115
cdef long weight_limit = 400
cdef long age_limit_adult = 60
cdef long age_limit_child = 18
cdef long weight_limit_infant = 15
cdef long age_limit_infant = 4
cdef long population = 4873057333
cdef long infant = 633497453  # 13%
cdef long child = 1461917200  # 30%
cdef long adult = 2095414652  # 43%
cdef long elder = 682228028  # 14%
cdef list infant_population = []
cdef list child_population = []
cdef list adult_population = []
cdef list elder_population = []

cdef list adult_race_pool = []

cpdef dict adult_generation(long adult=adult, long age_limit_adult=age_limit_adult, long weight_limit=weight_limit, list race=race, list hair=hair, list eyes=eyes, list gender=gender, long age_limit_child=age_limit_child):
    cdef long age_ = secrets.choice(range(age_limit_child, age_limit_adult))
    cdef long weight_ = secrets.choice(range(90, weight_limit))
    cdef str race_ = secrets.choice(race)
    cdef str hair_ = secrets.choice(hair)
    cdef str eyes_ = secrets.choice(eyes)
    cdef str gender_ = secrets.choice(gender)
    # adult_population.append({"AGE": age_, "WEIGHT": weight_, "RACE": race_, "HAIR": hair_, "EYES": eyes_, "GENDER": gender_})
    return {"AGE": age_, "WEIGHT": weight_, "RACE": race_, "HAIR": hair_, "EYES": eyes_, "GENDER": gender_}

# adult_generation()
# cdef long adult_processes = adult // 4
# adult_processes = 10000
# cdef long adult_processes = 100000
cdef long adult_processes = adult  # Fallback
cdef long adult_percent = adult_processes // 10000
cdef double percent = 0
cdef long current = 0
cdef double t1 = time.time()
cdef double master = t1
cdef long i

for i in Cython.parallel.prange(adult_processes, nogil=True):
    if current == adult_percent:
        percent += 0.01
        t2 = time.time()
        t3 = t2 - t1
        os.system("tput reset")
        print(f"{percent}% Complete | {current} Profiles in {t3} Seconds")
        current = 0
        t1 = time.time()
    adult_population.append(adult_generation())
    current += 1

cdef long F = 0
cdef long M = 0
cdef long X = 0
cdef long XXY = 0

cdef dict item

for item in adult_population:
    if item["GENDER"] == "Female":
        F += 1
    elif item["GENDER"] == "Male":
        M += 1
    elif item["GENDER"] == "X":
        X += 1
    elif item["GENDER"] == "XXY":
        XXY += 1
    else:
        pass


print(f"FEMALE: {F}")
print(f"MALE: {M}")
print(f"X: {X}")
print(f"XXY: {XXY}")
print(f"Total Time: {time.time() - master} Seconds")

The result should just be a giant list of pseudo random people like in the Python version but without the odd 70% CPU limit on all threads/cores.

However, in Cython, when made parallel or multiprocessed it returns "undefined symbol: PyInterpreterState_GetID" on import and just quits.

Testing was done with smaller numbers as the current numbers would take 146 days of processing... Changing the number of results still produces an error.

It was also expected to get hung up at the "cdef dict item", but again, it never even gets to that.

Answer 1

I can't say this is a definite answer, but, I noticed that the Python error output was substantially short. I took a little time and a gamble and straight compiled the Cython code into an embedded package. This package provided full error outputs, even though it was in a mess, it still led somewhere.

The interpreter state seemed to have something to do with threading/multiprocessing, so I tweaked that part, and found the error output to be complaining about the GIL. Apparently the error given to Python was from Cython but not shown to user until compiled as a standalone "C" app.

This error seems to center around the GIL, with the output complaining that it can't remove the GIL from functions that need the GIL, namely, every piece of Python code except return, yield, print, simple math, etc...

So, the prange and its parallel ability and very close to useless to someone wanting to boost a pure Python script. As anything relying on def, open, threading/multiprocessing module, or really anything Pythonic, will produce this error, and should the one using it be testing their Cython code through Python import, they will simply be at a dead end, with a one line error that does not come up in google search results.

The fix for the prange error requires rewriting the entirety of what is being interacted with in C, or import something that uses C instead of Python. This again makes it less useful to a pure Python developer.

In the end, I jumped to the map function, which allows multiprocessing without the overhead of spawning so many processes (in the end it would be near 3 billion).