1

I'm trying to use Sub-interpreter for having distinct environment, and having multi-thread on same environment(interpreter).

However, when I tried to cleanup sub-interpreter, Python raise Fatal error with message:

Py_EndInterpreter: thread still has a frame

Here is the minimal reproducing code:

#include <Python.h>
#include <thread>
using namespace std;

void child(PyThreadState* ts) {
  PyEval_RestoreThread(ts);
  PyRun_SimpleString("print('hello world')");

  PyEval_SaveThread();
}

int main() {
  Py_Initialize();
  PyThreadState* main_ts = PyThreadState_Get();
  PyThreadState* sub_ts = Py_NewInterpreter();

  PyEval_SaveThread();
  thread t1(child, sub_ts);
  thread t2(child, sub_ts);
  t1.join();
  t2.join();

  PyEval_RestoreThread(sub_ts);
  Py_EndInterpreter(sub_ts);
  PyThreadState_Swap(main_ts);
  Py_Finalize();
}

Currently, I figure out two workarounds:

1. create a new thread under sub-interpreter

After spawning sub-interpreter, create a new thread state and use it

new_ts = PyThreadState_New(sub_ts->interp);

Before calling Py_EndInterpreter(), clear and delete the thread by

PyThreadState_Clear(new_ts); PyThreadState_Delete(new_ts);

2. Delete sub-interpreter manually

Since Py_EndInterpreter() will check lots of things, we can remove interpreter manually by:

  1. PyInterpreterState_Clear(sub_ts->interp);
  2. PyInterpreterState_Delete(sub_ts->interp);

Note: I afraid this method may cause memory leak.

However, I am curious what is wrong with my original answer? Is it related to the Py_NewInterpreter() that the doc has mention that

Note that no actual thread is created

Maxwen
  • 43
  • 3
  • You are passing a single thread state to two threads and trying to do "restore" in both, simultaneously. Intuitively, can't you see why that is a bad idea? – Tim Roberts Aug 19 '22 at 04:50
  • To @TimRoberts: AFAIK, restore will try to acquire GIL and bind to a real thread, and release will unbind. Actually I am quite confuse about the concept of "python thread" when use as embedded indeed, since python thread isn't a "real" thread, then what is it exist for in embedded python? – Maxwen Aug 19 '22 at 05:32
  • However, the first workaround also use same thread state in two threads, but it works (at least in my project it works correctly without unexpected result). So, is it really needs to 1 thread state for one thread? May you please explain the mechanism in it? That will be a great help! – Maxwen Aug 19 '22 at 05:39
  • Have you read through this: https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock ? – Tim Roberts Aug 19 '22 at 07:06
  • ok, after reread the part "Non-Python created threads", it actually state that the needs of creating new thread state. So, not just GIL, we need to mange thread state too. Now it sounds reasonable that most GIL operations are related to thread state function. – Maxwen Aug 19 '22 at 07:21
  • But I still wondering the needs of thread state, is the exist of thread state let Python able to help us to switch thread automatically? Also, could you @TimRoberts post the answer? thanks! – Maxwen Aug 19 '22 at 07:28
  • Each Python thread has a lot of state to track, like the stack and the local variables, which are different for each thread. – Tim Roberts Aug 19 '22 at 18:58

1 Answers1

0

You are passing a single thread state to two threads and trying to do "restore" in both, simultaneously. Intuitively, that seems like a Bad Idea. Each Python thread has a lot of state to track, like the stack and the local variables, which need to be different for each thread.

This has a lot of good information about this rather obscure topic:

https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock

Tim Roberts
  • 48,973
  • 4
  • 21
  • 30