I'm trying to use Sub-interpreter for having distinct environment, and having multi-thread on same environment(interpreter).

However, when I tried to cleanup sub-interpreter, Python raise Fatal error with message:

Py_EndInterpreter: thread still has a frame

Here is the minimal reproducing code:

#include <Python.h>
#include <thread>
using namespace std;

void child(PyThreadState* ts) {
  PyEval_RestoreThread(ts);
  PyRun_SimpleString("print('hello world')");

  PyEval_SaveThread();
}

int main() {
  Py_Initialize();
  PyThreadState* main_ts = PyThreadState_Get();
  PyThreadState* sub_ts = Py_NewInterpreter();

  PyEval_SaveThread();
  thread t1(child, sub_ts);
  thread t2(child, sub_ts);
  t1.join();
  t2.join();

  PyEval_RestoreThread(sub_ts);
  Py_EndInterpreter(sub_ts);
  PyThreadState_Swap(main_ts);
  Py_Finalize();
}

Currently, I figure out two workarounds:

1. create a new thread under sub-interpreter

After spawning sub-interpreter, create a new thread state and use it

new_ts = PyThreadState_New(sub_ts->interp);

Before calling Py_EndInterpreter(), clear and delete the thread by

PyThreadState_Clear(new_ts); PyThreadState_Delete(new_ts);

2. Delete sub-interpreter manually

Since Py_EndInterpreter() will check lots of things, we can remove interpreter manually by:

1. PyInterpreterState_Clear(sub_ts->interp);
2. PyInterpreterState_Delete(sub_ts->interp);

Note: I afraid this method may cause memory leak.

However, I am curious what is wrong with my original answer? Is it related to the Py_NewInterpreter() that the doc has mention that

Note that no actual thread is created

CodePudding user response：

You are passing a single thread state to two threads and trying to do "restore" in both, simultaneously. Intuitively, that seems like a Bad Idea. Each Python thread has a lot of state to track, like the stack and the local variables, which need to be different for each thread.

This has a lot of good information about this rather obscure topic:

https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock