[Python][C++] How to use pyarrow in the python sub-interpreter

[zeroRains]

Describe the usage question you have. Please include as many useful details as possible.

Hi, I want to execute the same function with different python sub-interpreter in different threads. The function codes are like that

void f(const char *tname)
{
    PyThreadState *_ts = Py_NewInterpreter();
    // sub.swap_to_self();
    std::string code = R"PY(
import pyarrow
print("TNAME gan ni niang")

a = 0
for i in range(100000000):
    a = a+1
    )PY";

    code.replace(code.find("TNAME"), 5, tname);
    PyRun_SimpleString(code.c_str());
    Py_EndInterpreter(_ts);
}

But I get a error when I import the pyarraow in the sub-interpreter

import pyarrow.lib as _lib
ImportError: module pyarrow.lib does not support loading in sub-interpreters

Dose I use the pyarrow in a wrong way? Or It dosen't allow use in the sub-interpreters?

If the later is, will it support in the future？

If it can use in sub-interpreter, how should I do?

Component(s)

python, c++

[raulcd]

cc @pitrou

[pitrou]

Unfortunately, Cython support for subinterpreters is experimental and unfinished for now (see CYTHON_USE_MODULE_STATE in http://docs.cython.org/en/latest/src/userguide/source_files_and_compilation.html#c-macro-defines).

Once Cython support is finished for it, we should probably try to enable it.

[zeroRains]

I use the the subinterpreters in CPython, not in Cython. It support for subinterpreters in the latest python version. (see in https://docs.python.org/3/c-api/init.html#sub-interpreter-support)

[pitrou]

PyArrow uses Cython, this is why it's not compatible with sub-interpreters.

[zeroRains]

OK，I got that， thanks