2

I have a really specific need : I want to create a python console with a Qt Widget, and to be able to have several independent interpreters. Now let me try to explain where are my problems and all tries I did, in order of the ones I'd most like to make working to those I can use by default

  • The first point is that all functions in the Python C API (PyRun[...], PyEval[...] ...) need the GIL locked, that forbid any concurrent code interpretations from C ( or I'd be really glad to be wrong !!! :D )

  • Therefore, I tried another approach than the "usual way" : I made a loop in python that call read() on my special file and eval the result. This function (implemented as a built extension) blocks until there is data to read. (Actually, it's currently a while in C code rather than a pthread based condition) Then, with PyRun_simpleString(), I launch my loop in another thread. This is where the problem is : my read function, in addition to block the current thread (that is totally normal), it blocks the whole interpreter, and PyRun_simpleString() doesn't return...

  • Finally I've this last idea which risks to be relatively slow : To have a dedicated thread in C++ which run the interpreter, and do every thing in python to manage input/output. This could be a loop which creates the jobs when there is a console needing to execute a command. Seems not to be really hard to do, but I prefer ask you : is there a way to make the above possibilities to work, or is there another way I didn't think about or is my last idea the best ?

Clifford
  • 88,407
  • 13
  • 85
  • 165
hl037_
  • 3,520
  • 1
  • 27
  • 58
  • I'm somehow missing the question mark in this "question". – mfro Mar 16 '14 at 10:04
  • :D yes, indeed I'll edit - done. In fact, I need advise on the best way to do this kind of multithreading in python, to have all thread executing concurently not to have one thread which blocks each others – hl037_ Mar 16 '14 at 10:05

2 Answers2

3

One alternative is to just re-use code from IPython and its Qt Console. This assumes by independent interpreters you imply they won't share memory. IPythons run the Python interpreter in multiple processes and communicates with them over TCP or Unix domain sockets with the help of ZeroMQ.

Also, from your question I'm not sure if you're aware of the common blocking I/O idiom in Python C extensions:

Py_BEGIN_ALLOW_THREADS
... Do some blocking I/O operation ...
Py_END_ALLOW_THREADS

This releases the GIL so that other threads can execute Python code while your function is blocking. See Python/C API Reference Manual: Thread State and the Global Interpreter Lock.

scottt
  • 7,008
  • 27
  • 37
  • Thank you ! indeed I didn't really understand how it works. I though that python thread are totally independent but no, there is only one interpreter which schedule the execution, am i right ? – hl037_ Mar 16 '14 at 13:26
  • 1
    @hl037_, the way I think about it: **processes** and **threads** are OS level concepts, while an **interpreter** here is just the state maintained by a piece of C library code. Among the state maintained by CPython is the GIL. CPython doesn't schedule threads. It just uses OS level thread operations. One CPython thread maps to one OS thread. – scottt Mar 16 '14 at 13:46
  • ok, I saw it because there are one OS thread per thread created by python. – hl037_ Mar 16 '14 at 13:49
0

If your main requirement is to have several interpreters independent from each other, you'd probably better suited doing fork() and exec() than doing multithreading.

This way each of the interpreters would live in it's own address space not disturbing one of the others.

mfro
  • 3,286
  • 1
  • 19
  • 28