0

CPython is a multi-threaded application, and as such on Unix it uses (p)threads. Python extensions (written in C, say) often need to hold GIL to make sure Python objects don't get corrupted in critical sections of the code. How about other types of data? Specifically, does holding GIL in a Python extension guarantee that all other threads of CPython stop?

The reason for asking is that I am trying to port to FreeBSD a Python extension (which works on Linux and OSX) that embeds a Lisp compiler/system ECL using Boehm GC, and which crashes during the initialisation of the embedded Boehm GC. Backtraces suggest that another thread kicks in and causes havoc (pthread implementation on Linux are sufficiently different from FreeBSD's to expect trouble along these lines, too). Is there another mutex in CPython that may be used to achieve the locking?

Dima Pasechnik
  • 336
  • 4
  • 16

1 Answers1

1

Specifically, does holding GIL in a Python extension guarantee that all other threads of CPython stop?

The short answer is no - if other threads are executing code without holding the GIL (e.g. if they're running a C extension that releases the GIL), then they will keep running until try try to re-acquire the GIL (usually when they try to return (data) to the world of python).

It's also possible that core parts of the CPython (the core interpreter and/or built in fucntions/packages could release the GIL in similar circumstances/reasons that you would do in an extension. I have no idea if they actually do though.

Tom Dalton
  • 6,122
  • 24
  • 35
  • Is GIL the only mutex used by CPython internally? Or are there more? – Dima Pasechnik Sep 22 '17 at 14:30
  • No idea. "Backtraces suggest that another thread kicks in and causes havoc". Havoc with what? do you have multiple (python) threads messing with your extension's memory? I'd have though you'd use a single python thread to initialise the embedded interpreter, even if you later on allow multiple python/system threads to use it? – Tom Dalton Sep 22 '17 at 15:02
  • it was ipython with its prompt toolkit. – Dima Pasechnik Nov 10 '20 at 12:32