How can I find out what's creating so many threads in my python program?

Question

I've noticed that my Python program creates a ton of threads that quickly exit when I start the script. I assume that one of the C extensions I'm using is creating the threads, although I'm also using multiprocessing to kick off one additional python process. Is there an easy way to figure out which extension is doing it, or if something else is spawning the threads?

Using gdb, I see this:

Starting program: /usr/bin/python ../myscript.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffee4be700 (LWP 29041)]
[New Thread 0x7fffedcbd700 (LWP 29042)]
[New Thread 0x7fffeb4bc700 (LWP 29043)]
[New Thread 0x7fffe8cbb700 (LWP 29044)]
[New Thread 0x7fffe64ba700 (LWP 29045)]
[New Thread 0x7fffe3cb9700 (LWP 29046)]
[New Thread 0x7fffe14b8700 (LWP 29047)]
[New Thread 0x7fffdecb7700 (LWP 29048)]
[New Thread 0x7fffdc4b6700 (LWP 29049)]
[New Thread 0x7fffd9cb5700 (LWP 29050)]
[New Thread 0x7fffd74b4700 (LWP 29051)]
[New Thread 0x7fffd4cb3700 (LWP 29052)]
[New Thread 0x7fffd24b2700 (LWP 29053)]
[New Thread 0x7fffcfcb1700 (LWP 29054)]
[New Thread 0x7fffcd4b0700 (LWP 29055)]
[New Thread 0x7fffcacaf700 (LWP 29056)]
[New Thread 0x7fffc84ae700 (LWP 29057)]
[New Thread 0x7fffc5cad700 (LWP 29058)]
[New Thread 0x7fffc34ac700 (LWP 29059)]
[New Thread 0x7fffc0cab700 (LWP 29060)]
[New Thread 0x7fffbe4aa700 (LWP 29061)]
[New Thread 0x7fffbbca9700 (LWP 29062)]
[New Thread 0x7fffb94a8700 (LWP 29063)]
[Thread 0x7fffc5cad700 (LWP 29058) exited]
[Thread 0x7fffb94a8700 (LWP 29063) exited]
[Thread 0x7fffe3cb9700 (LWP 29046) exited]
[Thread 0x7fffbbca9700 (LWP 29062) exited]
[Thread 0x7fffe8cbb700 (LWP 29044) exited]
[Thread 0x7fffbe4aa700 (LWP 29061) exited]
[Thread 0x7fffdecb7700 (LWP 29048) exited]
[Thread 0x7fffc0cab700 (LWP 29060) exited]
[Thread 0x7fffee4be700 (LWP 29041) exited]
[Thread 0x7fffc34ac700 (LWP 29059) exited]
[Thread 0x7fffd74b4700 (LWP 29051) exited]
[Thread 0x7fffc84ae700 (LWP 29057) exited]
[Thread 0x7fffcfcb1700 (LWP 29054) exited]
[Thread 0x7fffe14b8700 (LWP 29047) exited]
[Thread 0x7fffe64ba700 (LWP 29045) exited]
[Thread 0x7fffcacaf700 (LWP 29056) exited]
[Thread 0x7fffedcbd700 (LWP 29042) exited]
[Thread 0x7fffcd4b0700 (LWP 29055) exited]
[Thread 0x7fffdc4b6700 (LWP 29049) exited]
[Thread 0x7fffd24b2700 (LWP 29053) exited]
[Thread 0x7fffd4cb3700 (LWP 29052) exited]
[Thread 0x7fffeb4bc700 (LWP 29043) exited]
[Thread 0x7fffd9cb5700 (LWP 29050) exited]
20150201 22:13:27.896983 - INFO - myscript.py:517 - Begin logging
[New Thread 0x7fffb94a8700 (LWP 29070)]
[New Thread 0x7fffbbca9700 (LWP 29071)]

score 2 · Answer 1 · edited May 23 '17 at 12:06

2

Using gdb breakpoint on pthread_create I figured out that it was libblas creating all the threads. Setting OMP_NUM_THREADS to 1 fixed the issue.

edited May 23 '17 at 12:06

Community

1
1

answered Feb 02 '15 at 04:30

Thomas Johnson

10,776
18
60
98

score 0 · Answer 2 · answered Feb 02 '15 at 04:22

0

Maybe try crashing the threads and inspecting the stack trace. Without code it will hardly be possible to help more

answered Feb 02 '15 at 04:22

André Fratelli

5,920
7
46
87

Not helpful without details on how to 'crash the threads', but this answer was: http://stackoverflow.com/questions/1440643/gdb-breakpoint-on-pthread-create – Thomas Johnson Feb 02 '15 at 04:29
Simply raising an exception would do, but I guess breakpoints are a better idea. – André Fratelli Feb 02 '15 at 04:36

How can I find out what's creating so many threads in my python program?

2 Answers2