1

When I start my program in cuda-gdb, I get an output like:

[New Thread 0x7fffef8ea700 (LWP 8003)]
[New Thread 0x7fffe35b2700 (LWP 8010)]
[New Thread 0x7fffe2db1700 (LWP 8011)]
[New Thread 0x7fffe25b0700 (LWP 8012)]

I do not understand why these multiple threads are launched in the beginning. I have not launched my program in multi-threaded mode. I am using MPI, but I start one process. So, where are these threads coming from?

This does not affect my debugging process in any way. Its just that I don't understand what this means.

talonmies
  • 70,661
  • 34
  • 192
  • 269
shaunshd
  • 95
  • 5

2 Answers2

4

These threads you see are created by the CUDA runtime library, and aren't directly related to cuda-gdb itself. If you run the same code with gdb, you will also see the same messages.

If you want to see what happens what these threads are doing or where they're coming from, simply compile your code with the -g flag, set a breakpoint in your code (e.g., immediately before a CUDA kernel starts), run it, and then run the following command in the gdb console:

thread apply all backtrace

This command has the same effect of gdb's backtrace, except that it will show the backtrace for all threads created by your program.

In my case, I get the following messages after starting my program:

[New Thread 0x7fffeffb3700 (LWP 7141)]
[New Thread 0x7fffef731700 (LWP 7142)]
[New Thread 0x7fffeef30700 (LWP 7143)]

When I run the command mentioned above in my gdb console, I see the following output:

(gdb) thread apply all backtrace

Thread 4 (Thread 0x7fffeef30700 (LWP 7143)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ffff63c19b7 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#2  0x00007ffff6386bb7 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#3  0x00007ffff63c0f48 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#4  0x00007ffff79bf064 in start_thread (arg=0x7fffeef30700) at pthread_create.c:309
#5  0x00007ffff6cce62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7fffef731700 (LWP 7142)):
#0  0x00007ffff6cc5aed in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff63bf6a3 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#2  0x00007ffff642261e in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#3  0x00007ffff63c0f48 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#4  0x00007ffff79bf064 in start_thread (arg=0x7fffef731700) at pthread_create.c:309
#5  0x00007ffff6cce62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7fffeffb3700 (LWP 7141)):
#0  0x00007ffff6ccfa9f in accept4 (fd=13, addr=..., addr_len=0x7fffeffb2e18, flags=-1) at ../sysdeps/unix/sysv/linux/accept4.c:45
#1  0x00007ffff63c0556 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#2  0x00007ffff63b404d in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#3  0x00007ffff63c0f48 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#4  0x00007ffff79bf064 in start_thread (arg=0x7fffeffb3700) at pthread_create.c:309
#5  0x00007ffff6cce62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7ffff7fc0740 (LWP 7136)):
#0  main () at cuda_heap.cu:66

As you can verify, all threads that have been created at the beginning match both thread addresses and LWP (Light Weight Process) IDs. You can see that all of them come from libcuda.so.1.

In cuda-gdb, you're able to see some more detailed information:

(cuda-gdb) thread apply all bt

Thread 4 (Thread 0x7fffeef30700 (LWP 10019)):
#0  0x00007ffff79c33f8 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007ffff63c19b7 in cudbgApiDetach () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#2  0x00007ffff6386bb7 in cudbgApiDetach () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#3  0x00007ffff63c0f48 in cudbgApiDetach () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#4  0x00007ffff79bf064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007ffff6cce62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 3 (Thread 0x7fffef731700 (LWP 10018)):
#0  0x00007ffff6cc5aed in poll () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff63bf6a3 in cudbgApiDetach () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#2  0x00007ffff642261e in cuVDPAUCtxCreate () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#3  0x00007ffff63c0f48 in cudbgApiDetach () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#4  0x00007ffff79bf064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007ffff6cce62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 2 (Thread 0x7fffeffb3700 (LWP 10017)):
#0  0x00007ffff6ccfa9f in accept4 () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff63c0556 in cudbgApiDetach () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#2  0x00007ffff63b404d in cudbgApiDetach () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#3  0x00007ffff63c0f48 in cudbgApiDetach () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#4  0x00007ffff79bf064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007ffff6cce62d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 1 (Thread 0x7ffff7fc0740 (LWP 10007)):
#0  main () at cuda_heap.cu:66
pentschev
  • 350
  • 1
  • 4
  • I have a question related with this one, maybe you can shed some light? https://stackoverflow.com/questions/49631187/how-to-debug-cuda-program-with-multiple-threads – martinako Apr 03 '18 at 13:22
-2

i don't know what exactly it is, but I think cuda-gdb need to create multiple thread to catch the errors/exceptions like: memory violation or bank conflicts.

talonmies
  • 70,661
  • 34
  • 192
  • 269
dk1111
  • 188
  • 1
  • 5