0

I'm writing a library that hooks some CUDA functions to add some functionality. The "constructor" hooks the CUDA functions and set up message queue and shared memory to communicate with other hooked CUDA binaries. When launching several hooked CUDA binaries (by python subprocess.Popen('<path-to-binary>', shell=True)) some processes hangs. So I used gdb -p <pid> to attach one suspended process, hoping to figure out what's going wrong. Here's the result:

Attaching to process 7445
Reading symbols from /bin/dash...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.27.so...done.
done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.27.so...done.
done.
0x00007f9cefe8b76a in wait4 () at ../sysdeps/unix/syscall-template.S:78
78      ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0  0x00007f9cefe8b76a in wait4 () at ../sysdeps/unix/syscall-template.S:78
#1  0x000055fff93be8a0 in ?? ()
#2  0x000055fff93c009d in ?? ()
#3  0x000055fff93ba6d8 in ?? ()
#4  0x000055fff93b949e in ?? ()
#5  0x000055fff93b9eda in ?? ()
#6  0x000055fff93b7944 in ?? ()
#7  0x00007f9cefdc8b97 in __libc_start_main (main=0x55fff93b7850, argc=3, argv=0x7ffca7c7beb8, init=<optimized out>,
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffca7c7bea8) at ../csu/libc-start.c:310
#8  0x000055fff93b7a4a in ?? ()

I've added -g flag but it seems that the program hangs on wait4 before entering main.

Thanks for any insights on:

  • How can I load these debug symbols to get rid of ??
  • Where is ../csu/libc-start.c:310 located?
  • What else can I do to locate the bug?

System Info: gcc 6.5.0, Ubuntu 18.04 with 4.15.0-54-generic.

zingdle
  • 491
  • 5
  • 17
  • 1
    What is your library doing? Why does it need to use `__attribute__((constructor))`? What is that "constructor" code doing? – Some programmer dude Aug 05 '19 at 13:17
  • Providing a [mre] of a constructor function that exhibits the troublesome behavior would make it a lot more likely that we can provide a useful answer. Also, it would help to describe more clearly what the "when launching multiple processes" scenario looks like. – John Bollinger Aug 05 '19 at 13:20
  • @Someprogrammerdude The library is hooking some CUDA functions to add some functionality. The constructor hooks the functions and set up message queue and shared memory to communicate with other hooked CUDA binaries. – zingdle Aug 05 '19 at 13:31
  • 1
    @zingdle We're going to need you to be a whole lot more specific. – zwol Aug 05 '19 at 13:32
  • @JohnBollinger I'm trying to narrow it down to a minimal reproducible example but that may take some more time. I hooked several CUDA binaries and launching them concurrently. – zingdle Aug 05 '19 at 13:33
  • @zingdle I wonder how `/bin/dash` got involved at all, if you're messing with CUDA binaries. Did you use `system` for something? Also, do you know that you can edit your question? The tiny gray word "edit" under the tags is a button. Yes, we know it's bad UI design, sorry. Anyway, whenever you get comments asking for lots more information, please provide that information by editing the question, rather than responding with more comments. – zwol Aug 05 '19 at 13:35
  • @zwol Thanks for your advice and I've edited the question. I'm using python to launch multiple process and I guess that's where dash comes in. – zingdle Aug 05 '19 at 13:46
  • @zingdle Ohh, you're using `shell=True`. Then `/bin/dash` processes hanging around are totally normal. It's probably not deadlocked, it's just waiting for the actual program you ran to finish before it exits itself. See my edited answer for more explanation. – zwol Aug 05 '19 at 14:52

1 Answers1

2

How can I load these debug symbols to get rid of ??

You appear to need the debug symbols for /bin/dash, which are probably going to be in a package called dash-dbg or dash-dbgsym or something like that.

Also, I suspect your stack trace would make more sense if you compiled your library with -fno-optimize-sibling-calls.

Where is ../csu/libc-start.c:310 located?

See this answer.

What else can I do to locate the bug?

You said that you are writing a library that uses __attribute__((constructor)), but you showed a stack trace for /bin/dash (which I presume is DASH and not a program you wrote) that does not appear to involve symbols from your library. I infer from this, that your library is loaded with LD_PRELOAD into programs that are not expecting it to be there.

Both of those things -- LD_PRELOAD and __attribute__((constructor)) -- break the normal expectations of both whatever unsuspecting program is involved, and the C library. You should only do those things if you have no other choice, and you should try to do as little as possible within the injected code. (In particular, I do not think any design that involves spawning processes from a constructor function will be workable, period.) If you tell us about your larger goals we may be able to suggest alternative means that are less troublesome.

EDIT: 

subprocess.Popen('<path-to-binary>', shell=True)

With shell=True, Python doesn't invoke the program directly, it runs a command of the form /bin/sh -c 'string passed to Popen'. In many cases this will naturally produce a /bin/dash process sleeping (not hung) in a wait syscall for the entire lifetime of the actual binary. Unless you actually need to evaluate some shell code before running the program, try the default shell=False instead and see if that makes your problem go away. (If you do need to evaluate shell code, try Popen('<shell code>; exec <binary>', shell=True).)

zwol
  • 135,547
  • 38
  • 252
  • 361
  • Thanks, I have no acces to the the code right now but I'll try your suggestion asap. I'm redirecting log to file. How can I do this with `shell=False`? – zingdle Aug 05 '19 at 15:38
  • 1
    For instance, you can replace `subprocess.Popen('binary > log.txt', shell=True)` with `subprocess.Popen(['binary'], shell=False, stdout=open("log.txt", "w"))`. I'd suggest reading through the full list of `Popen` keyword arguments. – zwol Aug 05 '19 at 16:17
  • Now I can attach to the process by setting `shell=False`. It turns out that one of the process run out of resources then crashes and others are waiting for syncing. – zingdle Aug 06 '19 at 02:07