5

The title says it most, really. On Linux it would be easy with strace and possibly lsof or /proc, and it used to be easy on OSX until truss was removed from OSX Leopard, along with underlying system calls (afaik).

The obvious approach is to tackle this problem with dtrace, but alas as far as I understand dtrace won't do because it catches events as they happen - and in my case, the blocking system call has already started. I'd love to stand corrected if this can be solved with dtrace, by the way.

I saw Xcode's Instruments has a monitor which achieves something similar by taking periodic samples of a process' stack (not sure what system calls it relies on to do that!), maybe something similar on the command line could be good enough (as it would display the stack all the way to the library call that wraps the system call). To be useful for my usecase, this "sampling commandline tool" would have to find and parse the arguments it finds on the stack to be useful to determine what file/file descriptor are we blocked on.

One last thing - on Linux, you could usually do this as a regular user (assuming no ptrace_scope tricks). It would be great if the OSX solution won't require root, either.

Yaniv Aknin
  • 4,103
  • 3
  • 23
  • 29
  • Would `gdb attach PID` and then `bt` give you what you need? – dubek Oct 29 '12 at 19:18
  • Probably not `gdb` on OSX, but rather `lldb` as `@kobyk` suggested privately. I'm still looking into the details, but with some manual tinkering I was able to see (a) which system call caused the block (b) what arguments it had using `lldb`. No root required, a signal was sent to the process (not sure why). I will post a more complete answer if I get somewhere better than `sample` as suggested below. Thanks! – Yaniv Aknin Oct 29 '12 at 19:31

2 Answers2

5

You can use sample: sample PID -e

E.g. for the nc -l localhost 5999 you'd get a file with call graph:

Call graph:
    9046 Thread_242504   DispatchQueue_1: com.apple.main-thread  (serial)
      9046 start  (in libdyld.dylib) + 1  [0x7fff90a847e1]
        9046 ???  (in nc)  load address 0x102453000 + 0x166c  [0x10245466c]
          9046 __accept  (in libsystem_kernel.dylib) + 10  [0x7fff94445996]

And other useful information like loaded Binary Images.

Kentzo
  • 3,881
  • 29
  • 54
2

I'm proposing a solution that assumes the following:

  1. the process can be signaled.
  2. the process is single threaded (a signal might not be handled in the blocking thread).
  3. you have root access.

dtruss supports printing a complete stack trace for every system call (the -s argument).

example:

terminal 1:

$ python
>>> import socket
>>> s = socket.socket()
>>> s.bind(('', 1234))
>>> s.listen(1)
>>> s.accept() # blocking!

terminal 2:

$ dtruss -s -p `pgrep python` # or your python pid if you don't have pgrep (port install proctools)

terimal 1:

press Ctrl-C

terminal 2:

sigreturn(0x7FFF5FBFD660, 0x1E, 0x10031B3D0)             = 0 Err#-2
          libsystem_kernel.dylib`__accept+0xa # HERE IT IS!
          libpython2.7.dylib`PyEval_EvalFrameEx+0x42bf
          libpython2.7.dylib`fast_function+0xb3
...

the first frame in the stack if sigreturn (teardown code for signal handlers: http://linux.die.net/man/2/sigreturn).

the second frame is the standard library wrapper for our system call: accept.

Alon Horev
  • 116
  • 4
  • Dumping the stack is a nice trick and takes us further than I've reached so far, but we're missing which descriptor we were blocking on, we must disturb the process with a signal and we need root. I'll mark this as the answer, but will be happy to receive (and mark...) a better answer. Thanks mate! – Yaniv Aknin Oct 27 '12 at 09:22