How to identify which thread holds the std::recursive_mutex by gdb?

Question

If I use std::mutex and meet a deadlock, I can find which thread is holding the mutex by gdb

$ p mutex
$3 = (std::__1::mutex &) @0x7f70b56ffc90: {__m_ = {__data = {__lock = 2, __count = 0, __owner = 28539, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}},

The __owner field says the mutex is held by thread 28539.

However, when it comes to std::recursive_mutex or std::shared_mutex, things are getting harder, since I can only get the <incomplete type>. The same error to std::shared_mutex.

90  in /usr/local/bin/../include/c++/v1/__mutex_base
(gdb) p __m
$5 = (std::__1::lock_guard<std::__1::recursive_mutex>::mutex_type &) @0x7f163db94558: <incomplete type>

If I specify the type then

p (struct std::__recursive_mutex_base) __m
No struct type named std.

I wonder how can I know which threads holds the std::recursive_mutex or std::shared_mutex.

Hi calvin- at the point of deadlock, maybe you can get a clue by doing "thread apply all bt" - the stacks might show which threads are effectively stuck so you can infer which threads are contending the lock? I'd agree it would be nicer to do as per the std::mutex though. — gremto, Jun 18 '23 at 13:21
@gremto Hi gremto, in my scenario, there are many threads being affected. There is more than one lock, so one thread a with lock A held can acquire lock B which is owned by thread b. These threads can wait in a circle like a -> b -> c -> a. So if we run `thread apply all bt`, we can see many threads are stuck in different locks. Maybe we can re-construct the waiting chain by checking all threads' `backtrace` result, however, it can be very hard work. — calvin, Jun 18 '23 at 13:34
Hmm, tricky- just thinking out loud, I'm wondering then if it's possible with a bit of experimentation to determine the address offset of the data holding the lock owner value, maybe do some manual memory dereferencing and interpretation in gdb working on a simple standalone harness- just a minimal app to be able to check the std::shared_mutex memory layout? — gremto, Jun 18 '23 at 15:09
Yes, it may be a solution. However, we lack the means to judge whether the value we get is right. — calvin, Jun 19 '23 at 03:06

calvin · Accepted Answer · 2023-06-19T09:49:29.523

I find two ways to "fix" this:

I find there is an error
```
90  /usr/local/bin/../include/c++/v1/__mutex_base: No such file or directory.
```
In my g++-9, there is no __mutex_base or v1 provided. However, I find a mutex which is quite similar to this __mutex_base file. So I have this file modified that line 90 is exactly the definition of recursive_mutex, then I directory /usr/include/c++. After that p (struct std::__recursive_mutex_base) __m will work.

Another way comes from Can we define a new data type in a GDB session I write the following codes

#include <mutex>
#include <shared_mutex>

struct r_wrapper: std::recursive_mutex {
    void f() {
        printf("dummy");
    }
};

struct s_wrapper: std::shared_mutex {
    void f() {
        printf("dummy");
    }
};

struct r_wrapper r_wrapper_instance;
struct s_wrapper s_wrapper_instance;

And make sure r_wrapper_instance is a symbol. Then I just add-symbol-file a.o 0, and p (struct r_wrapper&) __m will print.

How to identify which thread holds the std::recursive_mutex by gdb?

1 Answers1