A boost thread_group is a group of threads.
All the threads are, by definition, distinct and unique.
So, if your mutex fails to... "mut-ex" (mutually exclude) this indicates a programmer error elsewhere.
On a whim, I'd suggest that perhaps your expectations of mutexes (recursive or not) is not accurate (in similar way as the expectations of a thread group were)?
The thing you might have missed is that all parties accessing the shared objects /must/ acquire the mutex at all times of these accesses. It is not enough to have one thread "guard" the shared objects, to magically keep other threads out.
Mutexes are a coöperative proposition. The term "critical section" is actually a bit nicer in this respect: you "mark" the critical sections in code, as opposed to "marking" the critical data (whatever that would be).