c++11 interprocess atomics and mutexes

Question

I have a Linux program which spawns several processes (fork) and communicates through POSIX Shared Memory. I'd like to have each process allocate an id (0-255). My intention is to place a bitvector in the shared memory region (initialized to zero) and atomically compare and swap a bit to allocate an id.

Is there a c++11-friendly way to do this? Can I create an atomic bitset? Can I use a mutex across processes? How do I assure that constructors get called once and only once across all processes?

fork returns a pid of child process to the parent proces, why not just use that instead of generating yet another one? Or I'm misunderstanding your question — , Oct 31 '12 at 15:22
@aleguna Because I would like a value ranging from 0-255 and when a process leaves this program it should free it's ID to be reused. — dschatz, Oct 31 '12 at 15:24
@dschatz: "*Because I would like a value ranging from 0-255*" That doesn't explain *why* you need that. Especially when the PID mechanism is so much *better*. — Nicol Bolas, Oct 31 '12 at 16:12
@NicolBolas I took my problem and selected the minimal necessary information to ask a question that would allow me to solve my problem. I'm not looking for people to change what the problem is. I could create an arbitrary synchronization problem and ask it with respect to posix shared memory across processes and the question still holds. If it helps you, pretend I asked about a shared data structure and how c++ mutexes or atomics would work with respect to that. — dschatz, Oct 31 '12 at 16:14
@dschatz: And thus, you have a [classic XY question.](http://meta.stackexchange.com/questions/66377/what-is-the-xy-problem) — Nicol Bolas, Oct 31 '12 at 16:38
@NicolBolas You clearly missed the point based on your first question. I'm asking generally how c++11 mutexes and atomics work across processes (Hey, the title matches!). I give a specific example to make it concrete. You then proceed to nitpick on the example. If you don't have an answer, that's fine, but there's no reason to try to move the goalposts to answer my question. — dschatz, Oct 31 '12 at 16:53
@dschatz: "*I'm asking generally how c++11 mutexes and atomics work across processes (Hey, the title matches!).*" Then you should not give a very specific, contrived, and useless example as the first paragraph of a general question. If you had stated it as, "I want to know how C++11 threading primitives work across processes. For example, blah", then it would have been clear that the question was general-purpose. A generic title with a specific internal question is common on SO. So much so that I generally ignore titles once I get into reading the question. — Nicol Bolas, Oct 31 '12 at 17:16
C++11 does not have inter-process functions. Boost::Interprocess might have some tools for you, to use for this problem. — Christopher, Oct 31 '12 at 17:17
You can use interprocessmutex from Boost ([enter link description here](https://www.boost.org/doc/libs/1_51_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.mutexes.mutexes_interprocess_mutexes)) — cleverman, Aug 20 '18 at 21:43
Not really what you're asking, but I'd think you'd want to use a bool instead of bit, since you probably want your compare-and-swap to depend just on that single and not 7 additional values. — interestedparty333, Sep 07 '19 at 17:36
Regarding bitsets: if atomic access is needed, it would probably be better to use std::vector with one bit per vector element. That way the individual bits could be accessed independently. — user2373145, Apr 13 '21 at 23:56
This question isn't very good because it contains too many sub-questions. — user2373145, Apr 14 '21 at 00:02

Nicol Bolas · Accepted Answer · 2021-04-14T02:05:35.123

19

The C++11 threading primitives (mutexes, atomics, etc) are threading primitives. The C++ standard doesn't reference processes, and most of these tools don't interoperate across processes.

The only mention of processes in the standard is in a non-normative notation that says that lock-free atomics are intended to be OK for IPC:

Operations that are lock-free should also be address-free. That is, atomic operations on the same memory location via two different addresses will communicate atomically. The implementation should not depend on any per-process state. This restriction enables communication by memory that is mapped into a process more than once and by memory that is shared between two processes.

Outside of this non-normative notation, the threading primitives are not intended to be a means of achieving inter-process communication. The behavior of such objects when placed in shared memory (aside from lock-free atomics as noted above) is undefined.

edited Apr 14 '21 at 02:05

answered Oct 31 '12 at 17:19

Nicol Bolas

449,505
63
781
982

14

[atomics.lockfree] includes "[Note: Operations that are lock-free should also be address-free. That is, atomic operations on the same memory location via two different addresses will communicate atomically. The implementation should not depend on any per-process state. This restriction enables communication by memory that is mapped into a process more than once and by memory that is shared between two processes. — end note ]" – Jeffrey Yasskin Jun 23 '14 at 15:49
1

Dear @Nicol Bolas, your answer is great, but I cannot find anywhere or any books writing about that so I still get some confuses about using C++11 mutex in shared memory. (as I know, pthread mutex is ok if using PTHREAD_PROCESS_SHARED) – Trung Nguyen Nov 07 '16 at 04:48
I really wish this was included at references such as http://en.cppreference.com/w/cpp/thread/mutex – sage Jun 15 '18 at 18:45
This answer is completely wrong. Jefrrey Yasskin already pointed at one reason why (std::atomic is in fact perfectly suited for shared memory IPC, and sometimes necessary) in his comment, but his comment is easy to gloss over. Mutexes, too, can be used for shared memory IPC, although it's true that the standard library mutexes are not meant for such usage. – user2373145 Apr 13 '21 at 23:53
2

And, just to be clear, the intent of the standard is clearly that atomics should be usable for IPC. Jeffrey's quote from the standard shows that this is true. That said, I don't know if that was in the standard already in C++11. – user2373145 Apr 14 '21 at 00:06
@user2373145: "*the intent of the standard is clearly that atomics should be usable for IPC*" If it is "clear", why is it in a notation instead of actual text? The standard never mentions processes in normative text. There's a notation about lock-free being "address free", but the C++ memory and object models themselves recognizes no such concept. – Nicol Bolas Apr 14 '21 at 02:01
It's "clear intent" because it's right there in the standard, no need to read between the lines. It's only *intent*, not officially required, because it's not normative. ISO C++ doesn't standardize a process model so they couldn't do that. But all mainstream implementations have lock-free `std::atomic` that does *Just Work* in shared memory. (C++20 `.wait()` and `.notify()` work thanks to an intentional choice to use shared instead of private futex calls on Linux, but the rest of std::atomic just trivially works because CPUs don't care whether it's a process or a thread.) – Peter Cordes Aug 24 '22 at 18:00

score 10 · Answer 2 · edited Mar 28 '17 at 10:50

You can use mutex inside of shared memory block, but the mutex must be declared as SHARED, therefore is not unusual using mutexes inside of share memory, u can make own class, it is very simple:

class Mutex {
private:
    void *_handle;
public:
    Mutex(void *shmMemMutex,  bool recursive =false, );
    virtual ~Mutex();

    void lock();
    void unlock();
    bool tryLock();
};

Mutex::Mutex(void *shmMemMutex, bool recursive)
{
    _handle = shmMemMutex;
    pthread_mutexattr_t attr;
    ::pthread_mutexattr_init(&attr);
    ::pthread_mutexattr_setpshared(&attr, PTHREAD_PROCESS_SHARED);
    ::pthread_mutexattr_settype(&attr, recursive ? PTHREAD_MUTEX_RECURSIVE_NP : PTHREAD_MUTEX_FAST_NP);

    if (::pthread_mutex_init((pthread_mutex_t*)_handle, &attr) == -1) {
        ::free(_handle);
        throw ThreadException("Unable to create mutex");
    }
}
Mutex::~Mutex()
{
    ::pthread_mutex_destroy((pthread_mutex_t*)_handle);
}
void Mutex::lock()
{
    if (::pthread_mutex_lock((pthread_mutex_t*)_handle) != 0) {
        throw ThreadException("Unable to lock mutex");
    }
}
void Mutex::unlock()
{
    if (::pthread_mutex_unlock((pthread_mutex_t*)_handle) != 0) {
        throw ThreadException("Unable to unlock mutex");
    }
}
bool Mutex::tryLock()
{
    int tryResult = ::pthread_mutex_trylock((pthread_mutex_t*)_handle);
    if (tryResult != 0) {
        if (EBUSY == tryResult) return false;
        throw ThreadException("Unable to lock mutex");
    }
    return true;
}

Does this really need a `virtual` destructor?? Also, probably best to use `pthread_mutex_t *handle` instead of casting it every time you use it. — Peter Cordes, Aug 24 '22 at 18:06

c++11 interprocess atomics and mutexes

2 Answers2

Linked