How to enforce memory ordering with gcc on x86

Question

I want to share a data struct between threads (gcc, Linux, x86). Let's say I have the following code in thread A:

shared_struct->a = 1;
shared_struct->b = 1;
shared_struct->enable = true;

Thread B is a periodic task that checks that struct first for the enable flag.

I think that the compiler can reorder the writes in thread A, so thread B can see inconsistent data. I am familiar with memory barriers on ARM, but how do I ensure write ordering on x86? Is there a better way than volatile?

I just want to set a consistent state in the struct, "flush" everything to memory and set an enable flag at the end.

That is *not at all* what volatile is for. If there is an atomic flag, you should use C11 `` — Antti Haapala -- Слава Україні, Dec 21 '17 at 10:41
I know - it's a "byproduct" that volatile accesses are not reordered. — filo, Dec 21 '17 at 10:43
@AnttiHaapala What does atomic access have to do with instruction re-ordering? — Lundin, Dec 21 '17 at 10:44
@Lundin the `volatile` does guarantee write ordering and read ordering but it doesn't guarantee *consistency*. — Antti Haapala -- Слава Україні, Dec 21 '17 at 10:46
@AnttiHaapala The question is about execution ordering, not re-entrancy. — Lundin, Dec 21 '17 at 10:47
Why do you ask, and why don't you use `pthread_mutex_lock` ... — Basile Starynkevitch, Dec 21 '17 at 11:12
If you mark `a`, `b`,and `enable` in the structure as volatile then each of the assignment statements are guaranteed to be done in the order they appear. Each assignment is a full expression with a sequence point at the end. The standard says that volatile accesses can not be reordered across a sequence point. If they are marked volatile those assignment will be in order — Michael Petch, Dec 21 '17 at 12:45

Basile Starynkevitch · Accepted Answer · 2017-12-21T16:41:51.877

3

You really should use a mutex (since you mention Pthread), so add a pthread_mutex_lock mtx; field inside shared_struct (don't forget to initialize it with pthread_mutex_init) then

pthread_mutex_lock(&shared_struct->mtx);
shared_struct->a = 1;
shared_struct->b = 1;
shared_struct->enable = true;
pthread_mutex_unlock(&shared_struct->mtx);

and similarily in any other code accessing that shared data.

You might also look into atomic operations (but in your case, you'll better use a mutex like shown above).

Read some pthread tutorial.

Avoid race conditions and undefined behavior.

how do I ensure write ordering

You don't do that, unless you are implementing a thread library (and some parts of it should then be coded in assembler and use futex(7)), like nptl(7) implementation of pthreads(7) in GNU glibc (or musl-libc). You should use mutexes and you don't want to lose your time implementing a thread library (so use the existing ones).

Notice that most C standard libraries on Linux (including glibc & musl-libc) are free software, so you can study their source code (if you are curious to understand how Pthread mutexes are implemented, etc).

the compiler can reorder the writes

It is not mostly (and certainly not only) the compiler, but the hardware. Read about cache coherence. And the OS may also be involved (futex(2) sometimes called by pthread mutex routines).

edited Dec 21 '17 at 16:41

answered Dec 21 '17 at 10:55

Basile Starynkevitch

223,805
18
296
547

Thank you - I forgot about the obvious mutex. – filo Dec 21 '17 at 11:27
1

As a side note, `volatile` _does_ protect against hardware re-ordering. Neither the hardware nor the compiler is allowed to deviate from the C spec. A system where the hardware re-orders instructions, so that volatile variables aren't sequenced as specified by the programmer, is not conforming. If neither the hardware nor the compiler can guarantee that the C program is executed as required by the C standard's "abstract machine", then programs written in C cannot be used on that system. To lay the burden of handling memory barriers on the programmer does not make the system conforming. – Lundin Dec 21 '17 at 12:08
A `pthread_rwlock` is also appropriate here, and IMO fits the use-case better, allowing for multiple reader threads to not interfere with each other. – Andrew Henle Dec 21 '17 at 12:25
@Lundin *Neither the hardware nor the compiler is allowed to deviate from the C spec.* OK, that made me laugh, bringing up images of the "Compiler Police". – Andrew Henle Dec 21 '17 at 12:28
@AndrewHenle Well, they can legally deviate as much as they like :) But they can't claim compliance with ISO 9899:2011. – Lundin Dec 21 '17 at 12:43
4

It's not only the hardware that can reorder memory accesses, it's the entire **implementation**. On x86, for ordinary writes, it **is the compiler** that is reordering them, not the hardware, since x86 is TSO. – EOF Dec 21 '17 at 16:40

Peter Cordes · Answer 2 · 2017-12-21T19:13:59.703

If you only need to be able to set enable = true, then stdatomic.h with release / acquire ordering gives you exactly what you're asking for. (In x86 asm, normal stores/loads have release/acquire semantics, so yes blocking compile-time reordering is sort of sufficient. But the right way to do that is with atomic, not volatile.)

But if you want to be able to set enable = false to "lock out" readers again while you modify it, then you need a more complicated update pattern. Either re-invent a mutex manually with atomics (bad idea; use an standard library mutex instead of that), or do something that allows wait-free read-only access by multiple readers when no writer is in the middle of an update.

Either RCU a or seqlock could be good here.

For a seqlock, instead of an enable = true/false flag, you have a sequence number. A reader can detect a "torn" write by checking the sequence number before and then again after reading the other members. (But then all the member have to be atomic, using at least mo_relaxed, otherwise it's data race undefined behaviour just from reading them in C, even if you discard the value. You also need sufficient ordering on the loads that check the counter. e.g. probably acquire on the first one, then acquire on the shared_struct->b load to make sure the 2nd load of the sequence number is ordered after it. (acquire is only a one-way barrier: an acquire load after a relaxed load wouldn't give you what you need.)

RCU makes the readers always completely wait-free; they just dereference a pointer to the currently-valid struct. Updates are as simple as atomically replacing a pointer. Recycling old structs is where it gets complicated: you have to be sure every reader thread is done reading a block of memory before you reuse it.

Simply setting enable = false before changing the other struct members doesn't stop a reader from seeing enable == true and then seeing inconsistent / partially-updated values for the other members while a writer is modifying them. If you don't need to do that, but only ever release new objects for access by other threads, then the sequence you describe is fine with atomic_store_explicit(&foo->enable, true, memory_order_release).

The `memory_order_release` store should be paired with a `memory_order_acquire` load of the enable flag in the reader, right? — caf, Dec 22 '17 at 01:44
@caf: yes. acquire operations (and seq_cst) operations "synchronize-with" release (and seq_cst) operations. http://preshing.com/20120913/acquire-and-release-semantics/ has some nice details about how this actually works on some real machines, and what it does/doesn't give you as far as ordering. — Peter Cordes, Dec 22 '17 at 05:11

How to enforce memory ordering with gcc on x86

2 Answers2