Early stop of multiple RwLock::write waiting in Rust

Question

My Rust code uses RwLock to process data in multiple threads. Each thread fills a common storage while using the read lock (e.g. filling up a database, but my case is a bit different). Eventually, the common storage will fill up. I need to pause all processing, reallocate storage space (e.g. allocate more disk space from cloud), and continue.

// psudo-code
fn thread_worker(tasks) {
  let lock = rwlock.read().unwrap();
  for task in tasks {
    // please ignore out_of_space check race condition
    // it's here just to explain the question 
    if out_of_space {
      drop(lock);
      let write_lock = rwlock.write().unwrap();
      // get more storage
      drop(write_lock);
      lock = rwlock.read().unwrap();
    }
    // handle task WITHOUT getting a read lock on every pass
    // getting a lock is far costlier than actual task processing
  }
  drop(lock);
}

Since all threads will quickly hit out of space at about the same time, they can all release the read lock, and get a write. The first thread that gets the write lock will fix the storage issue. But now I have a possible temporary deadlock situation - all other threads are also waiting for the write lock even though they no longer need it.

So it is possible for this situation to happen: given 3 threads all waiting for write, the 1st gets the write, fixes the issue, releases write, and waits for read. The 2nd enters write but quickly skips because issue already fixed and releases. The 1st and 2nd threads will enter read and continue processing, but the 3rd is still waiting for write and will wait for it for a very long time until the first two either run out of space or finish all their work.

Given all threads waiting for write, how can I "abort" all other thread's waits from the first thread after it finishes its work, but before it releases the write lock it already got?

I saw there is a poisoning feature, but that was designed for panics, and reusing it for production seems wrong and tricky to get done correctly. Also Rust devs are thinking of removing it.

P.S. Each loop iteration is essentially a data[index] = value assignment, where data is a giant memmap shared by many threads. The index is slowly growing in all threads, so eventually all threads run out of memmap size. When that happens, memmap is destroyed, file reallocated, and a new memmap is created. Thus, it is impossible to get a read lock on every loop iteration.

Answer 1

First note that depending on your target platform, your code may already work as is. For example for platforms where Rust threads rely on libpthread (e.g. Linux), and any platform where write locks take precedence over read locks.

If you want a cross-platform solution, all you need to do is switch to parking-lot which provides a fair implementation of a RwLock. In particular this means that readers trying to acquire the lock will block even if the lock is unlocked when there are writers waiting to acquire the lock.

Here's the sequence of events with a fair RwLock:

• Initially all threads are running and hold the read lock.
• First thread to run out of space releases the read lock and requests the write lock. Since the other threads still hold the read lock, the first thread is blocked.
• One after the other, the other threads run out of space, release the read lock and request the write lock.
• Once all the threads have released the read lock, one of them acquires the write lock.
• The thread that got the write lock allocates more memory, releases the write lock and requests the read lock. Since the other threads waiting on the write lock take precedence, the read request blocks.
• One after the other, the other threads acquire the write lock, notice that there is memory available, release the write lock and request the read lock.
• Once all the threads have acquired and released the write lock, they all acquire the read lock and proceed.

Note that there is a theoretical race condition that could keep one of the thread blocked once memory has been allocated if the other threads are able to proceed in the time it takes to release the read lock and request the write lock, e.g.:

drop(lock);
// Another thread gets the write lock, allocates memory and releases the lock
// All the other threads acquire and release the write lock
// At least one other thread acquires the read lock
let write_lock = rwlock.write().unwrap();

Given the time it takes to allocate memory alone, the probability of this happening in real life is so vanishingly small that it can be discounted.

Answer 2

Looking at your code, you could get away with an extra mutex:

// pseudo-code
fn thread_worker(tasks) {
  for task in tasks {
    if out_of_space {
      drop(lock);
      {
        let mutex = mutex.lock();      
        if out_of_space { // potentially updated by another worker
          let write_lock = rwlock.write();
          // get more storage
          ...
          // drop(write_lock); is automatic here
        }
        // drop(mutex); is automatic here
      }
      lock = rwlock.read();
    }

    // copy memory for the task
    ...
  }
}

The pattern used here is known as a Double-checked locking.

This solves the issue you have that after reallocation the next party is not gonna wait on rwlock.write forever, because it will not pass the out_of_space check inside the mutex critical section.

However this solution still has an issue that the first failed worker will wait for all the other workers to encounter out_of_space condition before it can proceed with reallocation, because it needs to wait for all read() locks to be dropped.

I'd recommend to refactor this code to move the reallocation logic out of this method.

Also try to avoid explicit drops if possible in favor or RAII which is usually a good practice.