Threadsafe, ordered mapping/hash in C++?

Question

What's the best way to implement a threadsafe ordered(note1) mapping/hash in C++? Aka, a fast-lookup data structure (aka, not a queue) that different threads can iterate across, occasionally inserting or deleting elements without interfering with the other threads' activity?

• std::map is not threadsafe, and its operations are non-atomic - although only erasure invalidates iterators

• wrapping every function in the whole map class doesn't fix the problem - you can have loose iterators out there pointing to a node that gets erased by another thread. It should either lock and prevent the deletion until the current thread is the only one referencing it, or use the UNIX-filesystem-style "dangling but still valid reference upon deletion" approach

• tbb::concurrent_hash_map is designed to be threadsafe, but its iterators still invalidate upon deletion of their key. Even if the Value is a smart pointer the data won't get preserved, as the reference in the iterator gets lost.
• One can use tbb:concurrent_hash_map by iterating through keys instead of iterators (one can look up a key as O(1) instead of O(log N) like std::map), but since it's a hash, it lacks order-dependent functions like upper_bound, lower_bound, etc. Critically, if a key is left dangling by an erase in another thread, there's no obvious way to tell the code to jump back to the nearest element to that key in the hash.
• std::unordered_map has the possibility that one could try to jury-rig finding the nearest element if your key gets deleted via the bucket access functions. But std::unordered_map is not considered threadsafe (although it could potentially be wrapped to be threadsafe). tbb::concurrent_hash_map is threadsafe (subject to the constraints above) but it doesn't allow sufficient bucket access to do this.
• std::map, accessed by keys instead of iterators, could find the next nearest element in the map if a key gets deleted by another thread. But every lookup would be O(log N) instead of O(1).
• Also, as mentioned, std::map is non-atomic. It would have to be wrapped. Using keys instead of iterators also requires wrapping the whole class because one has to change all of the functions that normally take or return iterators to take or return keys instead.
• Someone on one site that I read said that std::map doesn't work well in threaded environments regardless (compared to hashes) because threads don't play nicely with tree rebalancing, although I don't really know why that would be or whether it's even generally applicable/accurate.

What do you think would be best?

** Note1: When I wrote "ordered", I meant that only from the perspective of "has a reliable ordering that one can iterate across", not "the iteration must go across in the order of the keys". After I wrote that, I realized that a couple of my use cases actually do care about the order of iteration (most don't). But either way, I could probably jury-rig a proper ordering by working a linked list into the Value type. Just more slow ugliness and potential for problems/oversights...

** Note2: New thought. Too bad map isn't templated over its iterator type... how hard would it be to change the iterator std::map is built on? I had before been playing around with the idea of having iterators be reference counting (like std::shared_ptr) but I kept mistakenly thinking of implementing the reference counts with a secondary data structure inside the iterators themselves, and that always turned out way too ugly/slow/impractical. But it just occurred to me now that one could include the reference counting inside the Value of the map's key:value pair. That is, every Value would include A) a reference counter (default=0) that each interator increments whenever it reaches it (operator=, operator++, operator--, etc) and decrements whenever it leaves it; and B) an erasure flag (default=false) that the erase function sets. Whenever an iterator decrements the reference counter to zero, if the erasure flag is set then it would then actually erase it.

Seems to me that while it's a performance hit (the extra increments/decrements/checks), it's certainly nothing on the order of having to do a full map lookup every time you want to step through the structure. Can anyone think of a practical way to implement this?

Answer 1

For some reason you missed tbb::concurrent_unordered_map which is the hash table with thread-safe iteration support. It is based on split-ordered list algorithm where elements are connected in container-wide list structure in addition to the hash table and thus the iteration is straight-forward. But it does not fit completely well in your requirements since it does not support concurrent erasure.

This is a fundamental issue that in absence of a memory reclamation mechanism, it is hard to fuse both fast traversal and safe erasure properties in one concurrent data structure at the same time, you have to choose here: safety/consistency or speed.

With certain limitations and caution, you can do both traversing and deletion at the same time as described in this blog. Basically, it says that as long as you can interleave (mutually exclude) traversal and erasure, tbb::concurrent_hash_map can be used for concurrent traversal along with find&insert. The blog suggests additional optimization with double-check pattern. But it can be simplified to the following:

for(iterator = table.begin(); iterator != table.end(); iterator++ ) {
    accessor acc;
    // a key cannot be changed thus it is safe to read it without lock
    table.find( acc, iterator->first );   // now get the get the lock
    if( acc->second.market_for_deletion )
        table.erase( acc );               // erase only by accessor
}

It is essentially similar to your note 2 applied to concurrent_hash_map case since the most overhead goes not from the lookup (for neighbor elements a cache miss is less likely) but from synchronization with two locks (internal bucket's lock and element's accessor).

But if the speed of such a traversal approach is too slow or it is too hacky (relying on implementation details) for you, but you still desperately need to be able to delete elements of the concurrent hash table, consider using a RW-lock like tbb::spin_rw_mutex along with tbb::concurrent_unordered_map. You need to find an optimal place where the read locks can be acquired not too frequently to enable iteration, lookup & insertion without too much overhead and do the erase under the write lock not too often as well. Probably it requires additional scheme to mark and collect enough elements before they will really be deleted. E.g. here is a pseudo-code for such a hash table class:

class concurrent_hash_table_with_erase_and_traverse {
    tbb::concurrent_unordered_map my_map;
    tbb::spin_rw_mutex            my_lock; // acquired as writer for cleanup only
    tbb::atomic<size_t>           my_trash_count; // indicates # of items for erase

public:
    void init_thread_for_concurrent_ops() { my_lock.lock_read(); }
    void release_thread()                 { my_lock.unlock(); } // assuming reader lock
    mapped_type read(key_type k) {
        // assert: under read lock (thread is initialized)
        if(my_trash_count > threshold) {  // time to remove items
            my_lock.unlock(); // release reader
            // waiting all the threads to enter this container
            // TODO: re-implement with try_lock and checking the condition 
            my_lock.lock();   // acquire writer

            if(my_trash_count > threshold) { // double-check
                my_trash_count = 0;
                for( auto it = my_map.begin(); it != my_map.end(); ) {
                    auto _it = it++;
                    if( _it->is_marked_for_erase )
                        my_map.unsafe_erase( _it );
                }
            }
            my_lock.unlock();    // release writer
            my_lock.lock_read(); // acquire reader
        }
        return my_map[k]; // note: access is not protected like in concurrent_hash_map
    }
    void safe_erase(key_type k) {
        // assert: under read lock
        my_map[k].is_marked_for_erase = true;
        my_trash_count++;
    }
};

Answer 2

Okay, so this took ages... and so much work that I decided to make a github project out of it ;) But I finally have a threadsafe map class. Complete with a full test suite and lots of configurable options. Hopefully if anyone else needs this they'll make use of it!

https://github.com/KarenRei/safe-map#

Answer 3

Sample thread safe code that is probably not what you really wanted.

Warning untested code

template<typename kType, typename dType>
class Locked {
  std::mutex mut;
  std::map<kType, dType> theMap; // change types as required
public:
  const dType get(const kType& key) const {
    std::lock_guard<std::mutex> g(mut);
    auto it = theMap.find(key);
    if (it != theMap.end())
      return *it;
    // throw or return buggy dType
    return dType(-1); // or whatever
  }
  void set(const kType& key, const dType& data) {
    std::lock_guard<std::mutex> g(mut);
    theMap[key] = data;
  }
  void delete(const kType& key) {
    std::lock_guard<std::mutex> g(mut);
    auto it = theMap.find(key);
    if (it != theMap.end()) {
      theMap.erase(it);
      return;
    }
    // throw?
  }
}

I don't think this will work if dType is a pointer unless its a shared_ptr.

There can be only one.

It can be extended to have a reader counter so only set/delete blocks, set can be a read on the tbb map as it allows for thread safe insertion.

C++14 has std::shared_timed_mutex which makes the reads a bit easier on performance.

C++17 has std::shared_mutex which remove the time element from it.

Now there are loads of different lock-free, wait-free etc. implementations to get around the performance problems.

Depending on the actual load some spin_lock instead of the mutexes might help, up to a certain point.