How to protect an object using a lock in Python?

Question

I've come accross functionality which required the following pattern:

from threading import Lock

the_list = []
the_list_lock = Lock()

and to use it:

with the_list_lock:
     the_list.append("New Element")

Unfortunately, this does not require me to acquire the lock, I could just access the object directly. I would like some protection against that (I'm only human.) Is there a standard way of doing this? My own approach is to create a HidingLock class that can be used like this:

the_list = HidingLock([])
with the_list as l:
    l.append("New Element")

But it feels so basic that either it should exist in the standard library or it's a very unconventional way to use locks.

Answer 1

I think the reason there's nothing in the standard library is because for it to be there it would need to make cast iron access guarantees. To provide anything less would give a false sense of security that could lead to just as many concurrency issues.

It's also nearly impossible to make these guarantees, without making substantial performance sacrifices. As such, it is left up to the user to consider how they will manage concurrency issues. This is in line with one of Python's the philosophies of "we're all consenting adults". That is, if you're writing a class I think it's reasonable that you should know which attributes you need to acquire a lock before accessing the attribute. Or, if you're really that concerned, write a wrapper/proxy class that controls all access to the underlying object.

With your example there are a number of ways in which the target object could accidentally escape. If the programmer isn't paying enough attention to the code they're writing/maintaining, then this HiddenLock could provide that false sense of security. For instance:

with the_lock as obj:
    pass
obj.func() # erroneous

with the_lock as obj:
    return obj.func() # possibly erroneous
# What if the return value of `func' contains a self reference?

with the_lock as obj:
    obj_copy = obj[:]

obj_copy[0] = 2 # erroneous?

This last one is particularly pernicious. Whether this code is thread safe depends not on the code within the with block, or even the code after the block. Instead, it is the implementation of the class of obj that will mean this code is thread safe or not. For instance, if obj is a list then this is safe as obj[:] creates a copy. However, if obj is a numpy.ndarray then obj[:] creates a view and so the operation is unsafe.

Actually, if the contents of obj were mutable then this could be unsafe as regardless (eg. obj_copy[0].mutate()).

Answer 2

My current solution (the one I talk about in the question) looks like this:

import threading

class HidingLock(object):
    def __init__(self, obj, lock=None):
        self.lock = lock or threading.RLock()
        self._obj = obj

    def __enter__(self):
        self.lock.acquire()
        return self._obj

    def __exit__(self, exc_type, exc_value, traceback):
        self.lock.release()

    def set(self, obj):
        with self:
            self._obj = obj

and here's how one would use it:

locked_list = HidingLock(["A"])
with locked_list as l:
    l.append("B")

Answer 3

What about creating a shared_list that has a list and implements the desired class methods using a threading.Lock:

import threading

class SharedList(object):
    def __init__(self, iterable=None):
        if iterable is not None:
            self.list = list(iterable)
        else:
            self.list = list()
        self.lock = threading.Lock()
        self.index = None

    def append(self, x):
        with self.lock:
            self.list.append(x)

    def __iter__(self):
        shared_iterator = SharedList()
        shared_iterator.list = self.list
        shared_iterator.lock = self.lock
        shared_iterator.index = 0
        return shared_iterator

    def next(self):
        with self.lock:
            if self.index < len(self.list):
                result = self.list[self.index]
                self.index += 1
            else:
                raise StopIteration
            return result

    # Override other methods

if __name__ == '__main__':
    shared_list = SharedList()
    for x in range(1, 4):
        shared_list.append(x)
    for entry in shared_list:
        print entry

Output

1
2
3

As Georg Shölly pointed out in the comments, this would require a lot of work to implement every method. However, if all you need is a list you can append to and then iterate over, this example provides the starting point.

Then you can just write

the_list = SharedList()
the_list.append("New Element")