Do I need to use threading locks in this example?

Question

I append values to a nested list in one thread, and I copy the growing nested list with list comprehension in the main thread. Do I need to use threading locks in my example? I know that the list.append() method is thread-safe, but would I need to use a lock when copying the data with list comprehension?

If I do need to use locks, would it make any sense to use a lock in copy_data() but not in GrowList._add_to_list()?

import threading
import time

class GrowList(object):
    def __init__(self):
        self.data = []
        self.stop_event = threading.Event()
        self.add_to_list()

    def _add_to_list(self):
        sample = [1, 2, 3, 4]
        while not self.stop_event.is_set():
            self.data.append(sample)
            time.sleep(1.0)

    def add_to_list(self):
        self.t = threading.Thread(target=self._add_to_list)
        self.t.start()

    def stop(self):
        self.stop_event.set()
        self.t.join()


def copy_data(nested_list):
    return [row[:] for row in nested_list]

Depends on what your requirements are. You may end up not copying all elements of the list, since more elements may be added while the list comprehension runs is evaluated. For the current implementation of CPython, adding to the end of a list while iterating over it in forward direction usually works as expected. The worst case happens is unexpected results of the copy (e.g. skipped elements). — dhke, Dec 19 '16 at 19:48
My requirement for `copy_data` is to retrieve all of the elements in the nested list at the time that the function was called. It would be OK if items added during copying were not copied. With this requirement in mind, would it be acceptable to not use a lock? — jkr, Dec 19 '16 at 20:22
There's no guarantee to that. Python's guarantee is that you cannot "damage" the list during multithreaded access. As is, if items are added during the list comprehension, you will get some of the contents of the list. If that's okay, only the GIL is needed. Otherwise, use a lock. Note that the GIL is there, anyway. — dhke, Dec 19 '16 at 20:28

Gribouillis · Accepted Answer · 2016-12-19T23:41:30.633

I think you need a lock at least for iterating in one thread while appending in the other, and I think it wouldn't make sense to use a lock when copying the data when you don't use a lock when appending to the list. The point of using a lock is to claim some ownership on the list.

Another question that you may ask is what is the point of not using a lock ? You could keep your code clean by simply replacing self.data = [] by self.data = MySafeList() and by writing separately a small thread safe list class with a lock. It could be written very easily by using one of the numerous @synchronized decorators available here and there. For example the __iter__ method that allows list comprehension could be written as

@synchronized
def __iter__(self):
    return iter(list(list.__iter__(self)))

Do I need to use threading locks in this example?

1 Answers1

Linked