Python: Is it safe to get value from the same list iterator in multiple threads?

Question

I know that generators are not thread safe. When accessed by two threads at the same time, an error occurs:

ValueError: generator already executing

But iterators are different from generators, when used in a multiple threads, no exception is raised, but there will be race conditions. See this article for the comparision between generators and iterators.

In that articles, the author uses an example where a Counter is accessed by two threads at the same time, causing its state 'self.i' incorrectly updated:

class Counter:
    def __init__(self):
        self.i = 0

    def __iter__(self):
        return self

    def next(self):
        self.i += 1
        return self.i

so that when the threads stop, self.i will not equal the number of times it is updated by the two threads, but is smaller.

But will there be race conditions when the iterator is a list iterator? For example:

import threading

class Counter(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)
        self.num_seen = 0
        self.lock = threading.Lock()

    def run(self):
        global total
        for x in idata:
            self.num_seen += 1
        print('{0} {1}'.format(self.name, self.num_seen))
        with self.lock:
            total += self.num_seen

count = 10000000
idata = iter([i for i in xrange(count)])
total = 0
nthreads = 5
threads = [Counter() for n in xrange(nthreads)]
for t in threads:
    t.start()
for t in threads:
    t.join()
print('total ' + str(total))

Everytime total equals count, does this mean the iterator is thread safe?

I, too, noticed that [objects returned by `iter()` seem thread-safe](http://stackoverflow.com/questions/20043472/python-multithreading-why-generators-are-not-thread-safe-what-happens-when-it/20043642#comment29853210_20043642). Unfortunately, that's not a documented feature. — martineau, Nov 19 '13 at 19:36
BTW why might you need to use the same iterator in multiple threads? Why won't you use a [deque](http://docs.python.org/release/2.7.2/library/collections.html#collections.deque) which is explicitly thread-safe? — 9000, Nov 19 '13 at 20:33

score 0 · Answer 1 · answered Dec 08 '13 at 18:05

My guess is that what you're seeing depends on the implementation of Python that you're using. I'm assuming you're using CPython, and it looks like list iterators might be thread-safe in CPython. Note that I did not say they are thread-safe. There still may be edge cases or race conditions that your example does not trigger. Therefore, you should not depend on this because it is not a documented feature, and therefore it could change without warning.

If you need to use an iterator or generator concurrently in multiple threads then you should use a lock or mutex to guarantee thread safety. Or as suggested by @9000, use a deque which is documented to support "thread-safe ... appends and pops".

Python: Is it safe to get value from the same list iterator in multiple threads?

1 Answers1