26

I can't decide whether the following deque is thread-safe.
In short, I've created a class with a deque that displays its contents every 1 sec in a new thread (so it won't pause the main program while printing).
The deque is filled from the main thread, so basically there SHOULD be a chance of collision.
HOWEVER, the deque is filled using a class method, so essentially it is accessed from within the instance itself, therefore from the same thread.
Here's the simplified code:

import threading
import time
from collections import deque

class MyQueue(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)
        self.q = deque()
        self.start()

    def run(self):
        # pop out queue items every 1 sec
        # (please ignore empty deque for now)
        while True:
            print self.q.popleft()
            time.sleep(1)

    def add_to_q(self, val):
        # this function is called from outside
        self.q.append(val)

# main
# fill the queue with values
qu = MyQueue()
for i in range(1:100):
    qu.add_to_q(i)

So, although adding and removing items from queue take place inside the instance, is there a risk due to the adding function being called from outside the instance?

EDIT:
Since I need to modify items in my deque, I had to use Deque. What I do is: roatate() to the given item, pop it out, modify, push it back in and rotate() it back to its original position.
Unless I find a way of implementing modifying items in a Queue, I'll have to stick to Deque

user1102018
  • 4,369
  • 6
  • 26
  • 33
  • 2
    If you just need a thread-safe queue (I don't see you using any deque-specific features), you should probably use the [built-in thread-safe queue](http://docs.python.org/library/queue.html). –  Dec 18 '11 at 19:23
  • David, thanks, I added the loop. delnan, I excluded some features. The deque is being rotated every now and then so items can be popped out, modified, pushed back in and rotate back to their original position. I didn't find a way of doing that with Queue – user1102018 Dec 18 '11 at 19:29
  • 8
    It's impossible to write code to test if something is threadsafe and instead you have to read the documentation or study the source code. Often, if something's not threadsafe it'll still work nearly all the time in a threaded context and then explode weeks later. –  Dec 18 '11 at 20:25
  • Rather than having to `rotate()`, deque allows `insert()` at an arbitrary position, at least in python 2.7 / 3.7. – fantabolous Oct 03 '19 at 23:44

3 Answers3

31

Deque is thread-safe (http://docs.python.org/library/collections.html#deque-objects) for appends and pops from opposite sides. Beneath here, the docs only mention that append() and popleft() are thread-safe.

There is a thread-safe implementation of the Queue itself. So you should be using it unless you have some strange requirements.

bspar
  • 40
  • 1
  • 5
King
  • 1,170
  • 2
  • 16
  • 33
  • 13
    Not only is deque threadsafe, but it's performance is much higher than the Queue module. The valuable thing that the Queue module adds is that `get` can block. – amcnabb Apr 05 '12 at 18:33
  • Only thread safe appends and pops as I read it (for example, you can't necessarily use `len()` across threads can you?) – Martin Thompson Dec 13 '13 at 14:48
  • 1
    deque definitely throws if you iterate and push at the same time – Brannon Jan 11 '18 at 07:01
  • 1
    "... Deques support thread-safe, memory efficient appends and pops from either side of the deque with approximately the same O(1) performance in either direction..." (https://docs.python.org/3/library/collections.html#deque-objects), based on this, it's thread-safe appending or popping from either direction... – Lingchao Cao Aug 02 '19 at 05:14
27

Deque author here.

The MyQueue() class looks correct to me (at least with respect to the question of thread-safety).

The append() and popleft() methods are both atomic.

The code does need EAFP logic to handle the case where the input is empty:

def run(self):
    while True:
        try:
            print self.q.popleft()
        except IndexError:
            pass
        time.sleep(1)
Raymond Hettinger
  • 216,523
  • 63
  • 388
  • 485
  • 3
    And btw, *EAFP* is not the `European Association of Fish Pathologists`, but "Easier to Ask for Forgiveness than Permission’". – Ben Butterworth Oct 27 '22 at 13:38
10

For information there is a Python ticket referenced for deque thread-safety (https://bugs.python.org/issue15329).

Title "clarify which deque methods are thread-safe", bottom line is:

The deque's append(), appendleft(), pop(), popleft(), and len(d) operations are thread-safe in CPython. The append methods have a DECREF at the end (for cases where maxlen has been set), but this happens after all of the structure updates have been made and the invariants have been restored, so it is okay to treat these operations as atomic.

Anyway, if you are not 100% sure and you prefer reliability over performance, just put a like Lock for print self.q.popleft() and self.q.append(val) ;)

BadWolf
  • 355
  • 3
  • 5