11

I have a queue (from the Queue module), and I want to get indexed access into it. (i.e., being able to ask for item number four in the queue, without removing it from the queue.)

I saw that a queue uses a deque internally, and deque has indexed access. The question is, how can I use the deque without (1) messing up the queue, (2) breaking thread-safety.

Ram Rachum
  • 84,019
  • 84
  • 236
  • 374
  • 1
    Why are you using a queue and not enqueuing and dequeueing things sequentially? Is this just a shared object between threads? – S.Lott Aug 18 '09 at 14:59
  • I'm not sure what you mean by "enqueuing and dequeueing things sequentially". I don't even understand your second question: Yes, the queue is shared between threads. – Ram Rachum Aug 18 '09 at 15:01
  • 1
    Queue's have things enqueued and dequeued in order. The idea of "in order" means that they are rarely accessed other than to append to one end and pop from the other end. Why are you breaking this default assumption around "queueing"? – S.Lott Aug 18 '09 at 15:17
  • 1
    @S.Lott, the queueing discipline in Queue.Queue is designed to be customizable by subclassing -- that's the reason `self.mutex`, `self.queue`, &c, have public names: exactly to indicate they allow advanced applications to (carefully;-) customize behavior (if Queue was meant to be a black box only, it would use a different naming scheme for its internals). – Alex Martelli Aug 18 '09 at 15:21
  • @Alex Martelli: Thanks, but, I'm still confused by the question -- is the queue used in order? If not, why is a Queue structure even being considered? Simply because of the mutex? Why not some other shared object with a mutex? Or a semaphore? Or some other kind of locking? Why a queue if the FIFO feature of the Queue isn't part of the problem? – S.Lott Aug 18 '09 at 17:13
  • @S.Lott: I'm using the queue mostly as a queue (i.e. FIFO), but in one case I want to access it like a list. Do you think I should do something else instead? – Ram Rachum Aug 18 '09 at 17:43
  • @cool-RR: if it's mostly FIFO, then a queue is good. The question (and your comments) were very confusing. A non-sequential (i.e., non-FIFO) queue isn't a proper queue. But if it's mostly accessed sequentially (i.e., FIFO), then it is a proper queue (perhaps with out-of-band or lookahead features.) – S.Lott Aug 18 '09 at 18:43

1 Answers1

14
import Queue

class IndexableQueue(Queue):
  def __getitem__(self, index):
    with self.mutex:
      return self.queue[index]

It's of course crucial to release the mutex whether the indexing succeeds or raises an IndexError, and I'm using a with statement for that. In older Python versions, try/finally would be used to the same effect.

Alex Martelli
  • 854,459
  • 170
  • 1,222
  • 1,395
  • 1
    So clean... I'm messing up with threads in Java now, and it's not that funny. – Bite code Aug 18 '09 at 14:43
  • great answer. Unfortunately, it isn't documented anywhere that Queue is suitable (and even meant, as you say) to be subclassed with access to its internal members. Their being "public" is the only clue. Not even in the source code of the Queue.py module does it say it explicitly, and this is a shame. Your snippet of code in this answer should be an example in the standard lib documentation of Queue, IMHO – Eli Bendersky Aug 18 '09 at 15:31
  • @eliben, you're right that the docs of Queue.py's internals is (and has long been) woefully inefficient, but at least it's getting a bit better... e.g. at the time we documented its subclassability (recipe 9.3 in the 2nd ed. of the Cookbook) that wasn't in the docs either, now at least a couple of useful subclasses are supplied...;-). – Alex Martelli Aug 18 '09 at 15:47
  • @Alex Martelli: I noticed that the mutex is a Lock, not an RLock. Do you think there's a good reason for this? If it was an RLock I could have assured no one was touching the queue during my entire session of accessing it. What do you think? – Ram Rachum Aug 18 '09 at 19:56
  • 2
    @cool-RR, the problem is that the locking in Queue is complicated -- there are Conditions based on that mutex, too. So calling different Queue methods while holding the mutex, which I assume is what you're after, is risky business -- it could deadlock even if it was an RLock! Why not just make a copy of the self.queue into your own local variable (while holding the mutex) and then releasing the mutex and accessing your local copy instead? (I do NOT recommend ALTERING the self.queue in any way except the "hallowed" ones, else you also need to signal the right conditions &c -- eep!-). – Alex Martelli Aug 19 '09 at 01:27