Get all items from thread Queue

Question

I have one thread that writes results into a Queue.

In another thread (GUI), I periodically (in the IDLE event) check if there are results in the queue, like this:

def queue_get_all(q):
    items = []
    while 1:
        try:
            items.append(q.get_nowait())
        except Empty, e:
            break
    return items

Is this a good way to do it ?

Edit:

I'm asking because sometimes the waiting thread gets stuck for a few seconds without taking out new results.

The "stuck" problem turned out to be because I was doing the processing in the idle event handler, without making sure that such events are actually generated by calling wx.WakeUpIdle, as is recommended.

How about a blocking `get` followed by a `get_nowait` loop? – SnzFor16Min Aug 23 '23 at 05:34 — SnzFor16Min, Aug 23 '23 at 05:34

Brian · Answer 1 · 2008-10-01T12:35:56.050

If you're always pulling all available items off the queue, is there any real point in using a queue, rather than just a list with a lock? ie:

from __future__ import with_statement
import threading

class ItemStore(object):
    def __init__(self):
        self.lock = threading.Lock()
        self.items = []

    def add(self, item):
        with self.lock:
            self.items.append(item)

    def getAll(self):
        with self.lock:
            items, self.items = self.items, []
        return items

If you're also pulling them individually, and making use of the blocking behaviour for empty queues, then you should use Queue, but your use case looks much simpler, and might be better served by the above approach.

[Edit2] I'd missed the fact that you're polling the queue from an idle loop, and from your update, I see that the problem isn't related to contention, so the below approach isn't really relevant to your problem. I've left it in in case anyone finds a blocking variant of this useful:

For cases where you do want to block until you get at least one result, you can modify the above code to wait for data to become available through being signalled by the producer thread. Eg.

class ItemStore(object):
    def __init__(self):
        self.cond = threading.Condition()
        self.items = []

    def add(self, item):
        with self.cond:
            self.items.append(item)
            self.cond.notify() # Wake 1 thread waiting on cond (if any)

    def getAll(self, blocking=False):
        with self.cond:
            # If blocking is true, always return at least 1 item
            while blocking and len(self.items) == 0:
                self.cond.wait()
            items, self.items = self.items, []
        return items

score 19 · Answer 2 · answered Sep 10 '14 at 14:36

19

I think the easiest way of getting all items out of the queue is the following:

def get_all_queue_result(queue):

    result_list = []
    while not queue.empty():
        result_list.append(queue.get())

    return result_list

answered Sep 10 '14 at 14:36

Gab

5,604
6
36
52

4

This could still give you an exception at some point. From the docs: If empty() returns True it doesn’t guarantee that a subsequent call to put() will not block. Similarly, if empty() returns False it doesn’t guarantee that a subsequent call to get() will not block. https://docs.python.org/2/library/queue.html – fantabolous Dec 04 '15 at 15:25
@kaptan the problem is if it does NOT block, and another thread tries to do something with it at the same time, resulting in exceptions or other bad behaviour. – fantabolous Apr 26 '18 at 09:47
@kaptan specifically in the example above another thread might empty out the queue between this thread's empty() and get() calls, in which case get() could raise an empty queue exception – fantabolous Apr 26 '18 at 13:24

score 9 · Accepted Answer · edited Dec 16 '20 at 10:44

9

I'd be very surprised if the get_nowait() call caused the pause by not returning if the list was empty.

Could it be that you're posting a large number of (maybe big?) items between checks which means the receiving thread has a large amount of data to pull out of the Queue? You could try limiting the number you retrieve in one batch:

def queue_get_all(q):
    items = []
    maxItemsToRetrieve = 10
    for numOfItemsRetrieved in range(0, maxItemsToRetrieve):
        try:
            if numOfItemsRetrieved == maxItemsToRetrieve:
                break
            items.append(q.get_nowait())
        except Empty, e:
            break
    return items

This would limit the receiving thread to pulling up to 10 items at a time.

edited Dec 16 '20 at 10:44

Silvan Rehm

23
1
7

answered Oct 01 '08 at 08:45

Jon Cage

36,366
38
137
215

It could cause long waits if he's calling it in a while 1: busyloop. If so, the consumer would not only be reducing the producer's timeslice, but would also spend a lot of time holding the Queue's lock, potentially livelocking the producer. (This is more likely if he has multiple consumer threads.) – Brian Oct 01 '08 at 09:33
Good point Brian. I got the impression that eliben wasn't calling this all _that_ frequenctly, but I could be wrong - eliben? – Jon Cage Oct 01 '08 at 10:04
Oops, you're right - I missed the part about calling it in an idle event. That wouldn't be enough to cause such problems. – Brian Oct 01 '08 at 12:06

score 5 · Answer 4 · answered Apr 04 '19 at 15:05

5

The simplest method is using a list comprehension:

items = [q.get() for _ in range(q.qsize())]

Use of the range function is generally frowned upon, but I haven't found a simpler method yet.

answered Apr 04 '19 at 15:05

EvertW

1,160
9
18

score 2 · Answer 5 · answered Feb 17 '15 at 21:05

2

If you're done writing to the queue, qsize should do the trick without needing to check the queue for each iteration.

responseList = []
for items in range(0, q.qsize()):
    responseList.append(q.get_nowait())

answered Feb 17 '15 at 21:05

Wraith404

92
4

4

`qsize` is similar to `empty` in that it's not guaranteed. From the docs: Return the approximate size of the queue. Note, qsize() > 0 doesn’t guarantee that a subsequent get() will not block, nor will qsize() < maxsize guarantee that put() will not block. https://docs.python.org/2/library/queue.html – fantabolous Dec 04 '15 at 15:27

score 1 · Answer 6 · answered Oct 01 '08 at 06:12

I see you are using get_nowait() which according to the documentation, "return[s] an item if one is immediately available, else raise the Empty exception"

Now, you happen to break out of the loop when an Empty exception is thrown. Thus, if there is no result immediately available in the queue, your function returns an empty items list.

Is there a reason why you are not using the get() method instead? It may be the case that the get_nowait() fails because the queue is servicing a put() request at that same moment.

I am not using get() because I don't want to block the GUI. It's OK if the call thinks theres nothing in the Q just when a new item is being put in, because I call it periodically — Eli Bendersky, Oct 01 '08 at 06:20

Get all items from thread Queue

6 Answers6

Linked