why queue is showing incorrect data?

Question

I have used queue for passing urls to download, however the queue gets corrupted when received in the thread:

class ThreadedFetch(threading.Thread):
    """ docstring for ThreadedFetch
    """
    def __init__(self, queue, out_queue):
        super(ThreadedFetch, self).__init__()
        self.queue = queue
        self.outQueue = out_queue

    def run(self):
        items = self.queue.get()
        print items

def main():
    for i in xrange(len(args.urls)):
        t = ThreadedFetch(queue, out_queue)
        t.daemon = True
        t.start()

    # populate queue with data 
    for url, saveTo in urls_saveTo.iteritems():
        queue.put([url, saveTo, split])
    # wait on the queue until everything has been processed
    queue.join()

output resulting execution of run() when I execute the main is :

['http://www.nasa.gov/images/content/607800main_kepler1200_1600-1200.jpg', ['http://broadcast.lds.org/churchmusic/MP3/1/2/nowords/271.mp3', None, 3None, 3]
]

while expected is

['http://www.nasa.gov/images/content/607800main_kepler1200_1600-1200.jpg', None, 3]
['http://broadcast.lds.org/churchmusic/MP3/1/2/nowords/271.mp3', None, 3]

You are asking why queue has incorrect data without showing what queue is... Maybe try making a runable example that reproduces the problem. — luk32, Jul 12 '14 at 13:34

score 1 · Answer 1 · answered Jul 12 '14 at 14:05

All of the threads print their data at once and the results are interleaved. If you want threads to display data in production code, you need some way for them to cooperate when writing. One option is a global lock that all screen writers use, another is the logging module.

hughdbrown · Accepted Answer · 2014-07-12T15:48:43.730

0

Populate your queue before you start your threads. Add a lock for I/O (for the reason @tdelaney says -- the threads are interleaving writes to stdout and the results appear broken). And modify your run method to this:

lock = threading.Lock()

def run(self):
    while True:
        try:
            items = self.queue.get_nowait()
            with lock:
               print items
        except Queue.Empty:
            break
        except Exception as err:
            pass
        self.queue.task_done()

You might also find that it is easier to do this with concurrent.futures. There is a solid example of using a method that returns a value that is called in a thread pool.

edited Jul 12 '14 at 15:48

answered Jul 12 '14 at 13:51

hughdbrown

47,733
20
85
108

in that example I didn't not understood this: `future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}` – Ciasto piekarz Jul 12 '14 at 14:09
It creates a dictionary, in python 2.7 and higher, that has key of the return value from `executor.submit` and value of url. `for url in URLS` iterates over the urls in URL. – hughdbrown Jul 12 '14 at 14:24

why queue is showing incorrect data?

2 Answers2