0

I am trying to fix a bug where multiple threads are writing to a list in memory. Right now I have a thread lock and am occasionally running into problems that are related to the work being done in the threads.

I was hoping to simply make an hash of lists, one for each thread, and remove the thread lock. It seems like each thread could write to its own record without worrying about the others, but perhaps the fact that they are all using the same owning hash would itself be a problem.

Does anyone happen to know if this will work or not? If not, could I, for example, dynamically add a list to a package for each thread? Is that essentially the same thing?

I am far from a threading expert so any advice welcome.

Thanks,

chrismead
  • 2,163
  • 3
  • 24
  • 36

2 Answers2

1
import threading

def job(root_folder,my_list):
    for current,files,dirs in os.walk(root):
        my_list.extend(files)
        time.sleep(1)

my_lists = [[],[],[]]
my_folders = ["C:\\Windows","C:\\Users","C:\\Temp"]
my_threads = []
for folder,a_list in zip(my_folders,my_lists):
    my_threads.append(threading.Thread(target=job,args=(folder,a_list)
for thread in my_threads:
   thread.start()
for thread in my_threads:
   thread.join()

my_full_list = my_lists[0] + my_lists[1] + my_lists[2]

this way each thread just modifies its own list and at the end combines all the individual lists

also as pointed out this gives zero performance gain (actually probably slower than not threading it... ) you may get performance gains using multiprocessing instead ...

Joran Beasley
  • 110,522
  • 12
  • 160
  • 179
  • Hi, thanks for the answer. Looks to me like my_list being local to the thread will keep them separate. very cool. if you have a moment, could you edit to include a comment with any considerations regarding accessing each thread's local my_list from the main thread periodically. In other words, my threads will run for a long time gathering data that i want to make a copy of now and again. do i just put a thread lock inside a class that also contains job()? – chrismead Jun 27 '14 at 18:29
1

Don't use list. Use Queue (python2) or queue (python3). There is 3 kinds of queue: fifo, lifo and priority. The last one is for ordered data.

You may put data at one side (with thread):

q.put(data)

And get at the other side (maybe in a loop for, say, database):

while not q.empty:
    print q.get()

https://docs.python.org/2/library/queue.html

Luciano Pinheiro
  • 85
  • 4