When I'm testing about multiprocessing and threading with python, and I meet a odd situation

Question

I am using process pools(including 3 processes). In every process, I have set (created) some threads by using the thread classes to speed handle something.

At first, everything was OK. But when I wanted to change some variable in a thread, I met an odd situation.

For testing or to know what happens, I set a global variable COUNT to test. Honestly, I don't know this is safe or not. I just want to see, by using multiprocessing and threading can I change COUNT or not?

#!/usr/bin/env python
# encoding: utf-8

import os
import threading
from Queue import Queue
from multiprocessing import Process, Pool

# global variable
max_threads = 11
Stock_queue = Queue()
COUNT = 0

class WorkManager:
    def __init__(self, work_queue_size=1, thread_pool_size=1):
        self.work_queue = Queue()
        self.thread_pool = [] # initiate, no have a thread
        self.work_queue_size = work_queue_size
        self.thread_pool_size = thread_pool_size
        self.__init_work_queue()
        self.__init_thread_pool()

    def __init_work_queue(self):
        for i in xrange(self.work_queue_size):
            self.work_queue.put((func_test, Stock_queue.get()))

    def __init_thread_pool(self):
        for i in xrange(self.thread_pool_size):
            self.thread_pool.append(WorkThread(self.work_queue))

    def finish_all_threads(self):
        for i in xrange(self.thread_pool_size):
            if self.thread_pool[i].is_alive():
                self.thread_pool[i].join()

class WorkThread(threading.Thread):
    def __init__(self, work_queue):
        threading.Thread.__init__(self)
        self.work_queue = work_queue
        self.start()

    def run(self):
        while self.work_queue.qsize() > 0:
            try:
                func, args = self.work_queue.get(block=False)
                func(args)
            except Queue.Empty:
                print 'queue is empty....'

def handle(process_name):
    print process_name, 'is running...'
    work_manager = WorkManager(Stock_queue.qsize()/3, max_threads)
    work_manager.finish_all_threads()

def func_test(num):
    # use a global variable to test what happens
    global COUNT
    COUNT += num

def prepare():
    # prepare test queue, store 50 numbers in Stock_queue
    for i in xrange(50):
        Stock_queue.put(i)

def main():

    prepare()
    pools = Pool()
    # set 3 process
    for i in xrange(3):
        pools.apply_async(handle, args=('process_'+str(i),))
    pools.close()
    pools.join()

    global COUNT
    print 'COUNT: ', COUNT


if __name__ == '__main__':
    os.system('printf "\033c"')

    main()

Now, finally the result of COUNT is just 0.I am unable to understand whats happening here?

You seem to be confusing threads and processes here. – Daniel Roseman Sep 08 '15 at 08:17 — Daniel Roseman, Sep 08 '15 at 08:17

score 3 · Accepted Answer · answered Sep 08 '15 at 08:23

You print the COUNT var in the father process. Variables doesn't sync across processes because they doesn't share memory, that means that the variable stay 0 at the father process and is increased in the subprocesses

In the case of threading, threads share memory, that means that they share the variable count, so they should have COUNT as more than 0 but again they are at the subprocesses, and when they change the variable, it doesn't update it in other processes.

Thank you very much, this problem has confuses me for long time! By this, I finally understand why. — xina1i, Sep 08 '15 at 08:30

When I'm testing about multiprocessing and threading with python, and I meet a odd situation

1 Answers1