Python: multiprocessing Queue seemingly unaccessible

Question

I am confused why this code seems to hang and do nothing? As I try to experiment, it seems that I cannot get functions to access the queue from outside the function that added the item to the queue. Sorry, I am pretty novice. Do I need to pip install something? Update: Win10, Python 3.7.8, and this problem seems to apply to other variables besides queues.

This code works:

from multiprocessing import Queue, Process

q = Queue()

def main():
    q.put('a')
    q.put('b')
    print(q.get(block=True) + ' gotten')
    print(q.get(block=True) + ' gotten')

if __name__ == '__main__':
    main()

This code does not work:

from multiprocessing import Queue, Process

q = Queue()

def main():
    putter_process = Process(target=putter)
    putter_process.start()
    print(q.get(block=True) + ' gotten')
    print(q.get(block=True) + ' gotten')

def putter():
    q.put('a')
    q.put('b')

if __name__ == '__main__':
    main()

Similarly, this code from this question also hangs up on me:

import time
from multiprocessing import Process, Queue

sensor_data_queue = Queue()


def reads_sensor_data():
    # Suppose we add a sensor reading every second; this simulates that. It runs 10 iterations. You could modify this
    # to run forever, or until something tells it to quit.

    for iteration in range(10):
        sensor_data_queue.put(random.random())  # Generate a random number.
        time.sleep(1)  # Sleep for 1 second

    sensor_data_queue.put(None)  # None means we're done.


def analyze_sensor_data():
    while 1:
        data = sensor_data_queue.get(block=True)
        if data is None:
            break
        else:
            print(f'Analyzing {data}... Beep, beep, boop... {data * 100}')
    print('All done!')

def main():
    # Run the reader process in the background...
    reader_process = Process(target=reads_sensor_data)
    reader_process.start()
    try:
        analyze_sensor_data()
    finally:
        reader_process.join()
    
if __name__ == '__main__':
    main()    

EDIT: I am not sure this has to do with queues, because when I try to change a regular text variable it also doesn't work.

from multiprocessing import Queue, Process
import time

text = 'start'

def main():
    putter_process = Process(target=putter)
    putter_process.start()
    time.sleep(2)
    print(text)

def putter():
    text = 'edited'

if __name__ == '__main__':
    main()

The above outputs start whereas I was expecting it to output edited

Answer 1

Hmmm. I tested the example I gave you on Linux, and your second block code, which you say doesn't work, does work for me on Linux. I looked up the docs and indeed, Windows appears to be a special case:

Global variables

Bear in mind that if code run in a child process tries to access a global variable, then the value it sees (if any) may not be the same as the value in the parent process at the time that Process.start was called.

However, global variables which are just module level constants cause no problems.

Solution

I can't test this because I don't have your operating system available, but I would try passing the queue then to each of your functions. To follow your example:

from multiprocessing import Queue, Process

def main():
    q = Queue()

    putter_process = Process(target=putter, args=(q,))
    putter_process.start()
    print(q.get(block=True) + ' gotten')
    print(q.get(block=True) + ' gotten')

def putter(q):
    q.put('a')
    q.put('b')

if __name__ == '__main__':
    main()

Based on my interpretation of the docs, that should work on Windows, but I can't say I've tested it myself.

Note on tuples

In your other question, you asked why the trailing comma is relevant in args=. Let me show you:

>>> my_string = 'hello, world!'
>>> type(my_string)
<class 'str'>
>>> print(my_string)
hello, world!

As you can see, I've created a string variable. Now, let me create a tuple, which is like a list, but varies in some important ways.

>>> my_tuple = ('hello, world', 'hola mundo')
>>> type(my_tuple)
<class 'tuple'>
>>> print(my_tuple)
('hello, world', 'hola mundo')

As you can see, I've created a tuple with two elements, each element being a string. But what if I want to create a tuple with just one element? This will not work:

>>> my_not_a_tuple = ('hello, world')
>>> type(my_not_a_tuple)
<class 'str'>
>>> print(my_not_a_tuple)
hello, world

As you can see, I've only created a string. But I can create a tuple with one element by adding a comma, which clarifies that the parentheses are there to show that it's a tuple, not to control the order of operations:

>>> my_tuple = ('hello, world',)
>>> type(my_tuple)
<class 'tuple'>
>>> print(my_tuple)
('hello, world',)

That extra comma matters because you're needing to pass a tuple, not a string.