Python - multiprocessing - queue: The reference of my queue changed even if it is the same object?

Question

I started using the multiprocessing not a long time ago and it is working on basic examples. Afterwards I tried to implement some kind of multi-sound input program and tried to canalize the input-flux via a queue to some processing module and that is currently badly failing. I will describe my problem in 3 points: folder-structure, process structure, what I tried.

Folder structure

• Root folder
  • Application
    • start_applicaton.py
    • input_cfg.ini
  • Core
    • core.py
    • gui.py
    • audio_recorder.py (Using sounddevice.InputStream)
    • x_recorder.py

Process structure When I start running my application, the gui is called and after I press the start-button the processes are created.

• Main Process
• audio_recorder_1 Process
• audio_recorder_ Process
• application Process

What I tried

core.py

from multiprocessing import Queue, Process
central_queue = Queue()
...
d = {}
d['output'] = central_queue
o = AudioRecorder('name', **d)

start_application.py

import core
def handle_queue_data():
    while True:
        print(str(core.central_queue.get()))
if __name__ == "__main__":
    Process(target=handle_queue_data, name="syncOutput").start()

audio_recorder.py

class AudioRecorder(object):
    def __init__(self, name, **d):
        ...
        self.output_queue = d['output']
    def run(self):
        queue = Queue()
        def callback(indata, frames, time, status):
            if status:
                print(status, flush=True)
            # Push the got data into the queue
            queue.put([indata.copy()])
        with sd.InputStream(samplerate=self.sample_rate, device=self.device_id, channels=self.channel_id, callback=callback):
            while True:
                self.output_queue.put(queue.get())

It was not working. After debugging, it seems like after the start from the core.py of the recorder, the reference of the queue had changed... FYI the debug information:

# in the audio_recorder.py object
centralized_queue = {Queue} <multiprocessing.queues.Queue object at 0x00000000086B3320>
 _buffer = {deque} deque([[array([[-0.01989746, -0.02053833],\n       [-0.01828003, -0.0196228 ],\n       [-0.00634766, -0.00686646],\n       ..., \n       [-0.01119995, -0.01144409],\n       [-0.00900269, -0.00982666],\n       [-0.00823975, -0.00888062]], dtype=float32)]])
 _close = {Finalize} <Finalize object, callback=_finalize_close, args=[deque([[array([[-0.01989746, -0.02053833],\n       [-0.01828003, -0.0196228 ],\n       [-0.00634766, -0.00686646],\n       ..., \n       [-0.01119995, -0.01144409],\n       [-0.00900269, -0.00982666],\n       [-0
 _closed = {bool} False
 _ignore_epipe = {bool} False
 _joincancelled = {bool} False
 _jointhread = {Finalize} <Finalize object, callback=_finalize_join, args=[<weakref at 0x00000000083A2638; to 'Thread' at 0x0000000004DF1B00>], exitprority=-5>
 _maxsize = {int} 2147483647
 _notempty = {Condition} <Condition(<unlocked _thread.lock object at 0x0000000004738198>, 0)>
 _opid = {int} 1320
 _reader = {PipeConnection} <multiprocessing.connection.PipeConnection object at 0x00000000086B34A8>
 _rlock = {Lock} <Lock(owner=None)>
 _sem = {BoundedSemaphore} <BoundedSemaphore(value=2147483645, maxvalue=2147483647)>
 _thread = {Thread} <Thread(QueueFeederThread, started daemon 9344)>
 _wlock = {NoneType} None
 _writer = {PipeConnection} <multiprocessing.connection.PipeConnection object at 0x00000000086B3518>

# in the handle_queue_data
centralized_queue = {Queue} <multiprocessing.queues.Queue object at 0x000000000479DA20>
 _buffer = {deque} deque([])
 _close = {NoneType} None
 _closed = {bool} False
 _ignore_epipe = {bool} False
 _joincancelled = {bool} False
 _jointhread = {NoneType} None
 _maxsize = {int} 2147483647
 _notempty = {Condition} <Condition(<unlocked _thread.lock object at 0x00000000058C8350>, 0)>
 _opid = {int} 7208
 _reader = {PipeConnection} <multiprocessing.connection.PipeConnection object at 0x000000000684C438>
 _rlock = {Lock} <Lock(owner=None)>
 _sem = {BoundedSemaphore} <BoundedSemaphore(value=2147483647, maxvalue=2147483647)>
 _thread = {NoneType} None
 _wlock = {NoneType} None
 _writer = {PipeConnection} <multiprocessing.connection.PipeConnection object at 0x00000000058DE6A0>

I also tried to use different things after, all unsuccessful, I don't manage to pass the data... Is it possible that queue is a mutable objects here? Or there is a bug in multiprocessing (very unlikely) or maybe the combination with sounddevice makes the queue unstable?

I'm sorry my description is pretty long...

I thank you in advance for your help!
Best regards,
Sebastian

Answer 1

I don't really have experience with multiprocessing, but it is my understanding that all objects in the module namespace of start_application.py are duplicated for each process. If I'm not mistaken, this includes the core module. Therefore, core.central_queue has a separate instance for each process. At least that seems to be the case on Windows and the Python docs recommend to "Explicitly pass resources to child processes" anyway.

You should use the if __name__ == '__main__': block to create a unique instance of Queue and a unique instance of AudioRecorder, too. Then you can pass those unique instances to your processes with the args argument of Process (as is shown in the above link).

Apart from that, I don't really know what you are trying to achieve. Do you want to process random chunks of the audio input with a random one of the available processes? Or do you want to provide the same audio input in its entirety to each of the processes?

In the latter case, you should have a separate queue per child process after all! The sd.InputStream should still be unique. And in your with statement, you should iterate over all child processes and put the current chunk of audio into each of the process queues separately.

PS: I just realized that you might want to start just a single additional process for some reason. In this case you should consider dropping the whole multiprocessing mess and just doing whatever you need to do in the with statement.

UPDATE:

If you want to use multiple audio devices (and therefore multiple PortAudio streams) at once, you still don't necessarily need multiprocessing. You can have a with statement with multiple context managers and do your processing in there.

Depending on what you want to achieve, you might have a single queue where all audio callbacks write to or one queue per callback.

If you have a good reason to use multiprocessing, it should also work fine if you start all audio streams in the main process and do your processing in a new child process.