4

Disclaimer: I'm new to Python in general. I have a small experience with Go, where implementing a queue using a channel is really easy.

I want to know how can I implement a Queue with ProcessPoolExecutor in Python 3.

I want my N number of process to access a single queue so that I can just insert many jobs in the queue via the main thread, then the processes will just grab the jobs in the Queue.

Or if there is a better way to share a list/queue between multiple processes. (Job Queue/ Worker Pool maybe ?)

Thanks.

cankentcode
  • 460
  • 1
  • 5
  • 20

1 Answers1

3

concurrent.futures does this for you. The executor object implements a queue internally, so when you submit tasks, they get put into the queue and your worker threads or worker processes pick jobs up and run them.

It may feel as though it's "too easy", but that's what concurrent.futures is all about - abstracting away all the complexity of managing threadpools or processpools, job queues, etc. so you can trade a little overhead for a lot of saved time and effort.

Here's what it looks like:

import concurrent.futures

def send_email(from, to, subject, message):
    # magic to send an email

executor = concurrent.futures.ProcessPoolExecutor()
future = executor.submit(send_email, 'me@example.com', 'you@example.com', 'Hi!', 'Nice to meet you')

That one simple submit call takes your function and its arguments, wraps them into a work item, puts them into a queue and the process pool that was initialised when you created your executor will pick off the jobs and run them.

daveruinseverything
  • 4,775
  • 28
  • 40