2

I have a following piece of code.

My worker returns a list and I want a master list which is union of all the list.

from multiprocessing import Pool, Manager
manager = Manager()
another_shared_list = manager.list()

def worker2(number):
    return [x for x in xrange(number)]

numbers = [5,7,2,4]
pool1 = Pool(4)
another_shared_list.extend(pool1.map(worker2, numbers))
print another_shared_list

It prints

[[0, 1, 2, 3, 4], [0, 1, 2, 3, 4, 5, 6], [0, 1], [0, 1, 2, 3]]

As you might have probably guessed I want another_shared_list to be

[0,1,2,3,4,0,1,2,3,4,5,6,0,1,0,1,2,3]

How should I approach it?

edit: I know it seems like a flattening the list question not for multiprocessing. But my preference would be to avoid itertools. I want something such that another_shared_list directly gets the flattened list from the call pool1.map or something else!!

Ishan Bhatt
  • 9,287
  • 6
  • 23
  • 44
  • Your question is unrelated to the `multiprocessing` module; you already got your list, you just need to flatten it, as shown for example in [this question](https://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-lists-in-python). – alexis Jun 06 '17 at 10:46
  • @alexis Can I get the flattened list directly from the call in pool1.map? – Ishan Bhatt Jun 06 '17 at 10:47
  • Why not? It returns an iterator, so you should be able to directly unpack the results: `another_shared_list.extend(e for lst in pool1.map(worker2, numbers) for e in lst)`. – alexis Jun 06 '17 at 11:08

1 Answers1

6

Use itertools.chain:

itertools.chain(*another_shared_list)

Working example:

another_shared_list = [[0, 1, 2, 3, 4], [0, 1, 2, 3, 4, 5, 6], [0, 1], [0, 1, 2, 3]]
import itertools
list(itertools.chain(*another_shared_list))
[0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 5, 6, 0, 1, 0, 1, 2, 3]

Notice that chain returns an iterator and you have to consume it to a list if you need it.

Or as commented below:

itertools.chain.from_iterable(another_shared_list) #to avoid unpacking
Netwave
  • 40,134
  • 6
  • 50
  • 93