Dynamically add arguments to a Pool of workers

Question

I'm looking a way to dynamically add arguments to a Pool of workers during the same iteration. So, in the case some of these fails I'm able to promptly re-process it.

from numpy import random
from multiprocessing import Pool
from time import sleep

def foo(x):
    sleep(0.1)
    # 50% chance to have a fault
    return x, x if random.rand() > 0.5 else -1

random.seed(3)      # seed
pool = Pool(2)      # process
args = range(5)     # arguments to process

for i,(id,x) in enumerate(pool.imap(foo, args)):
    print i,x
    if x != -1:
        args.remove(id)

print args

The output is

0 0
1 1
2 2
3 3
4 -1
[4]

but I'd like it to be

0 0
1 1
2 2
3 3
4 -1
5, 4
[]

within the same iteration. I meant, I don't want to create a new map to the same Pool of workers once the iteration is complete. I'd like to push new argument directly, so that it fails at the first iteration I don't have to wait till the end before using the available process! I hope it make sense...

Update: My problem above simplified, the "foo" function takes about 20 minutes to complete and it's spread across 24 process which run concurrently. As soon one process fails I need to re-processing as soon as possible, as I don't want to wait 20 minutes when I have available resources.

Answer 1

As far as I know, you can't add a task to a currently-running Pool (without creating a race condition or undefined behavior, as you're currently seeing). Luckily, since all you need to do is retry any failed tasks until successful completion, you don't actually need to add anything to the Pool. All you need to do is modify the mapped function to behave the way you want.

def foo(x):
    sleep(0.1)
    # 50% chance to have a fault
    return x, x if random.rand() > 0.5 else -1

def successful_foo(x):
    '''Version of the foo(x) function that never fails.'''

    result = -1
    while result == -1:
        result = foo(x)
    return result

Now you can pool.imap(successful_foo, args), and be assured that every process will complete successfully (or run forever). If it's possible that it could run forever and you want an option to abort after some number of tries or some amount of time, just replace the while loop with an appropriate counter or timer.

---

Of course, in many non-demo cases, having a special return value to indicate failure is impractical. In that situation, I prefer to use a specialized Exception to handle the sorts of predictable failures you might encounter:

class FooError(BaseException):
    pass

def foo(x):
    sleep(0.1)
    # 50% chance to have a fault
    if random.rand() > 0.5:  # fault condition
        raise FooError('foo had an error!')
    return x, x

def successful_foo(x):
    '''Version of the foo(x) function that never fails.'''

    while True:
        try:
            return foo(x)
        except FooError as e:
            pass  # Log appropriately here; etc.

Answer 2

You can't. You want to modify a mutable list during iteration, this is known to not work. The output you get is due to the fact that when you remove an item form the list, the list reduces its length by 1 and all items after the one you removed are moved one index before. Which means that the item that followed is skipped.

The problem has nothing to do with multiprocessing per se, but with plain lists:

In [1]: def f(x):
   ...:     print(x)
   ...:     

In [2]: args = [0, 1, 2, 3, 4, 5]

In [3]: for i, x in enumerate(args):
   ...:     print(i, x)
   ...:     if x % 2 == 0:
   ...:         args.remove(x)
   ...:         
0 0
1 2
2 4

In [4]: args
Out[4]: [1, 3, 5]

note how the loop iterated only over the even values and never saw the odd values.

You want to keep track of which items to remove and only do that at the end of the loop:

to_be_removed = []
for i, (ident, x) in enumerate(pool.imap(foo, args)):
    print(i, x)
    if x != -1:
        to_be_removed.append(ident)

for ident in to_be_removed:
    args.remove(ident)

Or, probably more efficient, you can use a set and re-build the args list:

to_be_removed = set()
for i, (ident, x) in enumerate(pool.imap(foo, args)):
    print(i, x)
    if x != -1:
        to_be_removed.add(ident)

args = [el for el in args if el not in to_be_removed]

This takes linear time instead of, possibly, quadratic time of the previous solutions.

---

You can create also create a custom iterator that can do arbitrarily complex decisions as to which elements to produce for every iteration, however I'm not sure this would really work with multiprocessing since I believe it does not consume items one by one (otherwise it wouldn't be able to parallelize), and hence you wouldn't be able to provide any guarantee that the modifications will actually be seen when you expect them.

Moreover such a thing is only asking for bugs.