Dynamic Hosts and Parallel Tasks with Fabric library

Question

Is there a way to dynamically modify hosts in between multiple parallel tasks? This is what I have so far.

def init_hosts():
    env.hosts = [host1,host2,host3,host4]

@parallel
def task_1():
    if condition is False:
        env.hosts.remove(env.host)

@parallel
def task_2():
    run('uname -s')

Obviously I'm missing some env paramenters, but I only want task_2 to run on hosts that satisfy the condition in task_1. It appears task_2's host list is initialized on startup, because it's running on all hosts in the initial env.hosts list defined in init_hosts(). I also tried dynamically modifying and building roledefs, but had the same result.

Edit: Also, is there a way to setup a parallel execution queue such that multiple parallel tasks are executed in parallel rather than sequentially?

Edit: I managed to get my desired end result by having each task return output, and parsing the output to build a new host list to pass to execute():

def init_hosts():
    env.hosts = [host1,host2,host3,host4]

@parallel
def task_1():
    if condition is False:
        return False
    else:
        return True

@parallel
def task_2():
   run('uname -s')

def run_tests():
   results = execute(task_1)
   successful_hosts = [k for k in results.iterkeys() if results[k]]
   execute(test_2, hosts=successful_hosts)

This works, but it's gross for many reasons. Is there a better way?

score 3 · Answer 1 · answered May 15 '12 at 19:54

Parallel execution uses forks, and as such doesn't share (back) state changes. So changing a env variable in one task that's running in parallel, doesn't affect any other instance of that task running, nor does it change anything globally set before it was called.

If all task_1 is ever doing is a check, why not just incorporate this logic into task_2?

Dynamic Hosts and Parallel Tasks with Fabric library

1 Answers1