Waiting for dependencies when running in parallel

Question

When running doit with -n option, the task doesn't seem to wait for the file_dep to be created by other sub-tasks. Here is a simple code that tries to show the issue:

def task_pexample():

    yield {
        "name":"test1",
        "actions": ["sleep 5", "touch tmp.txt"],
        "targets":["tmp.txt"]
    }

    yield {
        "name":"test2",
        "file_dep":["tmp.txt"],
        "targets":["tmp2.txt"],
        "actions": ["cp tmp.txt tmp2.txt"],
    }

doit -s pexample runs without a problem. However when running doit -n 2 -s pexample, it starts the second subtask right away, without waiting for the first one to complete. This then generates an error since the file doesn't exist.

My question is whether doit does or doesn't look for such dependencies in subtask when running in parallel?

score 1 · Accepted Answer · answered Mar 31 '20 at 21:02

1

Yes. doit looks for dependencies when executing in parallel.

The problem with your example is that you are using the -s/--single command line option. -s instructs doit to execute the specified task ignoring the dependencies.

-s is supposed to be used on special occasions while developing your task's code, not be used during standard execution.

answered Mar 31 '20 at 21:02

schettino72

2,990
1
28
27

indeed simply running `doit -n 2 pexample` actually works. Thank you! Can you explain what `-s` does exactly? – tlamadon Apr 01 '20 at 01:18

Waiting for dependencies when running in parallel

1 Answers1