How to cache/target tasks with the same name in a Flow with prefect?

Question

I am trying to find a target pattern or cache config to differentiate between tasks with the same name in a flow.

As highlighted from the diagram above only one of the tasks gets cached and the other get overwritten. I tried using task-slug but to no avail.

@task(
    name="process_resource-{task_slug}",
    log_stdout=True,
    target=task_target
    )

Thanks in advance

score 4 · Accepted Answer · answered Jul 17 '20 at 20:35

4

It looks like you are attempting to format the task name instead of the target. (task names are not template-able strings).

The following snippet is probably what you want:

@task(name="process_resource", log_stdout=True, target="{task_name}-{task_slug}")

answered Jul 17 '20 at 20:35

Josh

293
1
5

@Josh-The target is a template with `task_full_name` ... so I thought I could pass the `name` (slug) and the target would pick up the new name. – Itay Livni Jul 17 '20 at 21:14

score 1 · Answer 2 · answered Jul 19 '20 at 15:38

After further research it looks like the documentation directly addresses changing task configuration on the fly - Without breaking target location templates.

@task
def number_task():
    return 42

with Flow("example-v3") as f:
    result = number_task(task_args={"name": "new-name"})

print(f.tasks) # {<Task: new-name>}

How to cache/target tasks with the same name in a Flow with prefect?

2 Answers2