3

I am running Airflow (1.10.9) through Cloud Composer (1.11.1) on GCP. Whenever I update a DAG's code I can see the updated code refreshed in the Airflow GUI but for at least 10 minutes the DAG's tasks still run the old code.

A couple of questions:

  1. Why does this delay occur and can it be decreased?
  2. How can I know when the task's code has been updated to make sure nobody runs old code?
AYR
  • 1,139
  • 3
  • 14
  • 24
  • Did you delete the previous DAGs code or have any old version of the code in the bucket that content the /dag layer? (Something like /dag/dag or /dag/jar or /dag/...). I have seen other similar scenarios where this structure seems to cause problems when running fresh DAGs code since I believe the default behavior is to recourse into directories [1]. If this is the case, I would recommend using an .airflowignore file if you want to keep files in the `dags/` directory without risk of loading them accidentally. – davidmesalpz Jun 03 '21 at 07:32
  • [1]- https://github.com/apache/airflow/blob/a091c1fbf9dfec84953bbe2c938081a1b4fc0f63/airflow/utils/file.py#L93 – davidmesalpz Jun 03 '21 at 07:34
  • I'm bumping in the same issue, are there any updates? – GabrielBoehme Aug 09 '23 at 18:55

0 Answers0