4

Was attempting to add a new DAG to our Google Cloud Composer instance - we have 32+ DAGs currently - and doing the usual things in https://cloud.google.com/composer/docs/how-to/using/managing-dags doesn't appear to be having any effect - we can't see these DAGs in the webserver/UI and I don't see that they are necessarily being loaded. I do see them being copied to the appropriate bucket in the logs but nothing beyond that.

I even tried setting a dummy environment variable to kick off a full restart of the Composer instance but to no avail.

Finally I've put together an entirely stripped down DAG and attempted to add it. Here is the DAG:

from airflow import models
from airflow.contrib.operators import kubernetes_pod_operator
from airflow.operators.python_operator import BranchPythonOperator
from airflow.operators.dummy_operator import DummyOperator

dag = models.Dag(
    dag_id="test-dag",
    schedule_interval=None,
    start_date=datetime(2020, 3, 9),
    max_active_runs=1,
    catchup=False,
)

task_test = DummyOperator(dag=dag, task_id="test-task")

Even this simple DAG isn't getting picked up so I'm wondering what I can try next. I looked through https://github.com/apache/airflow/blob/master/airflow/config_templates/default_airflow.cfg in an effort to see if perhaps there was anything I might tweak in here in terms of DagBag loading time limits, etc. but nothing jumps off. Totally stumped here.

Yeager
  • 87
  • 7

2 Answers2

2

Your example is not picked up by my environment either. However, I've tried with the following format and was picked up without issues:

from airflow import DAG
from datetime import datetime
from airflow.contrib.operators import kubernetes_pod_operator
from airflow.operators.python_operator import BranchPythonOperator
from airflow.operators.dummy_operator import DummyOperator

with DAG(
    "my-test-dag",
    schedule_interval=None,
    start_date=datetime(2020, 3, 9),
    max_active_runs=1,
    catchup=False) as dag:

        task_test = DummyOperator(dag=dag, task_id="my-test-task")
itroulli
  • 2,044
  • 1
  • 10
  • 21
  • Thanks for replying - I tried this but it still didn't work - one thing we picked out of the logs is it looks like there is some error happening copying the data from the DAGs folder: ``` type: "k8s_container" } severity: "ERROR" textPayload: "Copying gs://us-central1-composer-0f124775-bucket/dags/test.py... " timestamp: "2020-03-11T15:34:48.668270555Z" } ``` – Yeager Mar 11 '20 at 16:50
  • 1
    Actually that appeared to work - let me try it with the other dag – Yeager Mar 11 '20 at 16:57
  • @Yeager please accept the answer if the outcome is positive in the other DAG too. – itroulli Mar 11 '20 at 20:47
  • in the code above, you have used constructor method to create DAG. In such cases, if alias part " as dag" is missed, then also DAG is not picked up. I encountered same issue, and had figured that out the harder way, spending 2 days to this small piece. Adding a note for everyone else's reference. Also point out syntax for constructor method for creating DAGs with DAG( ) as : = () – Nitesh Jul 21 '22 at 14:06
1

We eventually figured out the issue:

dag = models.Dag(

should be

dag = models.DAG(

Yeager
  • 87
  • 7