3

We have several Airflow DAGs in Cloud Composer that previously worked fine. The code for the DAGs and Operators has not been changed, but after a recent deployment, we now get this error from the DAGs:

Broken DAG: [...] Relationships can only be set between Operators; received PythonOperator

Again, that DAG has not been changed for several weeks and was previously working fine.

The PythonOperator there is the PythonOperator provided by Airflow, imported like this:

from airflow.operators.python_operator import PythonOperator

And used in the DAG like this:

OPERATOR_READ_HEADER = PythonOperator(
    task_id='read_header',
    params={'object_name': 'dependencies/MT940 Header.txt'},
    python_callable=read_object,
    provide_context=True
)

SENSOR_INPUT_FILE_PROVIDED >> OPERATOR_READ_HEADER

Why would this error have started occurring?

This is on Airflow 1.9.0-composer in Google Cloud Composer.

MHG
  • 1,410
  • 3
  • 19
  • 32
  • 1
    Looks similar to [this](https://stackoverflow.com/questions/54336037/airflow-error-importing-dag-using-plugin-relationships-can-only-be-set-between) – y2k-shubham Jan 24 '19 at 13:43

1 Answers1

2

There was a bug introduced in composer-1.4.2 release which we have fixed by now, try create a new Composer environment and that DAG error should go away. Meanwhile, we'll also apply that fix automatically on existing 1.4.2 environments automatically over the next few days.

Feng Lu
  • 691
  • 5
  • 6
  • If you don't want to wait on the auto fix rollout or create a new environment, you may apply the fix manually by running the following commands in your Composer GKE cluster: `kubectl get deployment airflow-scheduler -o yaml | sed -e 's/cloud_composer_service_2019-01-10-RC0/cloud_composer_service_2019-01-10-RC1/'|kubectl replace --force -f -` `kubectl get deployment airflow-worker -o yaml | sed -e 's/cloud_composer_service_2019-01-10-RC0/cloud_composer_service_2019-01-10-RC1/'|kubectl replace --force -f -` – Feng Lu Jan 25 '19 at 07:45