1

I'm trying to use the Cloud Composer to run my workflow. I wanted to use "GoogleCloudStorageToGoogleCloudStorageOperator" operator which is available from Apache Airflow v1.10, but not being supported in current cloud composer (it supports only Apache Airflow v1.9 for now (2019/01/16)).

Following the guidance of the Google's blog post, I added the operator to a cloud composer environment by myself, and it worked well until a few days ago.

However, for now, when I tried to create a new cloud composer env and to deploy the same DAG that worked well previously, I got a following error message on the Airflow Web UI. And DAG is failed.

Broken DAG: [/home/airflow/gcs/dags/xxx.py] Relationships can only be set between Operators; received GoogleCloudStorageToGoogleCloudStorageOperator

I couldn't understand why this error occurred even if I used the same code and followed the same procedure to deploy my DAG to the cloud composer.

I appreciate for those who give me any advice to solve this problem.

reddikih
  • 11
  • 1
  • 1
    FYI, Cloud Composer now supports Airflow v1.10.0. Check the release note here: https://cloud.google.com/composer/docs/release-notes – Ryan Yuan Jan 17 '19 at 01:25
  • Yeah, I saw the release note. But I couldn't find out how to select Airflow v1.10.0. Is there any document to explain how to select Airflow's version? – reddikih Jan 17 '19 at 04:45
  • Click into Composer, there is a "Enable Beta Features" checkbox. After you check it, you can select Airflow 1.10.0. – Ryan Yuan Jan 17 '19 at 05:50
  • Thanks for your advice. I could run Airflow v1.10.0 on my environment. And my DAG could be executed without above issue. Again, thanks a lot! – reddikih Jan 19 '19 at 07:26
  • I have been encountering this issue with our custom operators only on composer-1.4.2, is that the version you are on? It is really annoying because I cannot now recreate any environments, as this works on all previous composer/airflow image combinations. – HulaHoof Jan 23 '19 at 05:32
  • Yes, it was composer-1.4.2 and airflow-1.9.0 image. > @HulaHoof – reddikih Jan 25 '19 at 01:44
  • Thanks @reddikih, we got onto google support and they confirmed they are looking into this issue for composer-1.4.2. Our current workaround is to ignore AirflowPlugin and reference the operator files directly as modules. – HulaHoof Jan 25 '19 at 04:52
  • 1
    Possible duplicate of [Airflow error importing DAG using plugin - Relationships can only be set between Operators](https://stackoverflow.com/questions/54336037/airflow-error-importing-dag-using-plugin-relationships-can-only-be-set-between) – VictorGGl Feb 13 '19 at 14:31

2 Answers2

0

This was due to a bug in Composer 1.4.2 which was already fixed.

Airflow error importing DAG using plugin - Relationships can only be set between Operators

VictorGGl
  • 1,848
  • 10
  • 15
-5

Try out the DAG on Astronomer Cloud (http://astronomer.io/cloud), free 30 day trial.

Disclosure: I work at Astronomer.

ryw
  • 9,375
  • 5
  • 27
  • 34
  • Thank you for your suggest. The service is interesting. But I may use Airflow directly if I couldn't solve the problem. I don't wanna struggle with black box wrapping layer, even if it proved managed service. – reddikih Jan 17 '19 at 04:49
  • it's not black box - all our code is on Github, and our platform runs vanilla Airflow deployments with full customizability. – ryw Jan 23 '19 at 22:38
  • Oh, sorry. I don't check carefully. It's interesting. Let me check your service. thanks. – reddikih Jan 25 '19 at 01:47