0

I want to get my DAG configuration from datastore, so in file that configures workflows I write following import:

from google.cloud import datastore

I got import error, so I went to PYPI packages tab for my envionmment and added google-cloud-datastore==1.7.1. It was updating for some long time, and finally given me error message

Composer Backend timed out. Currently running tasks are [stage: TP_DEPLOYMENT_UPDATING response_timestamp { seconds: 1544610512 nanos: 565000000 } ].

Now I don't see any dags inside airflow at all. This discussion suggests that only option is to create new environment, and hide dependencies in kubernetes pods, but how to create DAG from inside kubernetes pod then?

Bunyk
  • 7,635
  • 8
  • 47
  • 79

1 Answers1

0

Turns out it is a matter of pure luck. After creating new composer environment, I was able to add dependency google-cloud-datastore==1.7.1 successfully.

It took insane amount of time, but worked.

Additionally, to overcome error "This DAG isnt available in the webserver DagBag object", I used workaround described in this answer.

Bunyk
  • 7,635
  • 8
  • 47
  • 79
  • 1
    I have been raising this issue with the developers, hopefully the next release is more stable. But yes, it seems to be flaky and erratic like a dice roll. – Gaurav Taneja Dec 13 '18 at 10:08
  • @Bunyk, This could've happened due to some conflicts between all your installed libraries. Next time provide all the installed PyPI packages as well as your Composer and Python Environment version for reproducibility purposes – VictorGGl Jan 08 '19 at 08:52
  • @VictorGGl google-cloud-datastore was the first dependency added to that environment, if I remember that correctly – Bunyk Jan 08 '19 at 09:53
  • Then it could be the case that the environment that was throwing the errors was a bit older and had a different version than the new environment you created. See that Composer has been very active related to releases lately (https://cloud.google.com/composer/docs/release-notes). Otherwise there must have been something wrong with your environment for some reason; this shouldn't just be a matter of luck. – VictorGGl Jan 09 '19 at 08:41