4

I'm using Google Cloud composer for a few days now mainly to move data from MySQL to BigQuery and it was working fine.

At some point, it stopped working:

  • Running tasks run for a very long time and then fail

  • Tasks don't start

  • New dags have the comment This DAG seems to be existing only locally. The master scheduler doesn't seem to be aware of its existence

I've cleaned all the DAG runs and tasks instances using the Airflow web UI and still doesn't work.

Is there a way to restart the environment without losing the results of finished tasks? Is there any other way to get the Airflow running manually?

Alex Sergeenko
  • 642
  • 5
  • 22
Ary Jazz
  • 1,576
  • 1
  • 16
  • 25

3 Answers3

5

This post is over a year old, but it might help others. To answer your 2 questions:

  1. How to restart Composer environment Each time you are adding a Python package or an environment variable into Composer, this will restart the whole environment installing what needs to be added. So a workaround to restart Composer environment is to add a dummy environment variable. Follow these steps:

    • go to your composer instance in GCP
    • ENVIRONMENT VARIABLE tabs
    • Edit, add your environment variable and Submit

  2. How to run Airflow manually You can also access Airflow via Composer command line: https://cloud.google.com/composer/docs/gcloud-installation

I hope it helps!

Romain
  • 281
  • 4
  • 5
2

You cannot stop/restart a cloud composer environment. And till date the metadata DB is associated with the composer environment, hence deleting this environment would delete everything associated with that task.

kaxil
  • 17,706
  • 2
  • 59
  • 78
  • How can I point it to the GCS with my DAGs? Will this also keep the DAG Runs history? – Ary Jazz Aug 08 '18 at 13:37
  • @AryJazz I have updated my answer. You won't be able to keep the DAG runs history. https://cloud.google.com/composer/docs/concepts/cloud-storage - this is the link that shows what you can copy paste to the new environment or bucket associated with the new environment. – kaxil Aug 09 '18 at 08:41
1

Edit and Submit new value in Composer Environment by Get to your Composer Instance in GCP -> Edit/Add value in Environment Variable tab -> Submit.

This will restart the whole environment.

Thái Lê
  • 23
  • 6