0

I am using the bash operator in order to schedule a container to run on a compute instance. 

gcloud beta compute instances create-with-container airflow-vm --zone us-central1-a --container-image gcr.io/cloud-marketplace/google/image-which-does-calculations:1.0

The container automatically starts a process to do some calculations. How Can I detect that the process has completed it's calculations?

JY2k
  • 2,879
  • 1
  • 31
  • 60
  • 1
    I'd probably have my GCE instance write to GCS and use the GCS Sensor http://airflow.readthedocs.io/en/latest/_modules/airflow/contrib/sensors/gcs_sensor.html – Tim Swast May 21 '18 at 20:38
  • then pub/sub is also an option. But how do I handle a failure? – JY2k May 22 '18 at 06:48
  • You're right. Ideally there would be a GCE instance sensor. You could put a write to pub/sub or gcs in the shutdown script for the instance, but that leaves the door open to flakiness. Combined with a timeout on the sensor, it should get you close. – Tim Swast May 22 '18 at 14:10

0 Answers0