-1

dataflow list enter image description here I want to change instance group location from asia-northeast1-a(zonal) to asia-norhteast1(regional) but dataflow job automatically select single zone.

I wrote terraform code below as reference but maybe this is not a terraform issue because it became the same issue when I create dataflow manually.

resource "google_dataflow_job" "to_data_source" {
  count = 1

  name              = "to-data-source"
  template_gcs_path = "gs://dataflow-templates/latest/Cloud_PubSub_to_Avro"
  temp_gcs_location = "gs://foobar/foo"

  lifecycle {
    create_before_destroy = true
  }

  max_workers            = 5
  on_delete              = "drain"
  machine_type           = "n1-standard-1"
  additional_experiments = ["enable_stackdriver_agent_metrics"]
}
  • google_dataflow_job uses provider zone.

please help me. thank you for reading.

hiko1129
  • 13
  • 2
  • Your tags suggest you are creating your Dataflow pipeline using Terraform. Please can you edit your question to include the Terraform code that creates this. Ideally this should be in the form of a [mcve] to allow people to run the minimal amount of Terraform code and create the minimal amount of resources to reproduce your issue. – ydaetskcoR Jul 13 '20 at 10:54
  • @ydaetskcoR Thank you for your advice. – hiko1129 Jul 14 '20 at 01:57

1 Answers1

1

Dataflow today runs the worker in a single zone within a region. As part of the dataflow pipeline job submission there is an option to specify a region in which case Dataflow job will select a region through Autozone placement else one can specify the zone explicitly.

HTH

Jayadeep Jayaraman
  • 2,747
  • 3
  • 15
  • 26
  • Thank you for answering. Does dataflow have an automatic recovery function when it fails in a particular zone due to zone failure? For example, GCP recreates Dataflow in another zone. – hiko1129 Jul 14 '20 at 01:33
  • Currently, there is no service which is GA but you should keep watching this space as there are some capabilities that will get launched which will solve this problem. If the answer helped pls do accept the answer – Jayadeep Jayaraman Jul 14 '20 at 03:19