2

I'm running a Dataflow job using the gcloud command with the --parameters labels='{"my_label":"my_value"}' parameter in order to set custom labels (which I can see in the Job Info panel in the UI).

But unfortunately, I'm not able to retrieve this custom label in the Logging stack: I can see the classic Dataflow labels (dataflow.googleapis.com/job_name, dataflow.googleapis.com/job_id, etc.) and the Resource ones (step_id, project_id, region, etc.) but not my my_label custom label.

Is there something I'm missing or is that label propagation I'm expecting does not exist?

Thanks!

EDIT Here is the full command I'm using to run my Dataflow job:

gcloud dataflow flex-template run "my-dataflow-job" \
      --template-file-gcs-location "gs://$(GCP_BUCKET)/my-dataflow-job/templates/my-dataflow-job.json" \
      --region "$(GCP_REGION)" \
      --parameters tempLocation="gs://$(GCP_BUCKET)/my-dataflow-job/temp" \
      --parameters stagingLocation="gs://$(GCP_BUCKET)/my-dataflow-job/staging" \
      --parameters usePublicIps=false \
      --project "$(GCP_PROJECT)" \
      --worker-machine-type="$(DATAFLOW_WORKER_MACHINE_TYPE)" \
      --num-workers=1 \
      --max-workers=1 \
      --parameters labels='{"my_label":"my_value"}'
Adrien
  • 31
  • 3
  • Could you provide more details on exactly how you're using gcloud to invoke the Dataflow job? – robertwb Mar 18 '21 at 00:58
  • Thanks for your feedback. I edited my question with this information. – Adrien Mar 18 '21 at 09:06
  • Did you try to use the `additional-user-labels` parameter as it said in the documentation: https://cloud.google.com/sdk/gcloud/reference/dataflow/flex-template/run#--additional-user-labels – Alex Chermenin Mar 19 '21 at 12:53
  • Does it work when you're launching the job without using flex-templates? – robertwb Mar 19 '21 at 18:36
  • Same thing happening to me, but I'm not using flex-template. I can see my custom labels in the UI and also they appear in the Billing Exports. They don't appear in the logging though. @robertwb is this some known issue? – VictorGGl Sep 28 '21 at 14:39

0 Answers0