0

I'd like to get notified when a Cloud Dataproc job finished. Unfortunately Cloud Dataproc does not seem to provide hooks or some ways to notify a job's lifecycle, I want to implement the mechanism in my own.

I'm planning to push to Pub/Sub when a job finished from my Spark application. But how do I know an information to identify the job inside a Spark application? If I could get User Labels of the job from the application underneath, I'm utilizing it by giving a unique label on submit, then include the label key and value in the Pub/Sub message.

yanana
  • 2,241
  • 2
  • 18
  • 28
  • 1
    If you're already computing a job id or user label before job submission, would it make sense to pass that in as an argument to your driver (via the args parameter on Dataproc's SparkJob: https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.jobs#SparkJob)? If using gcloud, arguments are specified as: gcloud dataproc jobs submit spark --cluster ... -- arg0 arg1 ... argN – Angus Davis Mar 25 '17 at 02:02
  • Thanks for your comment. It just works! – yanana Mar 26 '17 at 05:56

0 Answers0