0

How do I run more than one spark streaming job in dataproc cluster? I created multiple queues using capacity-scheduler.xml but now I will need 12 queues if I want to run 12 different streaming - aggregate applications. Any idea?

zero323
  • 322,348
  • 103
  • 959
  • 935
passionate
  • 503
  • 2
  • 7
  • 25

1 Answers1

0

Dataproc 1.2 image enabled fair mode in capacity scheduler which should do what you want without overhead of queues [1] [2].

[1] https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_yarn_resource_mgt/content/flexible_scheduling_policies.html

[2] https://community.hortonworks.com/questions/19342/yarn-fair-sharing-ordering-policy-for-capacity-sch.html

tix
  • 2,138
  • 11
  • 18