Avoid over-utilizing Cluster

Question

Given a default DataProc cluster, are there any configurations to avoid overloading the job queue with too many tasks on the YARN side?

For instance, if a spike of job submits occur, is there a way to force the cluster to honor a concurrency, such that the entire spike of jobs doesn't deplete/crash the YARN master?

Will [this solution](https://stackoverflow.com/questions/49556305/gcp-dataproc-configure-yarn-fair-scheduler) solve your problem? — Igor Dvorzhak, Jul 18 '18 at 15:56
I knew it was possible! I'll give that a shot on my cluster-init.sh — deepelement, Jul 18 '18 at 20:18

deepelement · Answer 1 · 2018-07-26T13:42:06.667

2

As @igor-dvorzhak from Google mentioned, the resolution for this is https://stackoverflow.com/a/49693693/1195652

Ref: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.3/bk_system-admin-guide/content/setting_application_limits.html

edited Jul 26 '18 at 13:42

answered Jul 20 '18 at 13:55

deepelement

2,457
1
25
25

Avoid over-utilizing Cluster

1 Answers1