Task nodes not scaling down with AWS EMR managed scaling

Asked Jul 28 '20 at 04:00

Active Jul 28 '20 at 08:03

Viewed 640 times

I have setup an EMR cluster with one step of Spark streaming and I have configured EMR managed scaling.

I submit adhoc spark jobs also on master with spark-submit. The cluster scales up when jobs are submitted but it doesn't scale down after the adhoc jobs are complete unless spark streaming job is killed. That means, cluster won't scale down till the cluster is idle (no job running on spark).

How to scale down cluster with EMR managed scaling with spark streaming running on the same cluster?

edited Jul 28 '20 at 08:03

asked Jul 28 '20 at 04:00

ashu82492

I don’t think you will have much control over managed scaling. By definition your scaling down shouldn’t be aggressive. But still it looks like a problem to me. – Snigdhajyoti Aug 02 '20 at 19:09
Are they task or core nodes? – thebluephantom Jun 22 '21 at 17:00
Tried both type of nodes. – ashu82492 Apr 17 '22 at 17:28

Task nodes not scaling down with AWS EMR managed scaling

0 Answers0