How does Spark limit the usage of cpu cores and memory?Does it use cgroups? How about Yarn?
Asked
Active
Viewed 4,036 times
2
-
You are asking multiple questions. Please be specific and show some research before asking - thank you! – Yawar Murtaza Apr 05 '17 at 14:19
2 Answers
0
In standalone cluster Spark only manages application predefined resource configs with provided resource pool. Resource pool combined based on executors which added as salves to cluster. Yarn uses containers and resource limitation applies config of container which defines minimum and maximum core and memory allocation.
FaigB
- 2,271
- 1
- 13
- 22
0
In YARN NodeManager is monitoring spark executors' memory usage and killing them if they use above spark.executor.memory
In case of CPU, spark.executor.cores is the amount of concurrent tasks executor can run. More information on Spark Configuration Documentation
You can enable cgroups in yarn and limit CPU usage or YARN containers (spark executors).
Artur Sukhenko
- 602
- 3
- 12