0

I am using Apache Spark on EMR and would like to choose my resources in the form of machines

I would like to give each executor all of the resources on the machine that runs it.

I found here how to select the amount of resources each executor has, but I would like to just allow it to exhaust all the resources on the worker running it.

Is that possible in Spark (mainly EMR)

thebeancounter
  • 4,261
  • 8
  • 61
  • 109
  • Emr uses yarn which doesn’t consume all resources like standalone does. Use —num executor cores to the number of yarn contains on emr – Salim Jan 20 '20 at 21:29
  • Use spark Dynamic Resource Allocation feature - ref:https://stackoverflow.com/questions/40200389/how-to-execute-spark-programs-with-dynamic-resource-allocation – Remis Haroon - رامز Jan 21 '20 at 09:24
  • @Salim I do not want to use that, because I don't know the spec of the workers, I just want to let every executor to consume all resources on the worker, because my application benefits from more CPU power and has no point of distributing inside the same machine, it only created high overhead – thebeancounter Jan 21 '20 at 15:17
  • @RemisHaroon Can you provide a working example as an answer? – thebeancounter Jan 21 '20 at 15:17

0 Answers0