1

I am running AWS EMR cluster (emr-5.30.1, Spark 2.4.5, Livy 0.7.0). My service pass jobs to livy and livy execute "spark-submit" in cluster mode to submit jobs to yarn. spark master is 8 core, 16GB machine.

I see dead jobs when ~15-20 jobs are submitted at once to livy. livy logs shows "spark-submit exited with code 143", suggesting the process was killed by kernel or oom handler. I am not able to find more logs anywhere for killed processes. monitoring master node when these jobs are submitted shows ~100% cpu and using ~80% memory.

I tried using a 32GB master. This node can handle 15-20 parallel submitted jobs but fails when the parallel jobs goes above ~30.

To solve this problem I am thinking to put a queue in my service and then gradually passing (a job every 8-10 seconds) jobs to livy. I am reluctant in adding a queue as this needs to be a distributed one.

I have few questions here

  1. This seems a insufficient memory problem but I don’t see explicit logs. Can i conclude this to be a memory error?
  2. What other alternative solutions/approach can be used to fix this.
Manjeet Rulhania
  • 81
  • 1
  • 6
  • Two options... you go up in memory for your master... but a better option for me, use Kinesis data stream, you can add a huge number of shards depends on the load – ZEE Aug 15 '20 at 22:58
  • How livy habdle. the conxurrent job? Why not pass the job as the spark steps? – Lamanus Aug 16 '20 at 02:44
  • @Lamanus Livy (https://livy.apache.org/) exposes REST endpoints from EMR master node, where you can post spark jobs. Then livy will execute a spark-submit to actually start the job. AWS recently enabled concurrent steps execution, that's why I have been using Livy. But it's a good point that I can test submitting jobs via steps. thank you. – Manjeet Rulhania Aug 16 '20 at 19:51
  • I'm facing the same problem. I'm not using AWS EMR Steps API because that API has a pretty small rate limit. – theseadroid Dec 04 '20 at 16:36
  • Faced a similar issue with Spark and YARN. Basically the Yarn Manager memory reached its limits and started rejected jobs, not even putting them in a queue. This sucks big time because in practice you need to have a very big master node just to handle the large amount of jobs. Once they are distributed in the cluster, the master's memory will be available but you will pay for it. I submitted the jobs via Step Functions and I controlled from there the amount of jobs and the interval at which I submit. Hope this helps – bogdan.rusu Jan 27 '23 at 10:49

0 Answers0