0

Facing a weird issue, trying to execute a spark-sql(Spark2) job using oozie action but the behavior of execution is quite weird, at times it executes fine but sometimes it continues to be in "Running" state forever, on checking the logs got the below issue.

WARN  org.apache.spark.scheduler.cluster.YarnClusterScheduler`  - Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

The strange thing is that we have already provided sufficient resources, the same can be seen from spark environment variables as well and as well under the cluster resources(cluster has sufficient cores and RAM). 

<spark-opts>--executor-memory 10G --num-executors 7 --executor-cores 3 --driver-memory 8G --driver-cores 2</spark-opts>

With the same configuration sometimes it is executing fine as well. Are we missing something?

Amit Kumar
  • 1,544
  • 11
  • 23
Sumit Khurana
  • 159
  • 1
  • 10

1 Answers1

0

The issue was related to jar conflict,following are the suggestions to identify the same. a)Check the maven dependency tree to make sure there is no transitive dependency conflict. b)While spark job is running check the environment variables being used using Spark UI. c)Resolve the conflict and run a maven clean package.

Sumit Khurana
  • 159
  • 1
  • 10