0

Not able to run Spark job in yarn(cluster or client) mode through Livy, I am able to run spark job using Livy, but the jobs are running local mode and not visible on resource manager. I want to run these jobs in yarn-cluster mode.

I am using Hortonworks HDP 2.6.1 hadoop distribution. HDP2.6.1 has two versions of spark (2.1.1 and spark 1.6.3) and two versions of livy (1 and 2).

We have Ambari to view/change conf files.

We have made changes in below files:-

  1. /usr/hdp/2.6.1.0-129/etc/livy/conf.dist/conf/livy.conf
  2. /usr/hdp/current/livy-server/conf/livy-env.sh
  3. /usr/hdp/2.6.1.0-129/spark/conf/spark-defaults.conf

Added below properties in above files:-

  1. spark.master yarn-cluster
  2. spark.submit.deployMode cluster

========data=========

data = {
  'code': textwrap.dedent("""
    print(sc.applicationId)
    """)
}

========curl command=========

curl hdpmaster:8998/sessions/0/statements -X POST -H 'Content-Type: application/json' -d '{"code":"1 + 1"}'

Can someone please help, in which configuration file we need to make changes to run spark job in yarn mode?

Shantanu Sharma
  • 3,661
  • 1
  • 18
  • 39

1 Answers1

0

As you cannot set master parameter in your conf of the job he is may be taking what is set in the jar ie 'local'. Check your code may be you hard coded the value

zaki benz
  • 672
  • 7
  • 21