Why does the same spark program runs well on spark-shell but fails & ends with exit status 1

Question

I am trying to run a pyspark program by using spark-submit:

from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import *
from pyspark.sql import SparkSession

SNOWFLAKE_DATA_SOURCE = 'net.snowflake.spark.snowflake'

def get_records(spark:SparkSession):
    sfOptions = {
        'sfURL' : 'url',
        'sfAccount' : 'acntname',
        'sfUser' : 'username',
        'sfPassword' : 'pwd',
        'sfRole' : 'role',
        'sfDatabase' : 'dbname',
        'sfSchema' : 'schema',
        'sfWarehouse' : 'warehousename'
    }
    rec_df = spark.read.format(SNOWFLAKE_DATA_SOURCE).options(**sfOptions).options('query','select AREA from SCHEMA.TABLENAME limit 1').load()
    rec_df.show()

if __name__ == "__main__":
        spark = SparkSession.Builder.master('yarn').appName('Check_Con').getOrCreate()
        sc = SparkContext("yarn", "Simple App")
        spark = SQLContext(sc)
        spark_conf = SparkConf().setMaster('local').setAppName('CHECK')
        get_records(spark)

Spark-submit:

spark-submit --master yarn --deploy-mode cluster --keytab /home/devuser/devuser.keytab --principal devuser@PRINCIPAL.COM --num-executors 2 --executor-memory 1G --executor-cores 2 --driver-memory 1G --jars /home/spark-snowflake_2.11-2.8.0-spark_2.4.jar,/home/snowflake-jdbc-3.12.9.jar --files /home/devuser/conn_props/core-site_dummy.xml,/home/devuser/conn_props/hdfs-site_dummy.xml check_snow.py

When I submit the code, it ends with ERROR ApplicationMaster: User application exited with status 1 There is no error description or anything as such. The job just ends abruptly.

20/07/17 19:12:42 ERROR ApplicationMaster: User application exited with status 1
20/07/17 19:12:42 INFO ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: User application exited with status 1)
20/07/17 19:12:42 ERROR ApplicationMaster: Uncaught exception:
org.apache.spark.SparkException: Exception thrown in awaitResult:
        at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
        at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:345)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply$mcV$sp(ApplicationMaster.scala:260)

I tried to pass the core-site.xml & hdfs-site.xml in the spark-submit command as well. But no changes are working.

If I submit the same code on the terminal, on spark-shell, the code runs well. I am working in spark for the last one year faced many exceptions but never this one. I don't understand this phenomenon.

Edit 1: Some of the suggestions were to remove the unwanted part in the code and also remove setting of yarn inside the code since it is being set in spark-submit.

if __name__ == "__main__":
        spark = SparkSession.Builder.appName('Check_Con').getOrCreate()
        get_records(spark)

Even after that, I still see the same exception.

Could anyone let me know if I had made any mistake and how can I fix this problem ?

`spark-submit --master yarn --deploy-mode cluster --keytab /home/devuser/devuser.keytab --principal devuser@PRINCIPAL.COM --num-executors 2 --executor-memory 1G --executor-cores 2 --driver-memory 1G --jars /home/spark-snowflake_2.11-2.8.0-spark_2.4.jar,/home/snowflake-jdbc-3.12.9.jar --files /home/devuser/conn_props/core-site_dummy.xml,/home/devuser/conn_props/hdfs-site_dummy.xml check_snow.py --verbose` will see the any other trace we get. — sathya, Jul 17 '20 at 19:54

sathya · Answer 1 · 2020-07-18T17:23:18.777

0

exit code: 13 failure is due to the multiple spark,SparkContext, SparkConf Initializations and misconfigurations between local and yarn, so the YARN AppMaster is throwing an exit code 13.

please comment out the following section and your code should run,

 sc = SparkContext("yarn", "Simple App")
 spark = SQLContext(sc)
 spark_conf = SparkConf().setMaster('local').setAppName('CHECK')

since spark got initialized with SparkSession

spark = SparkSession.Builder.master('yarn').appName('Check_Con').getOrCreate()

Reference : Spark runs on Yarn cluster exitCode=13:

Edit 1: please run the below snippet with your aws credentials to connect to snowflake.

from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import *
from pyspark.sql import SparkSession

SNOWFLAKE_DATA_SOURCE = 'net.snowflake.spark.snowflake'

def get_records(spark):
    sfOptions = {
        'sfURL' : 'url',
        'sfAccount' : 'acntname',
        'sfUser' : 'username',
        'sfPassword' : 'pwd',
        'sfRole' : 'role',
        'sfDatabase' : 'dbname',
        'sfSchema' : 'schema',
        'sfWarehouse' : 'warehousename'
    }
    rec_df = spark.read.format(SNOWFLAKE_DATA_SOURCE).options(**sfOptions).option('query','select AREA from SCHEMA.TABLENAME limit 1').load()
    rec_df.show()

if __name__ == "__main__":
        spark = SparkSession.builder.master('yarn').appName('Check_Con').getOrCreate()
        get_records(spark)

edited Jul 18 '20 at 17:23

answered Jul 17 '20 at 20:13

sathya

1,982
1
20
37

I commented the code you mentioned also removed master('yarn') from the code since I am setting it on spark-submit. But I still see the same exception. I have posted the code after implementing your suggestions. – Metadata Jul 18 '20 at 11:16
I have added Edit 1 to my code implementing your suggestions. Any idea after that ? – Metadata Jul 18 '20 at 15:49
please run the code I have added in the answer(fixed some syntax errors in the code). – sathya Jul 18 '20 at 16:29
Ok, In the first scenario, I have removed the lines and ran the code. It failed again. After that 1. I added master('yarn') to the code but removed it from spark-submit command, ran it, the code failed. 2. Removed master('yarn') from code but added it to spark-submit command and ran it. failed again. Should I pass any `config` files or path of any environment variables or files in the spark-submit ? – Metadata Jul 18 '20 at 16:53
1. are you using kerberos? 2.from where you are running the driver program (EMR or spark cluster? – sathya Jul 18 '20 at 17:12
`Traceback (most recent call last): File "python-snowflake.py", line 24, in get_records(spark) File "python-snowflake.py", line 19, in get_records rec_df = spark.read.format(SNOWFLAKE_DATA_SOURCE).options(**sfOptions).options('query','select AREA from SCHEMA.TABLENAME limit 1').load() TypeError: options() takes exactly 1 argument (3 given)` – sathya Jul 18 '20 at 17:14
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/218109/discussion-between-smart-coder-and-metadata). – sathya Jul 18 '20 at 17:20
@Metadata , i found another issue and fixed it (added the updated code in the edit 1 section). issue is -?`spark.read.format(SNOWFLAKE_DATA_SOURCE).options(**sfOptions).options('query','select AREA from SCHEMA.TABLENAME limit 1').load()` should be `spark.read.format(SNOWFLAKE_DATA_SOURCE).options(**sfOptions).option('query','select AREA from SCHEMA.TABLENAME limit 1').load()`. I found a mismatch in `options().options()` inistead of `options().option()`. could you please run now? – sathya Jul 18 '20 at 17:25

Why does the same spark program runs well on spark-shell but fails & ends with exit status 1

1 Answers1