I run the following command on the master node of an AWS EMR cluster (release label: emr-6.1.0, hadoop distribution: Amazon 3.2.1) -
% spark-submit \
--deploy-mode cluster \
--master yarn \
main.py
It throws the following exception
21/10/01 16:25:47 INFO ApplicationMaster: Waiting for spark context initialization...
21/10/01 16:25:47 WARN SparkConf: The configuration key 'spark.scheduler.listenerbus.eventqueue.size' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.scheduler.listenerbus.eventqueue.capacity' instead.
21/10/01 16:25:47 ERROR ApplicationMaster: User application exited with status 1
21/10/01 16:25:47 INFO ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: User application exited with status 1)
21/10/01 16:25:47 ERROR ApplicationMaster: Uncaught exception:
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:302)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:500)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:264)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:890)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:889)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:889)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: org.apache.spark.SparkUserAppException: User application exited with 1
at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:111)
at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:728)
21/10/01 16:25:47 INFO ApplicationMaster: Deleting staging directory hdfs://<ip-address>.ec2.internal:8020/user/root/.sparkStaging/application_<id>_0003
21/10/01 16:25:48 INFO ShutdownHookManager: Shutdown hook called
Does anybody know how to fix or debug this?
EDIT:
Here is the content of main.py. I am only attaching (what I think is) the relevant part of the spark session initialization.
from datetime import datetime
import logging
import os
import sys
import numpy as np
import pandas as pd
from pyspark.sql import SparkSession
from pyspark.sql.functions import udf, struct
from pyspark.sql.types import StringType
logger = logging.getLogger('py4j')
def main():
logger.info('initializing spark')
spark = (SparkSession
.builder
.master('yarn')
.appName('MySpark')
.enableHiveSupport()
.getOrCreate())
spark_context = spark.sparkContext
spark_context.setLogLevel('ERROR')
.
.
.
if __name__ == '__main__':
main()
