I recently upgraded to Spark 2.3.0. I had a existing spark job which used to run on spark 2.2.0. I am facing the Java Exception of AbstractMethodError
My simple code:
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils
if __name__ == "__main__":
print "Here it is!"
sc = SparkContext(appName="Tester")
ssc = StreamingContext(sc, 1)
This is working fine with Spark 2.2.0
With Spark spark 2.3.0, I am getting the following exception:
ssc = StreamingContext(sc, 1)
File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/streaming/context.py", line 61, in __init__
File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/streaming/context.py", line 65, in _initialize_context
File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py", line 1428, in __call__
File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py", line 320, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.streaming.api.java.JavaStreamingContext.
: java.lang.AbstractMethodError
at org.apache.spark.util.ListenerBus$class.$init$(ListenerBus.scala:35)
at org.apache.spark.streaming.scheduler.StreamingListenerBus.<init>(StreamingListenerBus.scala:30)
at org.apache.spark.streaming.scheduler.JobScheduler.<init>(JobScheduler.scala:57)
at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:184)
at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:76)
at org.apache.spark.streaming.api.java.JavaStreamingContext.<init>(JavaStreamingContext.scala:130)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:745)
I am using spark-streaming-kafka-0-8_2.11-2.3.0.jar for spark-submit command with —packages options.
I tried using spark-streaming-kafka-0-8-assembly_2.11-2.3.0.jar along with the --package and --jars options.
Python version: 2.7.5
I followed guide here: https://spark.apache.org/docs/2.3.0/streaming-kafka-0-8-integration.html
spark streaming kafka version 0-8 is deprecated in 2.3.0 but it is still there as per documentation.
My command looks like:
spark-submit --master spark://10.183.0.41:7077 --packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.3.0 Kafka_test.py
For sure, there something changed in scala-underlying code of Spark.
Does anyone have faced the same issue?
