spark streaming from kafka on spark operator(Kubernetes)

Asked Feb 09 '23 at 15:19

Active Feb 17 '23 at 10:09

Viewed 207 times

I have a spark structured streaming job in scala, reading from kafka and writing to S3 as hudi tables. Now I am trying to move this job to spark operator on EKS.

When I give the option in the yaml file.

spark.jars.packages: org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2,org.apache.hudi:hudi-spark3.1-bundle_2.12:0.11.1

But still I get the error at both the driver and executor

java.lang.ClassNotFoundException: org.apache.spark.sql.kafka010.KafkaBatchInputPartition .

How to add the package org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2, so it works.

Edit: Seems it is an existing issue fixed only in yet to be released version spark 3.4. Based on the suggestions here and here I had to bake all the jars (spark-sql-kafka-0-10_2.12-3.1.2 and its dependencies and also hudi jar) into the spark image. Then it worked.

edited Feb 17 '23 at 10:09

asked Feb 09 '23 at 15:19

haripriya rajendran

What version of Spark are you using? Make sure you use the same version for the spark-sql-kafka-0-10 JAR – OneCricketeer Feb 09 '23 at 15:57
The version on spark is also 3.1.2 – haripriya rajendran Feb 09 '23 at 17:53
Rather than in the YAML... In the code, can you put your `spark.jars.packages` there? Example - https://github.com/OneCricketeer/docker-stacks/blob/master/hadoop-spark/spark-notebooks/kafka-sql.ipynb – OneCricketeer Feb 09 '23 at 21:29

spark streaming from kafka on spark operator(Kubernetes)

0 Answers0