0

Could some one please help to rectify this error It is showing the below error when I am trying to update the data

py4j.protocol.Py4JJavaError: An error occurred while calling o84.save. : org.apache.hudi.exception.HoodieException: hoodie only support org.apache.spark.serializer.KryoSerializer as spark.serializer

Tried sc.setSystemProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer") as well.

The error is in Google data proc

Gopinath Thatha
  • 1
  • 1

1 Answers1

-1

When creating the Spark Session, you can specify the config. Example

spark = SparkSession.builder.config('spark.serializer', 'org.apache.spark.serializer.KryoSerializer').config('spark.sql.hive.convertMetastoreParquet', 'false').getOrCreate()
Arun A K
  • 2,205
  • 2
  • 27
  • 45