Hudi supports `update` operation?

Question

I have an exception when update record with spark sql for hudi as following.

update hudi.cow1 set price=1300 where id=2;
22/10/17 19:24:44 ERROR Executor: Exception in task 0.0 in stage 206.0 (TID 2442)
org.apache.avro.AvroRuntimeException: Not a valid schema field:
    at org.apache.avro.generic.GenericData$Record.get(GenericData.java:256)
    at org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal(HoodieAvroUtils.java:503)
    at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$write$11(HoodieSparkSqlWriter.scala:295)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:199)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
    at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
    at org.apache.spark.scheduler.Task.run(Task.scala:131)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1491)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
22/10/17 19:24:44 WARN TaskSetManager: Lost task 0.0 in stage 206.0 (TID 2442) (192.168.2.228 executor driver): org.apache.avro.AvroRuntimeException: Not a valid schema field:
    at org.apache.avro.generic.GenericData$Record.get(GenericData.java:256)
    at org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal(HoodieAvroUtils.java:503)
    at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$write$11(HoodieSparkSqlWriter.scala:295)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:199)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
    at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
    at org.apache.spark.scheduler.Task.run(Task.scala:131)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1491)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

I created the table as following.


  create table if not exists cow1 (
                      id int,
                      name string,
                      price double
                    ) using hudi
                    options (
                      type = 'cow',
                      primaryKey = 'id'
                    );

My env is: mac system; spark: spark-3.2.2-bin-hadoop3.2 hudi: hudi-spark3.2-bundle_2.12-0.12.0.jar I put the hudi jar in the jars dir under the spark home.

And I start spark sql with:

./spark-sql --jars ../../hudi-spark3.2-bundle_2.12-0.12.0.jar \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
--conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \
--conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog'

Is this a hudi bug?

score 0 · Answer 1 · answered Oct 25 '22 at 01:44

0

I asked in github. the preComineKey property is necessiary for update.

answered Oct 25 '22 at 01:44

Angle Tom

1,060
1
11
29

score 0 · Answer 2 · answered Jul 08 '23 at 09:31

0

Apache HUDI supports schema evolution.

For ignoring any column or adding new columns try these config

"hoodie.schema.on.read.enable": "true",
"hoodie.datasource.write.reconcile.schema": "true",

answered Jul 08 '23 at 09:31

ketankk

2,578
1
29
27

Hudi supports `update` operation?

2 Answers2