0

I'm currently trying to write data using Iceberg to an external Hive table which is partitioned by partition_date column.

Before writing the data with Iceberg format, test table has 2 rows,

("2015-01-02", "S01233", "3-goods-purchased")
("2015-01-02", "S01234", "4-goods-purchased")

After writing data as below Code:

val input = Seq(("2015-01-02", "S01233", "5-goods-purchased"))
.toDF("partition_date", "order_id", "goods_purchased")

input.write
.format("iceberg")
.partitionBy("partition_date")
.option("path","s3://some-bucket-path/test")
.option(
"replaceWhere",
s"order_id in ('S01233')")
.mode("overwrite")
.saveAsTable("default.test")

Table test gets overwritten and only one row is shown in the output.

("2015-01-02", "S01233", "5-goods-purchased")

What I expected is this,

("2015-01-02", "S01233", "5-goods-purchased")
("2015-01-02", "S01234", "4-goods-purchased")

replaceWhere option provided as part of write statement, doesn't seem to work as expected.

Is there anything I'm missing? or does Iceberg support replaceWhere option which is working with delta format.?

Leroy Mikenzi
  • 792
  • 6
  • 22
  • 46
  • Have you tried replaceWhere while reading or on the input dataframe instead on the writer? – Prabhakar Reddy Sep 21 '22 at 06:10
  • @PrabhakarReddy As far as I understand, replaceWhere is like a filter. To answer your question, I'm not reading anything here. I have a input dataframe, all I'm doing is just writing to Iceberg table, but with an option of `replaceWhere` which you can see in my code snippet. – Leroy Mikenzi Sep 21 '22 at 07:00
  • for delta it is mentioned that you should not be using it with writer-overwrite as per DELTA_REPLACE_WHERE_IN_OVERWRITE in https://learn.microsoft.com/en-us/azure/databricks/error-messages/ and not sure for Iceberg – Prabhakar Reddy Sep 21 '22 at 08:35

1 Answers1

0

As far as I know, iceberg does not have this at present. You can use the following alternatives: data.writeTo("prod.db.table").overwrite(functions.col("order_id").isin("S01233"))

liliwei
  • 294
  • 1
  • 8