delta write with databricks job is not working as expected

Asked Jan 11 '22 at 14:46

Active Oct 05 '22 at 08:47

Viewed 619 times

I wrote the following code in python:

val input = spark.read.format("csv").option("header",true).load("input_path")


input.write.format("delta")
       .partitionBy("col1Name","col2Name")
       .mode("overwrite")
       .save("output_path")

input is read properly and we have the col1Name and col2Name.

The problem here is that the data is written in parquet and the _delta_log folder stays empty(no items) so when I try to read the delta data I get an an error : 'output_path` is not a Delta table How can I change the code to make the data properly data written in data with the _delta_log folder properly filled in?

I am using the following different conf in the databricks cluster:Apache Spark 2.4.5, Scala 2.11 and Apache Spark 3.1.2, Scala 2.12) but all of them give the same result.

Any idea how to fix this please?

Thanks a lot

edited Oct 05 '22 at 08:47

Alex Ott

80,552
8
87
132

asked Jan 11 '22 at 14:46

scalacode

1,096
1
16
38

delete the entire path and try it. if not try using `vaccume` and see if that works – Gaurang Shah Jan 11 '22 at 15:28
also, can you print the sample output. `df.take(10).show` – Gaurang Shah Jan 11 '22 at 15:29
@GaurangShah when I try to vacuum, I get Error in SQL statement: IllegalArgumentException: requirement failed: No state defined for this table. Is this really a Delta table? Refusing to garbage collect. – scalacode Jan 11 '22 at 15:50
yes input.take(10).show() works properly actually – scalacode Jan 11 '22 at 15:51
got it.. paste `show create table` for you table. I guess you table is not delta. files i think are.. – Gaurang Shah Jan 11 '22 at 16:07
yes that is the whole issue the spark.write;format(delta- doesn't create a delta table for me – scalacode Jan 11 '22 at 16:38
where are you creating table. could you just paste the output of the command i shared with you – Gaurang Shah Jan 11 '22 at 17:41

delta write with databricks job is not working as expected

0 Answers0