1

Multiple times I've had an issue while updating a delta table in Databricks where overwriting the Schema fails the first time, but is then successful the second time. The solution to my problem was to simply run it again, and I'm unable to reproduce at this time. If it happens again I'll come back and post the exact error message, but it was in essence a Schema Mismatch error. Has anyone else had a similar problem?

overwriteSchema = True
DF.write \
.format("delta") \
.mode("overwrite") \
.option("overwriteSchema", overwriteSchema) \
.partitionBy(datefield) \
.saveAsTable(deltatable)
TonyRyan
  • 148
  • 1
  • 3
  • 8

1 Answers1

8

Key-value should be string, not Boolean. .option("overwriteSchema", "True") 

DF.write \
.format("delta") \
.mode("overwrite") \
.option("overwriteSchema", "True") \
.partitionBy(datefield) \
.saveAsTable(deltatable)
pvy4917
  • 1,768
  • 17
  • 23
geens_k
  • 91
  • 5
  • Good catch, that's what the [docs](https://docs.databricks.com/delta/delta-batch.html#change-column-type-or-name) point to as well. Can be very confusing and the Delta team should allow for booleans as well. – Hendrik F Mar 03 '22 at 21:59