In the DataFrameWriter API we can set options to enable certain Delta features. For example:
spark.range(1).write.format("delta").option("mergeSchema", True).mode("append").save("/tmp/test_delta")
However, when building a MERGE command, I couldn't find a similar API:
from delta.tables import DeltaTable
dt = DeltaTable.forPath(spark, "/tmp/test_delta")
dt.alias("dt").merge(spark.range(1).alias("df"), condition="dt.id=df.id").whenNotMatchedInsertAll().option("mergeSchema", True).execute()
This fails with the following error:
AttributeError: 'DeltaMergeBuilder' object has no attribute 'option'
I know that for this specific option I can just set the configuration spark.databricks.delta.schema.autoMerge.enabled, but in my case I have multiple merge commands executing concurrently on the same Spark Session, and I only want to enable this option for some of them.
Is there any way to achieve this with Delta 2.0.2? (I'm currently limited by Spark 3.2, but if this feature is available in later version of Delta it might help me justify the upgrade).
