I'm trying to write an unit test for a merge
operation on a DeltaTable
that I have. So far the approach has been to save the contents of a DataFrame
to a local file so that it can then be read via DeltaTable.forPath(...)
Seq(MyData(column_1 = "column_1_value", column_2 = "column_2_value").toDF()
.write.format("Delta").save(location)
val table = DeltaTable.forPath(location)
//execute code to be tested (includes a merge)
val resultOfMerge = table.toDF
//verify results of merge are correct
The problem is that takes too long, +20 seconds, sometimes +30 seconds. Is there a way I can construct a DeltaTable
with sample data and avoid the time consuming write and read operations to disk?