0

I have two delta tables, one source and one destination and I batch-stream (using Trigger.AvailableNow()) from source to destination.

When the source table is overwritten then the next run fails because the destination table does not recognize the source table.

How to handle this situation? How to automatically truncate the destination table and propagate the new source table without interruption?

Using Spark 3.3.2.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
pgrandjean
  • 676
  • 1
  • 9
  • 19
  • Are you using Delta Live Table (I don't think so because you mention trigger). How often do you overwrite the source table? – Alex Ott May 20 '23 at 18:08
  • No, not using delta live tables. I use `spark.readStream`, `spark.writeStream` and `Trigger.AvailableNow` to run this regularly (but not continuously), like once-a-day. I use Delta and specify a checkpoint location so that data is written to destination incrementally. The source table is overwritten every month. – pgrandjean May 20 '23 at 18:23

0 Answers0