The Delta Lake documentation states that to use automatic schema evolution, one has to stick with updateAll() and insertAll() methods when using Delta merge i.e. can't use sub-expressions/conditions to change column values selectively.
https://docs.delta.io/latest/delta-update.html#automatic-schema-evolution
This is fine until I see the need to run SCD type 2 merges on the same table.
For SCD type 2 I want to be able to 'retire' an existing / matching row and add another one with appropriate flags / dates.
It seems I'm going to have to choose which feature I can have in my etl process using Delta Lake.
Is there an alternative approach which isn't documented? Am I missing something obvious here?
