Is it possible to update the schema (change data-type of a column) of a non-empty table in Databricks (loaded by streaming Autoloader) without impacting the checkpoint folder ?
Is there any work-around to achieve this ?
Update: The data is read by autoloader like the following -
spark.readStream \
.format("cloudFiles") \
.option("cloudFiles.format", "parquet") \
.option("cloudFiles.schemaEvolutionMode", "rescue") \
.schema(schema_str)
Is it possible to implement the change without losing checkpoint history ?