Issue while running delta tables

Question

I am trying to run the delta tables

enter image description here

And i am getting error :

java.util.concurrent.ExecutionException: com.databricks.sql.transaction.tahoe.DeltaFileNotFoundException: abfss://tbueabfsbiscwe1@tbuestbiscwe1.dfs.core.windows.net/data_lakehouse/silver/object/lead_cdf/_delta_log/00000000000000000000.json: Unable to reconstruct state at version 3 as the transaction log has been truncated due to manual deletion or the log retention policy (delta.logRetentionDuration=30 days) and checkpoint retention policy (delta.checkpointRetentionDuration=2 days)

Regards Rohit

Answer 1

As far as my understandig the error message indiactes that the Delta log file for the Delta table has been truncated. or possibly deleted. This may even can possibily happen if the Delta log retention policy has been configured to delete log files. Once after a certain period of time or if the log file has been manually deleted.

Check for the retention period by default the retention policy is for 30 days.

Check for the versioning: #versioning in Delta #versioning provides details for all the changes happend to this table running #The row with version 0 (lower row) shows the initial version when table is created. The row version 1 shows when the optimization step.

display(spark.sql("DESCRIBE HISTORY flights"))

Use Delta Table Time Travel feature to recover the table to a previous version. You can do this by running the following command:

df=spark.read.format('delta').load('/FileStore/tables/delta3',timestampAsOf="2022-05-09 06:35:00").show()

If you still dont find from the above steps then you will need to Create DELTA TABLE and take time to time back up.