Difference in counts on a Delta Table immediately after write operation

Asked Jun 08 '22 at 08:41

Active Jun 08 '22 at 09:27

Viewed 238 times

I've a Databricks job the writes to a certain Delta Table. After the write has been completed, the job is calling another function that reads and calculates some metrics(counts to be specific) on top of the same Delta table. But the counts/metrics being calculated are less than the actual counts for the the given partition_dates. Just to verify that the function calculating the metrics is working fine or not, I called the function after the Databricks job got completed and found that the counts were correct in that run.

I feel Delta table is not completely updated by the time I try to read from it, even though the write operation had completed successfully.

Any help would be appreciated.

edited Jun 08 '22 at 09:27

Olaf Kock

46,930
8
59
90

asked Jun 08 '22 at 08:41

Anirudh Gupta

what cloud are you using? – Alex Ott Jun 08 '22 at 09:28
what mode (append, overwrite?) do you use to write to DT and what do you read from it (all or changes only or specific version?) – Dariusz Krynicki Jun 08 '22 at 10:00
@AlexOtt I'm using Azure – Anirudh Gupta Jun 09 '22 at 17:30
@DariuszKrynicki The writes are either appends or upserts depening on the condition. Once the write is completed, I'm reading directly from the abfs file path(Azure Blob File Storage) of the Delta Table – Anirudh Gupta Jun 09 '22 at 17:34

Difference in counts on a Delta Table immediately after write operation

0 Answers0