Will a Databricks Delta Live Table generate costs regardless of finding data to load? And would a solution in that case be to set the job to disabled if you know new data is not filling the source for a while?
Asked
Active
Viewed 573 times
1 Answers
3
Delta Live Tables job will incur costs only when it's running. Usually jobs are set to be triggered regularly, so even if you don't have data the cluster will be created, then DLT pipeline will be executed, and if no data is found, then (most probably, but depends on the job), it will just finish as there is no data to process. In this case the costs will be relatively small until you setup a pipeline in continuous mode.
If you know that no data is arriving, you can pause a job that triggers DLT pipeline.
Alex Ott
- 80,552
- 8
- 87
- 132