What is the best way to log metrics from Databricks Delta Live Tables in DataDog? I create connection between Datadog and Databricks, and I can send logs from Databricks to the Datadog. I have problem with sending logs for Delta Live Tables.
For getting metrics from Delta Live Tables, I use events and Delta Live History. In events I have all events for that table, and I need just events for last Delta Live Table process.
event_log = spark.read.format('delta').load(event_log_path)
deltaTableHistory = DeltaTable.forPath(spark, table_path).history().collect()
