What is the average latency in transferring Azure Databricks Logs to azure log monitor?

Question

I am trying to check status of my jobs running in Azure Databricks through Azure Log Monitor dashboard but most of the recent data is not visible can someone help me out in getting near real-time monitoring of my jobs

Answer 1

Log Analytics has a delay in sending metrics not just for Databricks but almost any resource in Azure has this delay. At this time it is:

• Data from resource logs take 2-15 minutes, depending on the Azure service.
• Azure platform metrics take 3 minutes to be sent to Log Analytics ingestion point.
• Activity log data will take about 10-15 minutes to be sent to Log Analytics ingestion point.

This is due to just how the data is read and inserted on the backend.
If looking for an increase ability to Monitor Databricks jobs look at integrating your Databricks Notebook with Azure Data Factory to create pipelines