Spark continuous structured streaming not showing input rate or process rate metrics

Question

I'm running my spark continuous structured streaming application on a standalone cluster. However I noticed that metrics like average input/sec or avg process/sec is not showing(as NaN) on the structured streaming UI. I have spark.sql.streaming.metricsEnabled = true

If I run the app in micro-batched structured streaming, I can see these metrics. I also tried this continous streaming app on databricks, somehow I'm able to see these metrics. I'm wondering how can I monitor the continuous streaming metrics on my standalone cluster. Thank you!

Answer 1

From https://docs.databricks.com/en/structured-streaming/stream-monitoring.html#defining-observable-metrics-in-structured-streaming

Batch mode: Use QueryExecutionListener.

QueryExecutionListener is called when the query completes. Access the metrics using the QueryExecution.observedMetrics map.

Streaming, or micro-batch: Use StreamingQueryListener.

StreamingQueryListener is called when the streaming query completes an epoch. Access the metrics using the StreamingQueryProgress.observedMetrics map. Databricks does NOT support continuous execution streaming.