1

I am using structured streaming for reading csvs and writing to kafka. The streaming tab is not showing in Spark UI (not using streaming context).

val userSchema = new StructType().add("name", "string").add("age", "integer")
val csvDF = spark
  .readStream
  .option("sep", ";")
  .schema(userSchema)      // Specify schema of the csv files
  .csv("/path/to/directory")

How can I get streaming metrics in the UI?

Shaido
  • 27,497
  • 23
  • 70
  • 73
sam8686
  • 37
  • 8
  • Streaming tab is for Spark Streaming, not Structured Streaming. What metrics are you looking for? Did you look under the SQL tab? – Harichandan Pulagam May 10 '19 at 00:32
  • How can i get visualization in ui for structured streaming something like streaming tab ? – sam8686 May 10 '19 at 15:13
  • Take a look at this link: https://forums.databricks.com/questions/11424/streaming-tab-is-not-available-on-spark-ui-for-str.html You'll have to implement a listener like @gaston has suggested, and push those events to a monitoring service – Harichandan Pulagam May 10 '19 at 17:29

1 Answers1

1

To see some metrics (in console), you need to add a listener

spark.streams.addListener(new StreamingQueryListener {
  override def onQueryStarted(event: StreamingQueryListener.QueryStartedEvent): Unit = logger.debug(s"QueryStarted [id = ${event.id}, name = ${event.name}, runId = ${event.runId}]")

  override def onQueryProgress(event: StreamingQueryListener.QueryProgressEvent): Unit = logger.warn(s"QueryProgress ${event.progress}")

  override def onQueryTerminated(event: StreamingQueryListener.QueryTerminatedEvent): Unit = logger.debug(s"QueryTerminated [id = ${event.id}, runId = ${event.runId}, error = ${event.exception}]")
})

QueryProgressEvent, displays info about offset, watermarks, source, sinks,etc.

This video can help you : Monitoring Structured Streaming Applications

gaston
  • 498
  • 7
  • 15