When reading or writing RDDs or Datasets/DataFrames, I usually use the function SparkContext.setJobDescription
to customize the display on the Spark UI and make it more readable.
However, it looks like this function does not work when streaming (Structured Streaming with readStream/writeStream). How to programmatically customize the Job Description (or similarly the Job Group) on the Spark UI when Structure Streaming? Is there something equivalent to SparkContext.setJobDescription
?