1

Lately realized the Spark SQL auxiliary statements (EXPLAIN, DESCRIBE, SHOW CREATE etc.,) not shown in Spark UI. I have an use-case to track all the queries executed through Spark SQL JDBC connection; just these statements go untracked.

So, my questions are,

  1. Why these are not shown in Spark UI?
  2. Is it possible to get these statements execution listed in Spark UI?
Kondasamy Jayaraman
  • 1,802
  • 1
  • 20
  • 25
  • I think you will see `show` since it triggers an action. I don't think you can get the `explain` and `describe` to show in the UI, since they don't trigger any actions on the cluster. They just show the dag that would go through the optimizer (in case of explain). – m_vemuri Apr 29 '22 at 06:27
  • Any possibility to extend/ hack, so that get listed in SparkUI? – Kondasamy Jayaraman Apr 29 '22 at 06:49
  • 1
    It may be possible, but I think it may also be quite challenging to do without knowing the underlying code architecture to begin with. I've never done it myself. It might be easier to log your commands and use a centralized logging solution like LogStash + Elastic Search which can then count up the number of different statements by application. – m_vemuri Apr 29 '22 at 18:14

0 Answers0