Get the number of running queries on Spark at the same time

Asked Jan 04 '19 at 06:47

Active Aug 11 '23 at 20:17

Viewed 66 times

I have a cluster with one Spark master and three Spark workers which are used to query on Cassandra. Also I have a UI app with java that the users can insert the query fields on the UI widgets.

Since some users use my system, so I have multiple queries with Spark on Cassandra at the same time.

I know that SparkListener can show start of a job or some thing like that, but I don't see any function to get the current running jobs on Spark in SparkListener. Just there are some functions that can handle some request like started job or task or ...

Now how can I get the number of queries which are running on my Spark by the users queries for example on every 5 seconds (in java or scala)?

I saw the following questions but my problem does not solve:

Get current number of running containers in Spark on YARN

Spark - How many Executors and Cores are allocated to my spark job

edited Aug 11 '23 at 20:17

marc_s

732,580
175
1,330
1,459

asked Jan 04 '19 at 06:47

ahmadi morteza ali

Spark History Server --https://spark.apache.org/docs/latest/monitoring.html – Taha Naqvi Jan 04 '19 at 08:21
@TahaNaqvi thanks but I see that "Spark History Server Configuration Options" is just some options that can control maxApplications or retainedApplications ... . But how can I see the number of the current queries running on Spark? – ahmadi morteza ali Jan 04 '19 at 09:56
yarn application -list | grep "SPARK" | grep "RUNNING" | wc -l – Taha Naqvi Jan 04 '19 at 13:33
@TahaNaqvi thanks, can I access this from java code? – ahmadi morteza ali Jan 04 '19 at 15:07
https://stackoverflow.com/questions/52717163/list-all-yarn-application-in-hadoop-cluster-through-java... .refer this and create objects of response and filter on type of app. – Taha Naqvi Jan 04 '19 at 18:15

Get the number of running queries on Spark at the same time

0 Answers0