Is it possible to submit multiple spark-submit using a single spark operator on k8s? Or is a dedicated spark-operator required for each spark-submit?
Asked
Active
Viewed 197 times
0
-
Were you able to run multiple jobs? – Alez May 16 '23 at 17:55
1 Answers
0
The Spark operator allows to add the Spark functionalities on Kubernetes and use them in the form of SparkApplication custom resource components.
After that, you can set and execute multiple spark applications on the same cluster and they will be executed until there are resources available
∑apps-resource < Cluster_resources
If the sum of the applications-required resources exceeds the cluster resources the Job will be queued.
Moreover, the operator uses spark-submit under the scene but you need to interact with it only through SparkApplication. You can also use the native spark-submit to execute spark workload but in this case you're not using the Kubernetes operator.
Check how to install the operator and how to write application examples you can find examples of commands.
Alez
- 1,913
- 3
- 18
- 22
-
Hi Alez, I think I need to re-phrase my question. I have the spark operator running and I have a spark job(say job 1) running. But is it possible to submit another job to the same spark cluster where job 1 is running? As it is just a small job running once a day, so want to re use the driver and executor for the job 1. – haripriya rajendran May 17 '23 at 06:59
-
driver and executor are assigned as defined in `SparkApplication` manifest. Not sure what you mean by re use. You mean driver and executor for Job 1 for another Job? Each `SparkApplication` deployment has its own driver and executor(s) and multiple `SparkApplication` can run as long as your k8s has resources. Also, if the job is just running once per day, you might want to consider running `ScheduledSparkApplication`. It is also something `spark-operator` supports. – it243 Jun 21 '23 at 03:35