The Spark hidden REST API (https://gist.github.com/arturmkrtchyan/5d8559b2911ac951d34a) has been proven useful to me for submitting Scala jobs. But is there any way to submit SparkR jobs through this API?
I tried it but got this error:
Exception in thread "main" java.lang.ClassNotFoundException:
It is looking for a main class, but naturally, R won't have a main class. I am just passing my R file like I would do for spark-submit
. Below is the request payload:
{
"action" : "CreateSubmissionRequest",
"appArgs" : [],
"appResource" : "file:/home/piyush/global_ui_pipeline.r",
"clientSparkVersion" : "2.3.1",
"environmentVariables" : {
"SPARK_ENV_LOADED" : "1"
},
"mainClass" : "",
"sparkProperties" : {
"spark.driver.supervise" : "false",
"spark.app.name" : "sparkR",
"spark.eventLog.enabled": "true",
"spark.submit.deployMode" : "cluster",
"spark.master" : "spark://localhost:6066"
}
}
If I remove the mainClass field from the json, it gives the following error:
Malformed request: org.apache.spark.deploy.rest.SubmitRestMissingFieldException: Main class is missing.
Can the hidden rest API be used for submitting sparkR jobs? If at all it can be, how can it be done?