1

The Spark hidden REST API (https://gist.github.com/arturmkrtchyan/5d8559b2911ac951d34a) has been proven useful to me for submitting Scala jobs. But is there any way to submit SparkR jobs through this API?

I tried it but got this error:

Exception in thread "main" java.lang.ClassNotFoundException:

It is looking for a main class, but naturally, R won't have a main class. I am just passing my R file like I would do for spark-submit. Below is the request payload:

{
  "action" : "CreateSubmissionRequest",
  "appArgs" : [],
  "appResource" : "file:/home/piyush/global_ui_pipeline.r",
  "clientSparkVersion" : "2.3.1",
  "environmentVariables" : {
    "SPARK_ENV_LOADED" : "1"
  },
  "mainClass" : "",
  "sparkProperties" : {
    "spark.driver.supervise" : "false",
    "spark.app.name" : "sparkR",
    "spark.eventLog.enabled": "true",
    "spark.submit.deployMode" : "cluster",
    "spark.master" : "spark://localhost:6066"
  }
}

If I remove the mainClass field from the json, it gives the following error:

Malformed request: org.apache.spark.deploy.rest.SubmitRestMissingFieldException: Main class is missing.

Can the hidden rest API be used for submitting sparkR jobs? If at all it can be, how can it be done?

Piyush Shrivastava
  • 1,046
  • 2
  • 16
  • 43

0 Answers0