Spark submit using mesos dcos cli

Question

I'm trying to start a spark streaming job on mesos using the DCOS cli. I'm able to start the job. My program expects a config file to be passed as cli parameter. How do I achieve this with dcos spark run --submit-args?

I tried --files http://server/path/to//file hoping it will download files but that didn't work. Driver starts but fails because config file is missing.

I also tried to roll up the jar and config file as tar and submitted it. I can see in Mesos logs that the tar was fetched and untar. Both config and jar file are seen in the working directory. But job fails with ClassNotFoundException. I suspect something was not right about how spark-submit was started.

dcos spark run --submit-args="--supervise --deploy-mode cluster --class package.name.classname http://file-server:8000/Streaming.tar.gz Streaming.conf"

Any hint on how to proceed? Also, in which log file can I see the underlying spark-submit command used by DCOS?

score 4 · Accepted Answer · edited Jun 20 '20 at 09:12

Here is the example of a command you should launch in order to make it work:

dcos spark run --submit-args='--conf spark.mesos.uris=https://s3-us-west-2.amazonaws.com/andrey-so-36323287/pi.conf --class JavaSparkPiConf https://s3-us-west-2.amazonaws.com/andrey-so-36323287/sparkPi_without_config_file.jar /mnt/mesos/sandbox/pi.conf'

Where

--conf spark.mesos.uris=... A comma-separated list of URIs to be downloaded to the sandbox when driver or executor is launched by Mesos. This applies to both coarse-grained and fine-grained mode.

/mnt/mesos/sandbox/pi.conf A path to the downloaded file which your main class receives as a 0th parameter (see the code snippet below). /mnt/mesos/sandbox/ is a standard path inside a container which is mapped to a corespondent mesos-task sandbox.

public final class JavaSparkPiConf {

  public static void main(String[] args) throws Exception {
    SparkConf sparkConf = new SparkConf().setAppName("JavaSparkPi");
    JavaSparkContext jsc = new JavaSparkContext(sparkConf);

    Scanner scanner = new Scanner(new FileInputStream(args[0]));
    int slices;
    if (scanner.hasNextInt()) {
      slices = scanner.nextInt();
    } else {
      slices = 2;
    }
    int n = 100000 * slices;
    List<Integer> l = new ArrayList<>(n);
    for (int i = 0; i < n; i++) {
      l.add(i);
    }

    JavaRDD<Integer> dataSet = jsc.parallelize(l, slices);

    int count = dataSet.map(new Function<Integer, Integer>() {
      @Override
      public Integer call(Integer integer) {
        double x = Math.random() * 2 - 1;
        double y = Math.random() * 2 - 1;
        return (x * x + y * y < 1) ? 1 : 0;
      }
    }).reduce(new Function2<Integer, Integer, Integer>() {
      @Override
      public Integer call(Integer integer, Integer integer2) {
        return integer + integer2;
      }
    });

    System.out.println("Pi is roughly " + 4.0 * count / n);

    jsc.stop();
  }
}

That was it. Missing piece was to look for downloaded files in /mnt/mesos/sandbox. — Cheeko, Apr 05 '16 at 00:47

score 2 · Answer 2 · answered Mar 31 '16 at 17:10

2

Streaming.conf is just a string that will be passed to your driver. Your driver must be able to see it. The easiest way to do this is to place it in an accessible location, the specify that you want it downloaded to the sandbox via spark.mesos.uris [1]. You could alternately write your application to support reading from a remote location, and just pass the location on the CLI.

--files is used to place files at the executors, but you're trying to pass a file to the driver, so that won't work.

[1] http://spark.apache.org/docs/latest/running-on-mesos.html

Michael Gummelt
Mesosphere

answered Mar 31 '16 at 17:10

Michael Gummelt

702
4
15

I know parameters are strings passed to program :) Driver already reads that file. I'm trying to make it available. I'll try your suggestion. At what point does the links specified in URI get downloaded? i read the conf file before starting the spark context. – Cheeko Apr 01 '16 at 02:31
one more, if I need to place extra files(--files) or jars(--jars) in executors, how do I achieve this? – Cheeko Apr 01 '16 at 02:39
spark.mesos.uris does download the file. I can see that in mesos fetcher logs. But its not in current directory or classpath. I'm not able to "see" the file in my code. I've resorted to reading directly from a shared location but it will be good to know how to manage and share resources. – Cheeko Apr 01 '16 at 20:35

Spark submit using mesos dcos cli

2 Answers2