1

I have problem with submit task in Mesos with cluster mode, first I using this syntax to running cluster mode on Mesos

$ cd spark
$ ./sbin/start-mesos-dispatcher.sh --master mesos://10.2.3.95:5050

After that I submit task in Mesos using spark-submit

./bin/spark-submit --master mesos://10.2.3.95:7077 \
                   --deploy-mode cluster \
                   --name scraping_twitter_1.py \
                   --py-files ~/scraping/scraping_twitter_1.py \
                   --driver-cores 4 \
                   --total-executor-cores 8 \
                   --conf spark.master.rest.enabled=true \
                   ~/scraping/scraping_twitter_1.py

And then problem like this enter image description here

I already changes the spark-submit script to run this task I already try to add

--driver-cores 4 \
--driver-memory 10G \
--executor-cores 4 \
--executor-memory 10G

But the result is still same Driver cores must be a positive number

The task run well enter image description here

I also open http://10.2.3.95:8081/ to see the log enter image description here

Calvin Valentino Gosal
  • 11
  • 3
  • The same problem was raised here, https://stackoverflow.com/questions/71691292/driver-cores-must-be-a-positive-number – Alexandru T. Jul 01 '22 at 11:15

0 Answers0