Stack Overflow
Stack Overflow

Questions tagged [spark-submit]

spark-submit is a script that is able to run apache-spark code written in e.g. java, scala or python

More information about spark-submit can be found here.

611 questions
1
vote
0 answers

Spark Launcher connection time

While connecting through spark launcher through remote from one Virtual machine to another Virtual machine in the same host(deploy mode is the client) it is taking 40seconds for connection establishment can anybody suggest me to decrease this time…
nishant
  • 67
  • 1
  • 5
1
vote
0 answers

How to spark-submit a Play Framework Project

I am working with Play2 Framework, (2.5.18) and Spark(2.3.0), Spark-Cassandra connector (2.3.0) and sbt 0.13.15 I am using Play framework to design an API, using Spark Cassandra Connector from Datastax to access data. I would like to know how to do…
jowdev
  • 11
  • 1
1
vote
0 answers

Spark Submit Executors == Spark Shell Tasks?

I am trying to understand the difference in speed between my spark-submit and spark shell jobs. I start the shell or submit with the same resource allocations but I seem to be getting very different performance. When I run it in the shell it take…
Max
  • 837
  • 4
  • 11
  • 20
1
vote
1 answer

How to get driver-Id in spark submission

Spark cluster information: Spark version: 2.2.0 Cluster contains a master node with 2 worker nodes Cluster Manager Type: standalone I submit a jar to spark cluster from one of the workers and I want to receive the driver-Id from the submission so…
Arsinux
  • 173
  • 1
  • 4
  • 13
1
vote
1 answer

Spark-submit Spring Boot profiles.active variable

Good day for you collegues. I am trying run my appliacation on a cluster. My script looks like: #!/usr/bin/env bash /usr/hdp/current/spark2-client/bin/spark-submit \ --driver-memory 12G \ --master yarn \ --driver-java-options…
Александр Шаповалов
  • 487
  • 1
  • 6
  • 19
1
vote
1 answer

Spark / Mesos / Tasks lost, slaves blacklisted, executors removed

I am running a spark-submit job on Spark 2.2.0 with Scala 2.11.11, SBT on Mesos 1.4.2. I have problem with tasks being lost and executors not being registered. Here are the symptoms: MesosCoarseGrainedSchedulerBackend launches tasks until…
astro_asz
  • 2,278
  • 3
  • 15
  • 31
1
vote
2 answers

spark-submit from outside AWS EMR cluster

I have an AWS EMR cluster running spark, and I'd like to submit a PySpark job to it from my laptop (--master yarn) to run in cluster mode. I know that I need to set up some config on the laptop, but I'd like to know what the bare minimum is. Do I…
mm_857
  • 171
  • 11
1
vote
2 answers

error: path does not exist in spark submit with hadoop

We are using the command /home/ubuntu/spark/bin/spark-submit --master yarn --deploy-mode cluster --class "SimpleApp" /home/ubuntu/spark/examples/src/main/scala/sbt/target/scala-2.11/teste_2.11-1.0.jar to run the script below import…
Paulo Viegas
  • 13
  • 3
1
vote
0 answers

Custom /src/main/resources/log4j.properties not loaded in Spark 2.2

I am trying to change the level of logging to StdErr console in Spark 2.2 under Mesos and using Scala, sbt and spark-submit in cluster mode. Background Recently we had our Spark/Mesos cluster reinstalled with new version of spark 2.2.0 vs previous…
astro_asz
  • 2,278
  • 3
  • 15
  • 31
1
vote
1 answer

Exec pyspark standalone script without spark-submit

I am new to installing spark on a Linux machine and have possibly a basic question : I have installed Spark version 1.6.0 and Python 2.6.6 In spark interactive mode, I am able to run these simple commands to count lines in the README.md…
aau22
  • 31
  • 6
1
vote
2 answers

Spark on Fargate can't find local IP

I have a build job I'm trying to set up in an AWS Fargate cluster of 1 node. When I try to run Spark to build my data, I get an error that seems to be about Java not being able to find "localHost". I set up the config by running a script that adds…
Adam
  • 344
  • 1
  • 15
1
vote
1 answer

Pass opt arguments in an application executed as a .jar through spark-submit --class and use the existing context

I am writting a scala project that I want to have classes that are executable from spark-submit as a jar class. (e.g. spark-submit --class org.project My problems are the following: I want to use the spark-context-configuration that the user sets…
ysig
  • 447
  • 4
  • 18
1
vote
0 answers

Error initializing SparkContext due to javax.net.ssl.SSLHandshakeException

I am having trouble following along with the instructions to submit a spark job on a kubernetes cluster. I am following the instructions at: https://spark.apache.org/docs/latest/running-on-kubernetes.html I have tried to submit the job…
Mshah
  • 11
  • 2
1
vote
4 answers

Output results of spark-submit

I'm beginner in spark and scala programing, I tried running example with spark-submit in local mode, it's run complete without any error or other message but i can't see any output result in consul or spark history web UI .Where and how can I see…
mahzad.mah
  • 21
  • 1
  • 5
1
vote
1 answer

Spark-submit with uber-jar failed to execute the job without any output or error logs

I am trying to generate uber-jar using sbt compile and sbt package commands for running my application on our remote server with spark installed as standalone mode there. I used deeplearning4j framework for building LSTM neural network and tend to…
rickyhai11
  • 11
  • 2
1 2 3
…
40 41