Stack Overflow
Stack Overflow

Questions tagged [spark-submit]

spark-submit is a script that is able to run apache-spark code written in e.g. java, scala or python

More information about spark-submit can be found here.

611 questions
0
votes
1 answer

spark startApplication never give me an AppId

I have a little problem with sparkSubmit, when I send my job, I would like to recover the appId to store it somewhere. But I can not retrieve my id. All sparkLauncher parameters are valid and the job runs well. try{ LOGGER.debug("Start…
marchat
  • 153
  • 1
  • 9
0
votes
1 answer

Which jar has org.apache.spark.sql.types?

I am on Spark 1.x, and attempting to read csv files. If I need to specify some data types, as per the documentation, I need to import the types defined in the package org.apache.spark.sql.types. import…
sudheeshix
  • 1,541
  • 2
  • 17
  • 28
0
votes
1 answer

Long running spark submit job

I am trying to run a script using spark submit as this spark-submit -v \ --master yarn \ --num-executors 80 \ --driver-memory 10g \ --executor-memory 10g \ --executor-cores 5 \ --class cosineSimillarity jobs-1.0.jar This script is implementing…
MasterGoGo
  • 98
  • 6
0
votes
0 answers

Amazon AWS cannot access my s3n url

I am trying to do spark-submit with my python file and in this python file, I would like to access my data.txt that is stored in my bucket. I am using the following command to access my file but I keep getting an error. dataFile = ("s3n://
Ray.R.Chua
  • 777
  • 3
  • 8
  • 27
0
votes
1 answer

SparkException using JavaStreamingContext.getOrCreate(): Only one SparkContext may be running in this JVM

Related to this question, I got the tip that the getOrCreate idiom should be used to avoid this issues. But trying: JavaStreamingContextFactory contextFactory = new JavaStreamingContextFactory() { @Override public JavaStreamingContext…
wikier
  • 2,517
  • 2
  • 26
  • 39
0
votes
2 answers

Is there any way to submit spark job using API

I am able to submit spark job on linux server using console. But is there any API or some framework that can enable to submit spark job in linux server?
Vimal Dhaduk
  • 994
  • 2
  • 18
  • 43
0
votes
1 answer

Memory parameters for Spark-submit command

How to calculate optimal memory setting for spark-submit command ? I am bringing 4.5 GB data in Spark from Oracle and performing some transformation like join with a Hive table and writing it back to Oracle. My question is how to come up…
Arvind Kumar
  • 1,325
  • 1
  • 19
  • 27
0
votes
1 answer

How to drop messages in console when using spark-submit?

When I run spark-submit job with scala, I can see a lot of status messages in console. But I would like to see only my prints. Can I put any parameter in order not to see these messages?
Татьяна Паскевич
  • 254
  • 1
  • 4
  • 15
0
votes
0 answers

Can't spark-submit to analytics node on DataStax Enterprise

I have a 6 node cluster, one of those is spark enabled. I also have a spark job that I would like to submit to the cluster / that node, so I enter the following command spark-submit --class VDQConsumer --master spark://node-public-ip:7077…
lostinplace
  • 1,538
  • 3
  • 14
  • 38
-1
votes
1 answer

Spark submit not working with protected nexus repository

Trying resolve builds from nexus not working if the nexus repo is password protected. giving --repositories http://:@:/ doesn't work
Vijay Jangir
  • 584
  • 3
  • 15
-1
votes
2 answers

how to get spark-sbumit logging result

When i submit spark job too terminal, it's has logging result like image in terminal. How can i get it and set it to value or object?
Diepdang
  • 37
  • 1
  • 8
-1
votes
1 answer

Spark-submit fails with return code 13 for example of wordCount

My spark-submit command is : spark-submit --class com.sundogsoftware.spark.WordCountBetterDataset --master yarn --deploy-mode cluster SparkCourse.jar And for defining the sparkSession, i use this : val spark = SparkSession .builder …
YounessGI
  • 101
  • 3
-1
votes
1 answer

I want to split one file into several in HDFS

I am trying to write a file to hdfs using spark-submit. When writing a file, I want to split one file into several, like the result of mapreduce rather than one.(ex. part-0000, park-0001) here is sample code What options should I set val conf = new…
myskbj
  • 31
  • 4
-1
votes
1 answer

How to kill spark job from ps -ef programatically

I want to kill the spark job programatically. Following is the scenario: When I kill the spark job using yarn application -kill it is getting killed, but if I do the ps -ef | grep then it is that spark job entry. How do I make…
Saurabh Deshpande
  • 1,153
  • 4
  • 18
  • 35
-1
votes
1 answer

SparkWordCount problem - java.lang.ClassNotFoundException

This is not a duplicate question and i have tried so many ways to make this work, but did not work I am trying to write a word count application so that i can run through spark-submit I am using IntelliJ IDEA, spark - 2.1.1 and scala - 2.11.8 My…
user10865587
1 2 3
…
40
41