Stack Overflow
Stack Overflow

Questions tagged [spark-shell]

More information can be found in the official documentation.

135 questions
2
votes
1 answer

Dataproc: update log level in Spark shell

I use Jupyter terminal for accessing the driver of Dataproc cluster. This is my gateway to the cluster, and I do not have direct SSH enabled for the driver machine. When I launch spark-shell , I keep getting these info, debug, Contextcleaner…
Jaysheel Kalgal
  • 46
  • 3
2
votes
1 answer

Is there a version compatibility issue between Spark/Hadoop/Scala/Java/Python?

I'm getting an error while running spark-shell command through cmd but unfortunately without any luck so far. I have Python/Java/Spark/Hadoop(winutils.exe)/Scala installed with versions as below: Python: 3.7.3 Java: 1.8.0_311 Spark:…
Saurabh
  • 199
  • 1
  • 3
  • 11
2
votes
1 answer

spark-shell exception org.apache.spark.SparkException: Exception thrown in awaitResult

Facing below error while starting spark-shell with yarn master. Shell is working with spark local master. admin@XXXXXX:~$ spark-shell --master yarn 21/11/03 15:51:51 WARN Utils: Your hostname, XXXXXX resolves to a loopback address: 127.0.1.1; using…
Ashish Mishra
  • 510
  • 4
  • 18
2
votes
1 answer

How to convert a list to a list of tuples in scala?

Input: Val l= List("k1","v1","k2","v2") Desired output: List(("k1","v1"),("k2","v2")) I have tried using zip,folding, slicing but no luck. Note:I have done it in python but couldn't able to do in scala.
mani sekhar
  • 57
  • 1
  • 3
  • 6
2
votes
1 answer

Executing Linux Command in Scala-Shell

I'm working on a project where I'm needing to execute some linux commands (sqoop command) in my Scala application. See sample command I tried executing with MySql on my VM. import sys.process._ "sqoop eval --connect…
Ope Baba
  • 65
  • 8
2
votes
1 answer

How to use createDataFrame and createDF on spark-shell

There are good examples to use it, as here, but on spark-shell " ... createDF is not a member of org.apache.spark.sql.SparkSession". PS: using Spark v2.2. EDIT: sorry all, it is an external lib. Little change in the question: how to import a Github…
Peter Krauss
  • 13,174
  • 24
  • 167
  • 304
2
votes
1 answer

Why spark shell throwing error, when i try to create variable with name "org"?

I tried to create variable/val with name "org" in spark shell(Scala version 2.10.5) , but throwing error I tried with both var and val. var org = List(1) error: value apache is not a member of List[Int] …
Santhosh Kumar
  • 21
  • 5
2
votes
0 answers

Spark-shell cannot read CSV file, failing with timeout in local mode

spark-shell --packages com.databricks:spark-csv_2.11:1.5.0 spark > val flightData2015 = spark.read.option("inferSchema", "true").option("header","true").csv("/Users/me/Desktop/2015-summary.csv") I am getting this exception. java.io.IOException:…
Sarath Sasikumar
  • 113
  • 10
2
votes
1 answer

Spark Shell allowing to redeclare the same immutable variable

I am working with Spark-shell for Scala and found a strange behaviour in Spark-shell REPL which is not there if i use any IDE. I can declare the same immutable variable again and again in REPL, but the same is not allowed in IDE. Here is the code in…
KayV
  • 12,987
  • 11
  • 98
  • 148
2
votes
1 answer

Can not start Spark-shell

Hello I've unzip and exported spark path. Whne I lunch it I got this error. export PATH=$PATH:/usr/local/spark/spark24/bin $ spark-shell ERROR Traceback (most recent call last): File "/usr/local/bin/find_spark_home.py", line 74, in …
abdoulsn
  • 842
  • 2
  • 16
  • 32
2
votes
2 answers

Why I take "spark-shell: Permission denied" error in Spark Setup?

I am new on Apache Spark. I am trying to setup Apache Spark to my Macbook. I download file "spark-2.4.0-bin-hadoop2.7" from Apache Spark official web site. When I try to run ./bin/spark-shell or ./bin/pyspark I get Permission denied error. I want…
stef
  • 91
  • 2
  • 10
2
votes
2 answers

Creating a Dataframe using spark2-shell and got this error

I am new to Spark-shell and I am getting this error when creating a dataframe from a csv file: scala> val bankDF = bankrdd.toDF() bankDF.registerTempTable("bankfull") …
Tony Davis
  • 21
  • 1
2
votes
0 answers

Spark-shell failed to start with exception:java.lang.reflect.InvocationTargetException

I'm new to Spark, installed several machines, runs spark-shell OK. But on one of my machines (RHEL5, a bit old), I got problem: I've installed JDK, Scala, Ppark from tar.gz files as below: export JAVA_HOME=/home/me/software/jdk1.8.0_181 export…
Hind Forsum
  • 9,717
  • 13
  • 63
  • 119
2
votes
1 answer

Pair RDD tuple comparison

I am learning how use spark and scala and I am trying to write a scala spark program that receives and input of string values such as: 12 13 13 14 13 12 15 16 16 17 17 16 I initially create my pair rdd with: val myRdd =…
AntarianCoder
  • 45
  • 5
1
vote
1 answer

How to release RAM memory after stopping a Spark-Shell instance?

I've observed that the RAM holds on to data even after stopping/quitting the spark-shell instance. How can I free RAM memory?
Buggy
  • 109
  • 1
  • 5
1
2
3
…
8 9