Stack Overflow
Stack Overflow

Questions tagged [spark-shell]

More information can be found in the official documentation.

135 questions
0
votes
1 answer

Which "JAR" file do i need to be able to import "org.apache.parquet" in Scala?

When I try this: scala> import org.apache.parquet It errors out: :23: error: object parquet is not a member of package org.apache import org.apache.parquet Question - which jar do i need to include in spark conf for this import to…
Alex B
  • 2,165
  • 2
  • 27
  • 37
0
votes
1 answer

spark-shell error: value builder is not a member of object com.amazonaws.services.s3.model.PutObjectRequest

I'm just getting started with EMR Hadoop/spark etc., I am trying to use spark-shell to run a scala code to upload a file to EMRFS S3 location however I am receiving below error - Without any Import If I run => val bucketName = "bucket" val…
RMu
  • 817
  • 2
  • 17
  • 41
0
votes
0 answers

my pyspark did not start on terminal,but with jupyter notebook

Not long ago, when I input pyspark in my terminal. the terminal will finally become...um...like this: some information >>> but now it start with jupyter notebook automatically. This phenomenon happened with spark-3.0.0-preview2-bin-hadoop3.2 I have…
user13268019
0
votes
0 answers

How to increase detail level of db2 exceptions in spark-shell?

I got an error after running a scala script in a db2 powered spark-shell. I can't find much information about the error online and wanted to know if there was a way to do that. I have tried executing the getNextException() method on the exception…
Matt
  • 113
  • 1
  • 1
  • 5
0
votes
2 answers

Running scala script with line breaks in spark-shell

I'm trying to run a scala script through spark shell using the following command: spark-shell -i myScriptFile.scala I can get the above command to work when I have single-line commands, but if I have any line-breaks in the script (for readability),…
NicolasCage
  • 105
  • 9
0
votes
0 answers

Livy spark interactive session

I'm trying to create spark interactive session with livy .and I need to add a lib like a jar that I mi in the hdfs (see my code ) . but the session is dead and the log is below. code : client =…
zekri sidi mohamed hicham
  • 57
  • 1
  • 10
0
votes
2 answers

RDD output in spark-shell differs from print(RDD) in idea

val rddData1 = sc.makeRDD(1 to 10, 2) println(rddData1.glom.collect) code in idea or spark-shell will output [[I@34a0ef00 but rddData1.glom.collect in spark-shell will output Array[Array[Int]] = Array(Array(1, 2, 3, 4, 5), Array(6, 7, 8, 9,…
chinayangyongyong
  • 37
  • 1
  • 9
0
votes
2 answers

spark-shell load existing hive table by partition?

In spark-shell, how do I load an existing Hive table, but only one of its partitions? val df = spark.read.format("orc").load("mytable") I was looking for a way so it only loads one particular partition of this table. Thanks!
kcode2019
  • 119
  • 1
  • 7
0
votes
2 answers

How to add library to spark shell

I have a library that I want to use in spark shell, how can I add this library to be accessible from the spark shell? sbt : resolvers += Resolver.bintrayRepo("unsupervise", "maven") libraryDependencies += "com.github.unsupervise" %% "spark-tss" %…
zekri sidi mohamed hicham
  • 57
  • 1
  • 10
0
votes
1 answer

Does the Spark Shell JDBC read numPartitions value depend on the number of executors?

I have Spark set up in standalone mode on a single node with 2 cores and 16GB of RAM to make some rough POCs. I want to load data from a SQL source using val df = spark.read.format('jdbc')...option('numPartitions',n).load(). When I tried to measure…
Priyank
  • 1,513
  • 1
  • 18
  • 36
0
votes
1 answer

How to read hive managed table data using spark?

I am able to read hive external table using spark-shell but, when I try to read data from hive managed table it only shows column names. Please find queries here:
Nano
  • 5
  • 1
  • 5
0
votes
1 answer

Set default packages variable for spark-shell

I trying to start spark-shell having packages variable set by an environment variable by default. The normal execution command is spark-shell --packages com.databricks:spark-csv_2.11:1.3.0 I would like to avoid to write always --packages…
Mauro Midolo
  • 1,841
  • 3
  • 14
  • 34
0
votes
1 answer

Is the SparkSession variable staged by spark-shell (scala) a val or a var?

I am trying to convert my Spark Scala scripts (written in spark-shell) as Scala Class, Object, methods (def), etc. so I create JARs for spark-submit. I make a lot of calls using Spark SQL that performs a lot of timestamp computations with respect…
geekyj
  • 402
  • 6
  • 10
0
votes
1 answer

spark is launched dir is working but spark-shell is not getting lanuch getting error 'spark-shell' is not recognized

cmd screenshotI have done everything as instructed but still not able to launch spark-shell dir is working I have placed all downloaded files as instructed but spark-shell is not working. C:\Users\admin>cd c:\spark c:\spark>dir Volume in drive C…
user11787825
  • 1
  • 1
  • 4
0
votes
0 answers

Hon to find particular sentence from given string in spark?

I want to extract a specific part from a string in Spark for e.g. my string is val b= "URL ftp://216.24.126.75/serversoftware/ocs/OCS_Inventory_NGInstallation_and_Administration_Guide_1.7_EN.odt MENTION cryptography 201564 …
Sabid Ansari
  • 1
  • 1
1 2 3
…
8 9