Stack Overflow
Stack Overflow

Questions tagged [spark-shell]

More information can be found in the official documentation.

135 questions
1
vote
1 answer

How to change spark session in spark shell?

I have been experimenting with multiple spark sessions using the same spark context. Once the spark-shell is launched, I can create a new spark session using the following. val spark2 = spark.newSession(); spark2: org.apache.spark.sql.SparkSession =…
Somesh Dhal
  • 336
  • 2
  • 15
1
vote
0 answers

Apache-spark installation error :unable to run spark-shell

I am facing thisenter image description here issue. Currently mac doesn't support the cask command . I tried installing scala 2.13.8 with jdk 17 and I am getting the exact same error message. What is the solution for this? I ran the following set…
Sairaj Nemmaniwar
  • 11
  • 1
1
vote
1 answer

Spark Shell: SQL Query doesn't return any results when data is integer/double

I am using the MongoDB Spark Connector to import data from MongoDB and then perform some SQL queries. I will describe the whole process before getting into the actual problem in case I have done something wrong since it's the first time I am using…
Antonis Pervanas
  • 11
  • 4
1
vote
1 answer

Spark-shell does not import specified jar file

I am a complete beginner to all this stuff in general so pardon if I'm missing some totally obvious step. I installed spark 3.1.2 and cassandra 3.11.11 and I'm trying to connect both of them through this guide I found where I made a fat jar for…
s1ri
  • 13
  • 3
1
vote
0 answers

Spark-shell returning the error : SparkContext: Error initializing SparkContext/Utils: Uncaught exception in thread main

I tried installing Spark on windows 10. I followed the steps in this order: Installed Java (outside Program Files folder in C drive) Validated the version of spark downloaded from Apache(spark-3.2.0-bin-hadoop3.2.tgz) unzip the spark in the folder…
Anu
  • 11
  • 1
1
vote
0 answers

Spark-shell can't access Hive MetaStore

This trouble drives me crazy. I've been spending for 5 days on it. I have a hadoop-3.1.3 environment with hive-2.3.7 and spark-3.0.0 ; hive works well. However, spark-shell can't access hive metadata. scala> spark.sql("show…
Daniel Hoover
  • 11
  • 1
1
vote
1 answer

Using s3a on linux machine fail for >100 columns parquet

I am using s3a to read from database into dataframe and write to .parquet(s3a://bucketname//folder). It works for <100 column dataframe but crashes.exits spark-shell for >~100 columns. cannot find any material if this is column limitation/version…
Nnerdscience
  • 21
  • 3
1
vote
1 answer

object SparkHadoopUtil in package deploy cannot be accessed in package org.apache.spark.deploy

Why SparkHadoopUtil is not accessible here whereas is accessible in lower version of spark even though they are imported? Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ …
supernatural
  • 1,107
  • 11
  • 34
1
vote
1 answer

json4s - overloaded method value parse with alternatives

I have a Spark project that is using json4s. It works fine when running normally submitted, but I encounter errors trying to parse JSON from spark shell. The simplest example from json4s readme (this way it is used in the project) throws an…
Kombajn zbożowy
  • 8,755
  • 3
  • 28
  • 60
1
vote
0 answers

error: not found: value in scala command wordcount

I am trying run some scala code in spark following an online video tutorial: it's a wordcount program val lines = sc.textFile("/data/mr/wordcount/big.txt") val words = lines.flatMap(x => x.split(" ")) val pairs = words.map(s => (s.toLowerCase(),…
eya_bklt
  • 305
  • 3
  • 10
1
vote
1 answer

Spark-shell backspace is printing '$<3>' on screen

I'm pretty new to scala & Spark and as I try to delete a line of code I see $<3> instead of deleting the desired line/character. enter image description here
user4516038
1
vote
1 answer

User class threw exception: java.util.MissingFormatWidthException: %-%

I am using Spark 2.4 and running below query, select ID from WEB_TBL where ID NOT LIKE '%-%' This query is giving proper output in Spark-shell but when I am running through Java class, its throwing below error. ERROR ApplicationMaster: User class…
Kalpesh
  • 694
  • 2
  • 8
  • 28
1
vote
3 answers

java.lang.NoClassDefFoundError: org/apache/spark/sq/sources/v2/StreamingWriteSupportProvider trying to pull from kafka topic in scala

I'm using a spark-shell instance to test the pulling of data from a client's kafka source. To launch the instance I am using the command spark-shell --jars spark-sql-kafka-0-10_2.11-2.5.0-palantir.8.jar, kafka_2.12-2.5.0.jar, kafka-clients-2.5.0.jar…
Cam
  • 2,026
  • 3
  • 25
  • 42
1
vote
0 answers

job execution in spark

I am very much new to spark and was trying to check the DAG creation in spark UI using : when I am reading the simple csv file using the command val df = spark.read.format("csv").option("header", "true").load("/home/user/test.csv") then in spark…
AGaur
  • 185
  • 1
  • 1
  • 15
1
vote
1 answer

Livy create session dead

I added to my spark config a package (in spark-default.conf) but when I create a new session with livy it causes me a problem (see the error below) and the session and death . ps: when I remove this package all work fine . 20/05/04 00:17:35 WARN…
zekri sidi mohamed hicham
  • 57
  • 1
  • 10
1 2
3
…
8 9