Stack Overflow
Stack Overflow

Questions tagged [spark-hive]

Used when using spark-hive module or HiveContext

Apache Spark Hive is a module for for "Hive and structured data processing" on Spark, a fast and general-purpose cluster computing system. It is the super set of Spark SQL and is used to create HiveContext, similar to SqlContext.

76 questions
0
votes
0 answers

Apache Spark with Hive on Eclipse IDE throw the privilege error - A read-only database issue

I try to test Apache Spark with Hive integration on Eclipse IDE. These are the versions of each projects - Hadoop 2.7.4, Spark 2.2 and hive-2.3.2 with MySQL 5.7 on Eclipse Mars IDE. The contents of hive-site.xml is like below, …
Joseph Hwang
  • 1,337
  • 3
  • 38
  • 67
0
votes
1 answer

How do I save spark.writeStream results in hive?

I am using spark.readStream to read data from Kafka and running an explode on the resulting dataframe. I am trying to save the result of the explode in a Hive table and I am not able to find any solution for that. I tried the following method but…
Divya
  • 13
  • 1
  • 4
0
votes
1 answer

Running Dependent Queries with SparkSQL using Spark Session

We have 3 queries which are currently running on HIVE. Using Spark 2.1.0 We are trying to Run that using Spark SQL but by using the SparkSession(like wrapping with Scala code making a Jar & then Submit using Spark-Submit) Now for Example lets say…
AJm
  • 993
  • 2
  • 20
  • 39
0
votes
1 answer

The declared package "org.apache.hive.service.cli.thrift" does not match the expected package "java.org.apache.hive.service.cli.thrift"

I import the spark source to eclipse: But I got some error: The declared package "org.apache.hive.service.cli.thrift" does not match the expected package "java.org.apache.hive.service.cli.thrift" What should I do?
Smith
  • 13
  • 1
  • 6
0
votes
1 answer

The constructor HiveContext(JavaSparkContext) is undefined error while create SQLContext object

I'm trying to create SQL context object using JavaSparkContext object as it's parameter like: SparkConf sparkConf=new SparkConf().setMaster("local").setAppName("Example"); JavaSparkContext sc=new JavaSparkContext(sparkConf); SQLContext sqlctx=new…
kaushik3993
  • 105
  • 1
  • 3
  • 10
0
votes
1 answer

Spark Hive: Can't retrieve column of DataFrame

I'm trying around with Spark on Hive. In the code I create a new DataFrame and fill it with custom data by using the HiveContext.createDataFrame method: JavaSparkContext sc = ...; HiveContext hiveCtx = new HiveContext(sc); StructField f1 = new…
D. Müller
  • 3,336
  • 4
  • 36
  • 84
0
votes
1 answer

object HiveContext in package hive cannot be accessed in package

HI Coders, I'm back again. I'm trying to create a hive table from a dataframe using HIve context in my scala code, im able to do it in sqlContext but when it comes to HiveContext, it is throwing this error [error]…
jack AKA karthik
  • 885
  • 3
  • 15
  • 30
0
votes
0 answers

Spark Dataframe OOM error when writing into Hive

I have an application used to put Spark dataframe data into Hive. The first time, the application use 100 cores and 10 GB of memory producing this OutOfMemory error after leaking a lot of 32 Mb chunks. After that I run the application with 100…
spoon
  • 41
  • 5
0
votes
0 answers

Spark Hivecontext not executing the query when run from Scala

I have a Scala program wherein I create a hivecontext and run a hive query from it. The hive query is a multi-table insert query. I am facing issues where the query runs successfully but I cannot see the out being generated. Interesting thing is…
Sriram
  • 53
  • 7
0
votes
0 answers

Casting string to bigint in hive returns incomplete data

How do I get the full value in hive? i.e. cast('0024' as bigint) will give only 24 but I need the complete 0024 as my output. Any idea how to do this?
user6510278
  • 1
0
votes
1 answer

Why is my SparkSQL query not working where as HIVE returns data

I have a bunch of csv files stored in the blob storage that contains records like this: 2016-04-19 20:26:01.0299,+05:30,ecc84966-9bc0-4bef-9cd2-ad79c25be278,test001,178.03499442294,,Good 2016-04-19…
Kiran
  • 2,997
  • 6
  • 31
  • 62
0
votes
3 answers

Creating External Table in Hive from Spark

I am getting below error while trying to create external table in Hive from Spark and saving the contents . The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw- I even tried setting permission to /tmp/hive…
Amit_Hora
  • 716
  • 1
  • 8
  • 27
0
votes
0 answers

GenericUDF of hive execute twice on Spark

Hello i facing some problem with creating genericUDF of hive and register as temporary function but when i call it its call twice see code given below i create a genericUDF with following code class GenUDF extends GenericUDF{ var queryOI:…
Sandeep Purohit
  • 3,652
  • 18
  • 22
0
votes
1 answer

MongoHadoop Connector used with Spark duplicates results by number of partitions

I am trying to read data into spark using the mongo-hadoop connector. The problem is that if I am trying to set a limit regarding the data read, I get in the RDD the limit * the number of…
user3452075
  • 411
  • 1
  • 6
  • 17
0
votes
0 answers

How can I update or delete records of hive table from spark , with out loading entire table in to dataframe?

I have a hive orc table with around 2 million records , currently to update or delete I am loading entire table in to a dataframe and then update and save as new dataframe and saving this by Overwrite mode(below is command),so to update single…
sudhir
  • 1,387
  • 3
  • 25
  • 43
1 2 3 4
5
6