Stack Overflow
Stack Overflow

Questions tagged [hivecontext]

Questions related to HiveContext class of Apache Spark

A variant of Spark SQL that integrates with data stored in Hive.

Configuration for Hive is read from hive-site.xml on the classpath. It supports running both SQL and HiveQL commands.

More documentation can be found:

106 questions
0
votes
0 answers

Spark 1.5 on Ubuntu - HiveContext does not work

I deployed an hadoop cluster with bdutil on google compute engine. My configuration - OS: Ubuntu 14 - Spark: 1.5 - Hive: 0.12 - 1 master node and 2 workers Hive metastore configuration: I copied the hive-site.xml from hive to…
Raouf
  • 989
  • 2
  • 11
  • 15
0
votes
0 answers

Spark Hivecontext not executing the query when run from Scala

I have a Scala program wherein I create a hivecontext and run a hive query from it. The hive query is a multi-table insert query. I am facing issues where the query runs successfully but I cannot see the out being generated. Interesting thing is…
Sriram
  • 53
  • 7
0
votes
1 answer

HiveContext not working from Oozie

I was trying to run some hive queries from my spark code using the HiveContext object. It was working fine when I run it from the command line using spark submit. But, I am getting the below exception when I try to run the same from a oozie…
yAsH
  • 3,367
  • 8
  • 36
  • 67
0
votes
1 answer

CDH5.4.2 spark can use HiveContent in spark-shell but can't open spark-sql

I am using spark(standalone) of CDH5.4.2 After copying hive-site.xml to $SPARK_HOME/conf,i can query from hive in spark-shell,such as below: scala> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc); hiveContext:…
陈志明
  • 21
  • 6
0
votes
1 answer

Pyspark: Getting current_timestamp in dynamic hive query

I am preparing Spark with python program which inserts data from 2 tables based on joins. The last column of the target table has a timestamp field which will have the value of create timestamp. I tried current_timestamp and…
Aavik
  • 967
  • 19
  • 48
0
votes
0 answers

Casting string to bigint in hive returns incomplete data

How do I get the full value in hive? i.e. cast('0024' as bigint) will give only 24 but I need the complete 0024 as my output. Any idea how to do this?
user6510278
  • 1
0
votes
0 answers

hivecontext in spark

I am trying to use hivecontext in spark. The following statements are running fine : from pyspark.sql import HiveContext sqlContext = HiveContext(sc) But, when i run the below statement, sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT,…
Zack
  • 2,078
  • 10
  • 33
  • 58
0
votes
0 answers

How can I update or delete records of hive table from spark , with out loading entire table in to dataframe?

I have a hive orc table with around 2 million records , currently to update or delete I am loading entire table in to a dataframe and then update and save as new dataframe and saving this by Overwrite mode(below is command),so to update single…
sudhir
  • 1,387
  • 3
  • 25
  • 43
0
votes
1 answer

Can we DELETE whole table in hive's latest version?

there is table "student" created in hive 0.14 version. i want to delete that table. can use DELETE command for that?
Swapnil Dixit
  • 101
  • 1
  • 8
0
votes
0 answers

Spark SQL :Job aborted due to stage failure: java.io.InvalidClassException: SpecificMutableRow; local class incompatible

I was trying to retrieve hive tables from Spark scala (Both Hive Cluster , spark cluster and my local system have same JDK version 8) var tableData= objHiveContext.hql("select * from intervalmeterdemandbymonth c") println(tableData) //Line1 var…
sudhir
  • 1,387
  • 3
  • 25
  • 43
0
votes
1 answer

Spark Scala : JniBasedUnixGroupsMapping: Error getting groups for XXX: The user name could not be found

I have been connecting to Hive (which is in cluster) from my local system with spark in scala IDE , finally placed my hive-site.xml properly in Spark/conf folder and in class path and able to connect to metastore but unable to access hive…
sudhir
  • 1,387
  • 3
  • 25
  • 43
0
votes
2 answers

Uisng Hive Context , Creating Hive Table locally in local system metastore_db instead on Cluster, where to place my hive-site.xml

I have Created a Spark Context object , and tried retrieving text from text file on hadoop server (Not in my local) and I was able to retrieve it. When I'm trying to retrieve Hive Table (which is on a stand alone machine, cluster) I was unable to…
sudhir
  • 1,387
  • 3
  • 25
  • 43
0
votes
1 answer

spark hivecontext working with queries issues

I'm trying to get information from Jsons to create tables in Hive. This is my Json schema: root |-- info: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- stations: array (nullable = true) | | | |--…
Sebastian Loeb Sucre
  • 71
  • 1
  • 1
  • 5
0
votes
1 answer

Query existing Hive tavle using PySpark.sql.HiveContext

I want to build query on Hive Table (table name : 'sample') using pyspark sql. Following is the simple pyspark code I compiled on pyspark shell From pyspark.sql import HiveContext sqlContext = HiveContext(sc) sqlContext.sql("SELECT * FROM…
Sandeep Reddy
  • 61
  • 3
-1
votes
1 answer

how to apply Windows function in HiveQL in spark

I have seen posts discussing the usage of windows function. But i have some questions. Since it is can only be used in HiveContext. How can i switch between SparkSQLContext and HiveContext given i am already using SparkSQLContext? How is that…
MYjx
  • 4,157
  • 9
  • 38
  • 53
1 2 3 4 5 6
7
8