Stack Overflow
Stack Overflow

Questions tagged [hivecontext]

Questions related to HiveContext class of Apache Spark

A variant of Spark SQL that integrates with data stored in Hive.

Configuration for Hive is read from hive-site.xml on the classpath. It supports running both SQL and HiveQL commands.

More documentation can be found:

106 questions
0
votes
0 answers

Not able to connect to Hive from Spark-Sql

I am trying to connect to Hive using Spark-SQL, but its not able to read tables from hive. Spark Version : 2.4.0 Am trying to connect to table that is in default Database in Hive. But its throwing below…
vasanth sonu
  • 13
  • 4
0
votes
1 answer

Oozie Spark Action (Containing HiveContext) Giving java.lang.OutOfMemoryError: PermGen space

I'm trying to run a spark-scala Self-Contained App in Oozie. Please note that I'm using CDH5.13 Quickstart VM with 20G of RAM (containing Cloudera Manager, HUE ..., and I uppgraded Java from 7 to 8). The code does pretty much nothing, it just create…
El Mehdi OUAFIQ
  • 152
  • 1
  • 13
0
votes
0 answers

Create hive table in pyspark hive context

I have 3 tables in abc hive database with Avro format. I want to create the another database(def) and create those 3 tables in hivecontext pyspark through data frames. More info: in abc database 3 tables all columns will b string but while I create…
RRRR
  • 1
0
votes
1 answer

Spark2 unable to find table or view on remote hdfs cluster

I am using HiveContext to query a hive table on a hdfs cluster remotely through spark 1.6.0 and am able to do so successfully. However, when doing so through spark 2.3.0, throws the following: org.apache.spark.sql.AnalysisException: Table or view…
udit
  • 101
  • 3
  • 17
0
votes
1 answer

Query hive database using hive context created on spark 2.3.0

I am able to create a hive context programmatically on spark 1.6.0 using : val conf = new SparkConf().setAppName("SparkTest").setMaster("local") val sc=new SparkContext(conf) val hc = new HiveContext(sc) val actualRecordCountHC = hc.sql("select…
udit
  • 101
  • 3
  • 17
0
votes
1 answer

registerTempTable() doesn't register all records

I tried to create a function which would get the data from relational database and insert them into Hive table. Since I use Spark 1.6, I need to register a temporary table, because writing dataframe directly as Hive table is not compatible with…
regiea
  • 121
  • 1
  • 5
  • 18
0
votes
2 answers

Spark HiveContext vs HbaseContext?

I have a data-set of size 10 Petabytes. My current data is in HBase where I am using Spark HbaseContext but it is not performing well. Will it be useful to move data from HbaseContext to HiveContext on Spark?
Amit khandelwal
  • 504
  • 5
  • 6
0
votes
1 answer

How to use hiveContext in streaming from executors?

I'm trying to use an instance of HiveContext in a Spark streaming application (1.6), but it fails with the following exception: java.lang.NullPointerException at org.apache.spark.sql.SQLConf.getConf(SQLConf.scala:638) In particolar, I…
Marco Catalano
  • 199
  • 14
0
votes
2 answers

Unable to execute Hive queries using spark-submit

I am not able run hive queries using spark-submit command. But, the same is getting executed in spark-shell. I am using AWS EMR as the cluster. Below is my code written in eclipse scala IDE object HiveTest { def main(args: Array[String]): Unit…
Vinay Kumar Dudi
  • 137
  • 2
  • 12
0
votes
1 answer

How to Stop or Delete HiveContext in Pyspark?

I'm facing the following problem: def my_func(table, usr, psswrd): from pyspark import SparkContext, SQLContext, HiveContext, SparkConf sconf = SparkConf() sconf.setAppName('TEST') sconf.set("spark.master", "local[2]") sc =…
Indy McCarthy
  • 51
  • 9
0
votes
2 answers

calculate median, average using hadoop spark1.6 dataframe, Failed to start database 'metastore_db'

spark-shell --packages com.databricks:spark-csv_2.11:1.2.0 1. using SQLContext ~~~~~~~~~~~~~~~~~~~~ 1. import org.apache.spark.sql.SQLContext 2. val sqlctx = new SQLContext(sc) 3. import sqlctx._ val df =…
Chandan Kumar Behera
  • 1
  • 2
0
votes
1 answer

spark 1.6 hive context setConf Issue

I am having trouble to run a sql which loads data to partition table in hive context , I did set dynamic partition = true but still I am having issue. SQL: insert overwrite table target_table PARTITION (column1,column2) select * , deletion_flag …
sri hari kali charan Tummala
  • 1,086
  • 3
  • 16
  • 40
0
votes
1 answer

Unable to append data into existing hive table with HiveContext

We are reading data from a hive table with hiveContext using a spark dataframe. After doing some aggregations on the data we store this data into another table (which already has data). But the new data is not being appended to the existing table...…
somu
  • 1
0
votes
1 answer

Unable to run hive by changing hive-site.xml to connect with spark-HiveContext

Below is my hive/conf/hive-site.xml: javax.jdo.option.ConnectionURL jdbc:mysql://127.0.0.1/metastore?createDatabaseIfNotExist=true metadata is stored in a MySQL…
Ku002
  • 117
  • 1
  • 2
  • 14
0
votes
1 answer

Comparing 2 dataframes in spark when using hivecontext for 1 dataframe and sqlcontext for the other

when i am storing hive table in one dataframe using HiveContext and DB2 table in another dataframe using sqlcontext on querying both the dataframes it is not detecting the Db2 while it detects hive. What is a common sqlcontext that can be used?
Prishi Kumar
  • 81
  • 1
  • 2
  • 9
1 2 3 4
5
6 7 8