Stack Overflow
Stack Overflow

Questions tagged [metastore]

118 questions
14
votes
3 answers

Where is an Avro schema stored when I create a hive table with 'STORED AS AVRO' clause?

There are at least two different ways of creating a hive table backed with Avro data: Creating a table based on an Avro schema (in this example, stored in hdfs): CREATE TABLE users_from_avro_schema ROW FORMAT SERDE…
tomek
  • 771
  • 1
  • 7
  • 19
8
votes
1 answer

Spark build in hive MySQL metastore isn't being used

I'm using Apache Spark 2.1.1 and I have put the following hive-site.xml on $SPARK_HOME/conf folder: javax.jdo.option.ConnectionURL …
José Carlos Guevara Turruelles
  • 481
  • 8
  • 23
7
votes
2 answers

Hive, how do I retrieve all the database's tables columns

I want to write the equivalent of this sql request in Hive : select * from information_schema.columns where table_schema='database_name' How can I access hive's metastore and retrieve all the columns of all the tables stored in a specific…
Erij Tounsi
  • 91
  • 1
  • 1
  • 2
6
votes
2 answers

Error creating transactional connection factory during running Spark on Hive project in IDEA

I am trying to setup a develop environment for a Spark Streaming project which requires write data into Hive. I have a cluster with 1 master, 2 slaves and 1 develop machine (coding in Intellij Idea 14). Within the spark shell, everything seems…
Shirui Tang
  • 63
  • 1
  • 1
  • 6
6
votes
2 answers

How to set configuration in Hive-Site.xml file for hive metastore connection?

I want to connect MetaStore using the java code. I have no idea how to set configuration setting in Hive-Site.xml file and where I'll post the Hive-Site.xml file. Please help. import java.sql.Connection; import java.sql.DriverManager; import…
mohit sharma
  • 259
  • 1
  • 4
  • 9
5
votes
1 answer

Hadoop Metastore Will Not Initialize

preamble: i'm new to hadoop / hive. have installed standalone hadoop and now am trying to get hive to work. i keep getting an error about initializing the metastore and cannot seem to figure out how to resolve. (hadoop 2.7.2 and hive…
RJ Cole
  • 2,592
  • 3
  • 18
  • 23
4
votes
2 answers

Can not ALTER or DROP a big Imapa partitionned tables - CAUSED BY: MetaException: Timeout when executing

I have a several impala partitionned tables that have more than 50k partitions, it work a good except the Hive Metastore operations, like DROP and ALTER ... RENAME, I face this error message: Query: drop table cars ERROR: ImpalaRuntimeException:…
Mohammed Acharki
  • 234
  • 2
  • 14
4
votes
1 answer

Spark and Metastore Relation

I was aware of the fact that Hive Metastore is used to store metadata of the tables that we create in HIVE but why do spark required Metastore, what is the default relation between Metastore and Spark Does metasore is being used by spark SQL, if so…
PhanidharSwarna
  • 125
  • 8
4
votes
0 answers

Spark 2.1 table loaded from Hive Metastore has null values

I am trying to migrate table definitions from one Hive metastore to another. The source cluster has: Spark 1.6.0 Hive 1.1.0 (cdh) HDFS The destination cluster is an EMR cluster with: Spark 2.1.1 Hive 2.1.1 S3 To migrate the tables I did the…
summerbulb
  • 5,709
  • 8
  • 37
  • 83
4
votes
2 answers

external hive metastore issue in EMR cluster

I am pointing my EMR cluster's hive metastore to exteral MySQL RDS instance. I have created new hive database "mydb" and I got the entry in external MySQL DB in hive.DBS table. hdfs://ip-10-239-1-118.ec2.internal:8020/user/hive/warehouse/mydb.db …
sam
  • 85
  • 3
  • 10
4
votes
1 answer

Hive Metastore column width limit

Using AWS EMR on the 5.2.1 version as data processing environment, when dealing with a huge JSON file that has a complex schema with many nested fields, Hive can't process it and errors as it reaches the current limit of 4000 characters column…
blamblam
  • 423
  • 6
  • 20
4
votes
1 answer

Use hive metastore service WITHOUT Hadoop/HDFS

I know the question is a little bit strange. I love Hadoop & HDFS, but recently work on SparkSQL with Hive Metastore. I want to use SparkSQL as a vertical SQL engine to run OLAP query across different datasources like RDB, Mongo, Elastic ...…
He Bai
  • 305
  • 4
  • 12
4
votes
1 answer

What is the metastore for in Spark?

I am using SparkSQL in python. I have created a partitioned table (~few hundreds of partitions) stored it into Hive Internal Table using the hiveContext. The hive warehouse is located in S3. When I simply do "df = hiveContext.table("mytable"). It…
ChromeHearts
  • 1,693
  • 1
  • 17
  • 27
4
votes
3 answers

Hive connectivity to MySQL: Access denied for user 'hive'@'localhost' hive

I am setting up Hive on CentOS, and I have installed hadoop, hive and MySQL on the same server. I have also setup metastore DB on mySQL and user ID - hive has access to the database. I have the below issue - Error Message is given below. Can some…
user3528338
  • 81
  • 1
  • 2
  • 7
4
votes
4 answers

Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

i just installed hive and mysql.. and copied the mysqlconnector to the hive_home/lib folder but when i try show databases and create table commands in the hive> prompt giving me the error as below: create database saty; FAILED: Error in metadata:…
VISHWA
  • 41
  • 1
  • 1
  • 2
1
2 3 4 5 6 7 8