HIVE Table available in SPARK but does not show up in Hive CLI

Question

This question is a possible duplicate of this one but the answers given are not satisfactory.

I 've ran the following simple code on Zeppelin : (Same scenario with pyspark CLI as well)

%spark2.pyspark
from pyspark.sql import HiveContext
sqlContext = HiveContext(sc)

df = sqlContext.read.format("csv").option("header", "false").option("mode", "DROPMALFORMED").load("/data/data1.csv")
df.write.mode('overwrite').saveAsTable("default.hive_spark");

Then :

%spark2.pyspark
sqlDF = spark.sql("show tables")
sqlDF.show()

It shows :

+--------+----------------+-----------+
|database|       tableName|isTemporary|
+--------+----------------+-----------+
| default|      hive_spark|      false|
+--------+----------------+-----------+

But when I login to the HIVE CLI (user:hive) this table does not show up :

0: jdbc:hive2://ip-xxx.eu-west-3.com>USE default;
0: jdbc:hive2://ip-xxx.eu-west-3.com>SHOW TABLES;

+-----------+
| tab_name  |
+-----------+
| hive_test |
+-----------+

I tried

sqlContext.uncacheTable("default.hive_spark")

I am confused.

I guess it might have something to do with users / permissions / ACLs... — Mehdi LAMRANI, Oct 17 '18 at 00:02
The problem might be in different `default` database locations, though if you add hive-site.xml to hive conf directory then the location should be the same. Try also `msck repair table default.hive_spark` in Hive CLI. — serge_k, Oct 17 '18 at 08:54
I didn't think `saveAsTable` generated Hive compatible tables — OneCricketeer, Oct 17 '18 at 12:35

score 0 · Answer 1 · answered Jun 04 '20 at 03:37

0

Use HiveWareHouseConnector since you are working with hive2

answered Jun 04 '20 at 03:37

Bishamon Ten

479
1
6
20

HIVE Table available in SPARK but does not show up in Hive CLI

1 Answers1