2

I am trying to access Hbase table by mapping it from hive through spark engine.

From Hive:

When I run the query on Hive view mapped with Hbase I could get all the desired result.

From Spark:

When i run query to fetch from hive table ,i could get it but when i do the same for hbase mapped hive table i get below error.

Error: java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.initSerdeParams(Lorg/apache/hadoop/conf/Configuration;Ljava/util/Properties;Ljava/lang/String;)Lorg/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe$SerDeParameters; (state=,code=0)

I like to know whether it is possible to do it through spark as I did not find any solutions from internet.

From the above error I could figure that this is some jar issue and this method exists in hive-serde jar and hive-exec jar,but tried with all possible alternative but could not succeed..

Can any one help on this..

Note:My main objective here to check the performance as it is taking good amount of time in hive and pig as well.

Ram Ghadiyaram
  • 28,239
  • 13
  • 95
  • 121
Ranjan Swain
  • 75
  • 1
  • 9

1 Answers1

0

I'm not sure which versions of software you are using hive, hbase & spark ...

Seems like this is version mismatch issue.

Spark 1.3.1 uses Hive 0.13 API, while Spark 1.5.1 uses Hive 1.2, hbase serde that comes with hive 2.3 is not compatible with Hive 0.13 API. I tried with old Serde for Hive 0.13, but there are lots of conflicts with Hbase API versions.

  • Depending on your need, you can try Spark+Hbase native integration instead of Hive-Hbase Serde.
  • Or You can use correct compatible versions of hive.

Also have a look at https://issues.apache.org/jira/browse/HIVE-12406 can be the same issue...

Ram Ghadiyaram
  • 28,239
  • 13
  • 95
  • 121
  • Thanks Ram,I will look into your sugestion for Spark+Hbse integrartion..The versions that i am using is HIve:1.2.1 Hbase:0.98 spark:2.0 hadoop:2.7.2 In hive-serde-1.2 version ,this method is not there.... – Ranjan Swain Sep 21 '16 at 07:55