6

Background:-

  • I am using HDP with spark1.6.0 and hive 1.2.1

Steps Followed:-

Create a hive table:-

hive>
CREATE TABLE orctest(PROD_ID bigint, CUST_ID bigint, TIME_ID timestamp, CHANNEL_ID bigint, PROMO_ID bigint, QUANTITY_SOLD decimal(10,0), AMOUNT_SOLD decimal(10,0) ) CLUSTERED BY (PROD_ID) INTO 32 BUCKETS STORED AS ORC TBLPROPERTIES ( "orc.compress"="SNAPPY", "transactional"="true" );

Insert records into orctest

hive>
insert into orctest values(1, 1, '2016-08-02 21:36:54.000000000', 1, 1, 10, 10000);

Try to access the orctest table from spark-shell

scala>
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)

val s = hiveContext.table("orctest")*

Exception Thrown:-

16/08/02 22:06:54 INFO OrcRelation: Listing hdfs://hadoop03:8020/apps/hive/warehouse/orctest on driver
16/08/02 22:06:54 
INFO OrcRelation: Listing hdfs://hadoop03:8020/apps/hive/warehouse/orctest/delta_0000005_0000005 on driver
**java.lang.AssertionError: assertion failed**
at scala.Predef$.assert(Predef.scala:165)
at org.apache.spark.sql.execution.datasources.LogicalRelation$$anonfun$1.apply(LogicalRelation.scala:39)
at org.apache.spark.sql.execution.datasources.LogicalRelation$$anonfun$1.apply(LogicalRelation.scala:38)
at scala.Option.map(Option.scala:145)
at org.apache.spark.sql.execution.datasources.LogicalRelation.<init>(LogicalRelation.scala:38)
at org.apache.spark.sql.execution.datasources.LogicalRelation.copy(LogicalRelation.scala:31)
at org.apache.spark.sql.hive.HiveMetastoreCatalog.org$apache$spark$sql$hive$HiveMetastoreCatalog$$convertToOrcRelation(HiveMetastoreCatalog.scala:588)

Any help will be really appreciated.

zero323
  • 322,348
  • 103
  • 959
  • 935
Priyanka.Patil
  • 1,177
  • 2
  • 15
  • 34
  • Have a look http://stackoverflow.com/questions/27171702/error-in-scala-compiler-java-lang-assertionerror-assertion-failed-even-when-p – BruceWayne Aug 03 '16 at 11:02
  • Thanks krishna for your comment. But I have not tried this with a scala project, I directly tried this in spark-shell. My thoughts:- If you create a hive table with transnational property as true then you can't able to access the content of the table though spark.(Please correct me if i was wrong) PS: I am using HDP(Spark 1.6.0 and hive 1.2.1) – Priyanka.Patil Aug 04 '16 at 05:23
  • I too am facing a similar issue. I cannot load a transaction enabled table to a spark dataframe. Have you found a workaround for this problem? I have tried setting hive transaction properties using sqlContext.setConf() before trying to load. I also tried creating a view on the source transaction table and hitting the view from spark but to no avail – Nitin Kumar Oct 14 '16 at 07:53

1 Answers1

2

Try setting : hiveContext.setConf("spark.sql.hive.convertMetastoreOrc", "false")

Blue
  • 22,608
  • 7
  • 62
  • 92
sandyyyy
  • 75
  • 11