0

I'm trying to integrate Shark 0.9.1 (for hadoop 1) with hive on datastax enterprise 4.0.3 Hadoop node. I have successfully intsall and configure Scala 2.10.3 and Spark 1.0.0. Scala and sparks shells are also working fine. Now when i'm trying open Shark shell with SHARK_HOME/bin/shark then I'm getting error

 14/06/26 10:04:14 DEBUG metastore.CassandraHiveMetaStore: Creating CassandraHiveMetaStore
 14/06/26 10:04:15 ERROR config.DatabaseDescriptor: Fatal configuration error
    org.apache.cassandra.exceptions.ConfigurationException: Cannot locate cassandra.yaml
    at org.apache.cassandra.config.YamlConfigurationLoader.getStorageConfigURL(YamlConfigurationLoader.java:64)
    at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:75)
    at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:135)
    at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:111)
    at org.apache.cassandra.service.StorageService.getPartitioner(StorageService.java:148)
    at org.apache.cassandra.service.StorageService.<init>(StorageService.java:142)
    at org.apache.cassandra.service.StorageService.<clinit>(StorageService.java:144)
    at com.datastax.bdp.hadoop.hive.metastore.CassandraClientHolder.<init>(CassandraClientHolder.java:69)
    at com.datastax.bdp.hadoop.hive.metastore.SchemaManagerService.<init>(SchemaManagerService.java:111)
    at com.datastax.bdp.hadoop.hive.metastore.CassandraHiveMetaStore.setConf(CassandraHiveMetaStore.java:114)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.hive.metastore.RetryingRawStore.<init>(RetryingRawStore.java:64)
    at org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:73)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:415)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:402)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:286)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4060)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2136)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2147)
    at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1043)
    at shark.memstore2.TableRecovery$.reloadRdds(TableRecovery.scala:49)
    at shark.SharkCliDriver.<init>(SharkCliDriver.scala:283)
    at shark.SharkCliDriver$.main(SharkCliDriver.scala:162)
    at shark.SharkCliDriver.main(SharkCliDriver.scala)
 Cannot locate cassandra.yaml
  Fatal configuration error; unable to start. See log for stacktrace.

Can anyone faced this kind of issue. Please suggest . Thanks

ChrisGPT was on strike
  • 127,765
  • 105
  • 273
  • 257
user3632180
  • 105
  • 2
  • 13

1 Answers1

1

Try DataStax Enterprise 4.5, it has built-in support for Spark and Shark.

Jacek L.
  • 1,376
  • 14
  • 19
  • I have upgraded the DSE to 4.5 but I'm not able to open spark master. http://stackoverflow.com/questions/24530266/unable-to-run-spark-master-in-dse-4-5-and-slaves-file-is-missing?noredirect=1#comment38053193_24530266 – user3632180 Jul 04 '14 at 09:10