Installing splice machine on HDP 2.6.5

Question

We tried almost all the guides that we can find online to splice machine as a ambari service.

But everytime we run the sqlshell.sh, it just says there is no server running and unable to connect to port 1527 on localhost.

We have a simple HDP sandbox version 2.6.5 and a three node 2.6.5 cluster. We are trying to install version 2.87 of splicemachine.

These are the guides we followed.

https://github.com/splicemachine/spliceengine/blob/branch-2.8/platforms/hdp2.6.5/docs/HDP-installation.md

That did not work on our three node cluster

Then we tried the sandbox with this tutorial

https://github.com/splicemachine/splice-ambari-service

Again the same result.

Please let us know if there is anything that we have missed in the guide/ or are there any extra steps.

score 0 · Answer 1 · answered Apr 09 '20 at 13:01

0

Can you confirm you are running sqlshell.sh on a server that is running a region server? I would look at the region server log files. You want to look for 'Ready to accept JDBC connections on 0.0.0.0:1527'. That is the indicator that a region server is up and running. If you do not see that, then can you look in the log file and see if there are any error messages?

answered Apr 09 '20 at 13:01

Erin

395
3
7

Thanks for the reply. We were able to see that exact message once this dependency issue was resolved. – Sharath Prakash Apr 09 '20 at 17:04

score 0 · Answer 2 · answered Apr 09 '20 at 17:03

Found out the answer - Apart from what is mentioned in the doc here - https://github.com/splicemachine/spliceengine/blob/branch-2.8/platforms/hdp2.6.5/docs/HDP-installation.md.

There were multiple NoClassDefFoundError errors being thrown in the hbase master logs for classes found in spark2 jars.

One of them was

    master.HMaster: The coprocessor com.splicemachine.hbase.SpliceMasterObserver threw java.io.IOException: Unexpected exception
java.io.IOException: Unexpected exception
        at com.splicemachine.si.data.hbase.coprocessor.CoprocessorUtils.getIOException(CoprocessorUtils.java:30)
        at com.splicemachine.hbase.SpliceMasterObserver.start(SpliceMasterObserver.java:111)
        at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:415)
        at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:256)
        at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:159)
        at org.apache.hadoop.hbase.master.MasterCoprocessorHost.<init>(MasterCoprocessorHost.java:93)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:774)
        at org.apache.hadoop.hbase.master.HMaster.access$900(HMaster.java:225)
        at org.apache.hadoop.hbase.master.HMaster$3.run(HMaster.java:2038)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: org/spark_project/guava/util/concurrent/ThreadFactoryBuilder
        at com.splicemachine.timestamp.impl.TimestampClient.<init>(TimestampClient.java:108)
        at com.splicemachine.timestamp.hbase.ZkTimestampSource.initialize(ZkTimestampSource.java:62)
        at com.splicemachine.timestamp.hbase.ZkTimestampSource.<init>(ZkTimestampSource.java:48)
        at com.splicemachine.si.data.hbase.coprocessor.HBaseSIEnvironment.<init>(HBaseSIEnvironment.java:146)
        at com.splicemachine.si.data.hbase.coprocessor.HBaseSIEnvironment.loadEnvironment(HBaseSIEnvironment.java:100)
        at com.splicemachine.hbase.SpliceMasterObserver.start(SpliceMasterObserver.java:81)
        ... 8 more
Caused by: java.lang.ClassNotFoundException: org.spark_project.guava.util.concurrent.ThreadFactoryBuilder
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 14 more

We found out that this class was in the spark-network-common jar and symlinked it to one of the folders that was part of HBASE_CLASSPATH. But then it threw another classdef not found error for a different class.

We ended up symlinking all the jars in spark2/jars folder to one of the folders that was part of HBASE_CLASSPATH.

After this the hbase master came up successfully and started the splice db process and we were able to connect with the sqlshell.sh

Nots: Make sure to do this symlinking on all nodes that have a regionserver.

What helped us in figuring this out were these two java files in splice documentation

https://github.com/splicemachine/spliceengine/blob/master/hbase_sql/src/main/java/com/splicemachine/hbase/SpliceMasterObserver.java

and

https://github.com/splicemachine/spliceengine/blob/ee61cadf17c97a0c5d866e2b764142f8e55311a5/splice_timestamp_api/src/main/java/com/splicemachine/timestamp/impl/TimestampServer.java

Installing splice machine on HDP 2.6.5

2 Answers2