Passing HBase credentials in oozie Java Action

Question

I need to schedule an oozie Java action which interacts with secured hbase, so I need to provide hbase credentials to the Java action. I am using a secured hortonworks 2.2 environment, my workflow XML is as below

<workflow-app xmlns="uri:oozie:workflow:0.4" name="solr-wf">
    <credentials>
         <credential name="hbase" type="hbase">
         </credential>
      </credentials>

    <start to="java-node"/>
    <action name="java-node" cred="hbase">
        <java>  
             <job-tracker>${jobTracker}</job-tracker>
             <name-node>${nameNode}</name-node>
             <main-class>com.test.hbase.TestHBaseSecure</main-class>
            <arg>${arg1}</arg>
        </java>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Java failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

I have also modified the oozie property to include HbaseCredentials Class

oozie.credentials.credentialclasses=hcat=org.apache.oozie.action.hadoop.HCatCredentials,hbase=org.apache.oozie.action.hadoop.HbaseCredentials

But I am not able to run the job it is throwing an error, below is the stacktrace

java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
    at org.apache.oozie.action.hadoop.HbaseCredentials.copyHbaseConfToJobConf(HbaseCredentials.java:60)
    at org.apache.oozie.action.hadoop.HbaseCredentials.addtoJobConf(HbaseCredentials.java:49)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1054)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:913)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1135)
    at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:228)
    at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
    at org.apache.oozie.command.XCommand.call(XCommand.java:281)
    at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
    at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
    at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Other jobs runs fine, it's only the job with hbase interaction which is failing. I have included all the hbase jar in my lib directory, I am not able to figure out the issue.

Updated workflow.xml:

<workflow-app xmlns="uri:oozie:workflow:0.4" name="${appName}">
<credentials>
        <credential name="hbase-cred" type="hbase">
            <property>
                <name>hbase.master.kerberos.principal</name>
                <value>hbase/_HOST@ABC.COM</value>
            </property>

            <property>
                <name>hbase.master.keytab.file</name>
                <value>/etc/security/keytabs/hbase.service.keytab</value>
            </property>

            <property>
                <name>hbase.regionserver.kerberos.principal</name>
                <value>hbase/_HOST@ABC.COM</value>
            </property>

            <property>
                <name>hbase.regionserver.keytab.file</name>
                <value>/etc/security/keytabs/hbase.service.keytab</value>
            </property>

            <property>
                <name>hbase.security.authentication</name>
                <value>kerberos</value>
            </property>

            <property>
                <name>hbase.zookeeper.quorum</name>
                <value>dev1-dn2,dev1-dn3,dev1-dn1</value>
            </property>

            <property>
                <name>zookeeper.znode.parent</name>
                <value>/hbase-secure</value>
            </property>
        </credential>


    </credentials>
    <start to="java-node" />
    <action name="java-node" cred='hbase-cred'>
        <java>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <main-class>com.test.hbase.TestHBaseSecure</main-class>
        </java>
        <ok to="end" />
        <error to="fail" />
    </action>
    <kill name="fail">
        <message>Java failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end" />
</workflow-app>

yes it worked, but after adding the jars there were some connection issue ,i think it was trying to connect to some default host to get the token,i added some property in credential section and it worked , i am adding the modified credential section in the question — Pravin yadav, Jul 29 '16 at 10:12
@RuslanOstafiichuk I have configured as per documentation from oozie site, initially the job was submitted without any issues in cmd prompt but in resource manager UI ,it’s does not shown anything like accepted or running state. Job run for long time and it was suspended due to exception in receiving Hbase credentials org.apache.hadoop.hbase.client.RetriesExhaustedException and no issues were logged at this time . Please suggest me is there any configuration need to set.? Thanks, — mathes, Oct 14 '16 at 05:27
@mathes i had the same issue,it was solved after adding the configuration section in hbase credentail section,refer the updated workflow.xml for required configuration,check it this works for you — Pravin yadav, Oct 14 '16 at 10:16
@CTDex I am facing similar issue right now even after adding proper credentials property in workflow.xml. Can you please share your Java code sample. I need to check if anything I am missing in the Java code. — Padmanabhan Vijendran, Feb 22 '17 at 09:45
@PadmanabhanVijendran there is nothing to do in java code it's the usual code you use to schedule any application. did you add relevant properties for the credentials, refer the updated workflow file. what is the error that you are getting?? — Pravin yadav, Feb 22 '17 at 10:54
@CTDex Thanks.. Your credentials properties worked.. Thanks.. — Padmanabhan Vijendran, Feb 23 '17 at 08:42

score 6 · Accepted Answer · answered Mar 14 '16 at 13:08

This solution was tested on HDP2.2.8:

Copy to /usr/hdp/current/oozie-server/oozie-server/webapps/oozie/WEB-INF/lib the following jars:
- hbase-client-*-hadoop2.jar
- hbase-common-*-hadoop2.jar
- hbase-protocol-*-hadoop2.jar
- hbase-server-*-hadoop2.jar
- htrace-core-2.04.jar
Restart Oozie server.

score 3 · Answer 2 · edited May 23 '17 at 12:25

These "credentials" are managed by the Oozie service, not by your job.

So, if HortonWorks had done a decent job of packaging their distro...

hbase-common-*-hadoop2.jar would have been deployed in /usr/hdp/current/oozie-client/libserver/ on installation
that JAR would not conflict with other JARs over the definition of class org.apache.hadoop.conf.Configuration
and in the end you would be able to manage HBase credentials in Oozie

It is not the case for HDP2.2.4 as installed on our Prod cluster. Arghh. The damn thing is broken in that damn release. You've got to manage the Kerberos ticket all by yourself, downloading a keytab <file> from HDFS and creating the TGT before actually connecting to HBase. We've been there.

Have a look at that post for some insights about how it can be done.

Well, actually, no I can't -- moved to another assignment, don't have access to that Oozie + HBase + Kerberos setup any more. — Samson Scharfrichter, Mar 14 '16 at 17:20

Passing HBase credentials in oozie Java Action

2 Answers2

Linked