3

I have a small spring service, which provides basic functionality like put/delete/get from hbase table. Everything seems to work, but there is one problem. After 10 hours after starting my tomcat server, my kerberos ticket expires, so I should renew it. I tried to use java api for hbase and in my code in every method, which connects to hbase I added this line:

UserGroupInformation.getLoginUser().checkTGTAndReloginFromKeytab();

I tried also with: 

UserGroupInformation.getLoginUser().reloginFromKeytab()

and:

SecurityUtil.login(configuration, keytabFilePath, kerberosUser)

but it doesn't help and I get this exception 10 hours after server restart: 

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(Unknown Source)
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:34142)
at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:128)
at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:53)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:733)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

I found also in logs these few lines:

2017-01-03 19:09:16 DEBUG UserGroupInformation:1638 - PrivilegedAction as:user@MAIL.COM (auth:KERBEROS) from:org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
2017-01-03 19:09:16 DEBUG UserGroupInformation:1618 - PrivilegedActionException as:user@MAIL.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2017-01-03 19:09:16 DEBUG UserGroupInformation:1638 - PrivilegedAction as:user@MAIL.COM (auth:KERBEROS) from:org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:643)
2017-01-03 19:09:16 WARN  UserGroupInformation:1113 - Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

Any hits or tips what I did wrong?

EDIT: I use this code to authenticate using kerberos: 

Configuration configuration = HBaseConfiguration.create();
configuration.addResource("some config file");
UserGroupInformation.setConfiguration(configuration);
UserGroupInformation.loginUserFromKeytab(kerberosUser, keytabFilePath);

I change version of hadoop dependencies to 2.6.5 in my project.

Marcin
  • 508
  • 2
  • 10
  • 28
  • Have you tried the solution http://stackoverflow.com/a/34691071/2443502 yet? Although it's Hadoop related but Hbase uses Hdoop's security API so you should give a try. – Marcin Kłopotek Jan 04 '17 at 08:56
  • yep, in my case the solution should look like in this answer: http://stackoverflow.com/questions/33211134/hbase-kerberos-connection-renewal-strategy/33243360#33243360. – Marcin Jan 04 '17 at 09:07
  • 1
    Check also http://stackoverflow.com/questions/41087997/auto-renewal-of-kerberos-ticket-not-working-from-java *(in case you don't use the default UGI)* and the comments under http://stackoverflow.com/questions/38213377/hbasekerberos-usergroupinformation-logging-in-from-keytab-expires *(i.e. "relogin" requires that you "login" first!)* – Samson Scharfrichter Jan 04 '17 at 10:56
  • I provided some more information. – Marcin Jan 10 '17 at 14:28

1 Answers1

4

A final solution, which seems to work: 1. Upgrade Hadoop dependencies to 2.6.5 version (hadoop-auth, hadoop-mapreduce-client-core, hadoop-common). 2. Create simple scheduler, which executes the code below every 5-6 minutes. 3. Execute code below before every action with a HBase database.

UserGroupInformation.getLoginUser().checkTGTAndReloginFromKeytab();
Marcin
  • 508
  • 2
  • 10
  • 28