0

We are using the method UserGroupInformation.loginUserFromKeytabAndReturnUGI(user, keytab) to authenticate a Java program to write to a remote HBase cluster. When the application first starts up we are all good and it's talking to HBase happily.

The krb5.conf ticket_lifetime is set to 24 hours, and what seems to happen after 24 hours is that the "TGT expires" and we start seeing exceptions like this: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

We don't have any relogin logic. I always thought logging in from keytab shouldn't require us to write any additional code to relogin since it's handled by the RpcClient. But here are a few weird things:

  1. Looks like some mechanism to relogin is kicking in but not doing anything, I'm not sure what method it tries to use org.apache.hadoop.security.UserGroupInformation Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

  2. When initially authenticated via keytab, isFromKeytab() returns FALSE! I wonder if this is why it's not trying to use reloginFromKeytab() and tries to look at the cache?

Should we try to catch the exception and use checkTGTAndReloginFromKeytab() or do something like this?

         if (UserGroupInformation.isLoginKeytabBased()) {
            UserGroupInformation.getLoginUser().reloginFromKeytab();
          } else if (UserGroupInformation.isLoginTicketBased()) {
            UserGroupInformation.getLoginUser().reloginFromTicketCache();
          }

Any help is appreciated!

Jason

jastang
  • 386
  • 5
  • 14
  • Duplicate of http://stackoverflow.com/questions/33211134/hbase-kerberos-connection-renewal-strategy – Samson Scharfrichter Jul 06 '16 at 10:23
  • Thanks, @SamsonScharfrichter. Your other post was helpful. – jastang Jul 06 '16 at 20:00
  • @SamsonScharfrichter does checkTGTAndReloginFromKeytab() rely on a jaas.conf? ugi.isFromKeytab() keeps returning false for me. – jastang Jul 13 '16 at 14:19
  • You should search the source code to be sure, but if I remember well, the `reloginFromKeytab()` methods *require* a previous `loginFromKeytab(keytab_file)` and a *private* Kerberos ticket. In other words, you set the keytab at login, then it is stored in the UGI and used whenever a *private* TGT has to be created/renewed/recreated. It will not work if the *public* ticket was used at login time. – Samson Scharfrichter Jul 13 '16 at 15:46
  • @SamsonScharfrichter ended up using your shell suggestion for now. I'm on Java 8 and the relogin methods weren't doing it for me. i find some consolation in the fact that UserGroupInformation executes "kinit -R" directly in its renewal thread :D – jastang Jul 15 '16 at 20:50
  • check the compatibility matrix for **Hadoop version** (core libs) vs. **JDK version** -- you need at least Hadoop 2.7 *(or a patched 2.6 as in recent CDH releases)* for JDK 1.8 *(and later releases of OpenJDK 1.7 because they accidentally back-ported some 1.8 code in these)*; otherwise ticket renewal won't work due to vicious changes in the internals of JAAS. – Samson Scharfrichter Jul 17 '16 at 10:45

0 Answers0