HBase Kerberos connection renewal strategy

Question

Recently I enabled kerberos in my cluster, everything works great until my kerberos login expires, at say, 12 hours. At that point any connections I have created, any tables created with those connections etc will throw when I use them. This could potentially crash my app depending on how I handle this.

I don't mind crashing hugely because my app is managed by slider which will resurrect the app if and when it goes down, however this will only happen when HBase is "used" (i.e. I call a method on a table with a now stale connection) which will probably be caused by a user interaction and this would lead to poor UX.

I don't want authentication implementation details to pervade my application and also don't want to create connection objects more often than is necessary because it is a costly operation which makes a large number of RPC calls (zookeeper metadata location to start with).

Is there a common strategy (preferably inbuilt in HBase client) for managing kerberos authentication expiry and renewing HBase connections/tables when that happens?

If this is automated, I would highly recommend to use keytabs instead of username/password. — Michael-O, Oct 19 '15 at 09:43
Yeah I am using keytabs. The keytabs refresh, but any Hbase Connection (and anything that uses one of those) will throw after the keytab expiry. I'm looking for a strategy for handling that without relying on periodic failure and without reinventing the wheel. — , Oct 19 '15 at 09:52
The keytab does not expire. The account password might do after some months. Oracle's Kerberos module has a `renewTGT` flag. Did you try that? More over, can you show the exception stack? — Michael-O, Oct 19 '15 at 11:15
catching up on this, Slider pushes out keytabs and HBase should renew them. If this isn't working for you —file a JIRA on issues.apache.org. — stevel, Jan 09 '16 at 18:33
@SteveLoughran ended up just not using slider, this was more of a generic HBase Kerberos question really. No issues found with slider. — , Apr 23 '16 at 10:58

Samson Scharfrichter · Accepted Answer · 2015-11-02T12:59:00.783

30

A Kerberos TGT has a lifetime (e.g. 12h) and a renewable lifetime (e.g. 7 days). As long as the ticket is still valid and is still renewable, you can request a "free" renewal -- no password required --, and the lifetime counter is reset (e.g. 12h to go, again).

The Hadoop authentication library spawns a specific Java thread for automatic renewal of the current TGT. It's kind of ugly, using a kinit -R command line instead of a JAAS library call, but it works - see HADOOP-6656

So, if you get Slider to create a renewable ticket on startup, and if you can bribe your SysAdmin to raise the default (cf. client conf) and the max (cf. KDC conf) renewable lifetime to, say, 30 days, then your app could run for 30 days straight with the initial TGT. A nice improvement.

~~~~~~~~~~

If you really crave for eternity... sorry, but you will actually have some programming to do. That means a dedicated thread/process in charge or re-creating automagically the TGT.

The Java Way: on startup, before you connect to HBase/HDFS/whatever, create explicitly an UGI with loginUserFromKeytab() then run checkTGTAndReloginFromKeytab() from time to time
The Shell Way: start a shell that (a) creates a TGT with kinit (b) spawns a sub-process that periodically fires kinit again (c) launches your Java app then kills the subprocess when/if your app ever terminates

Caveat: if some other thread happens to open, or re-open, a connection while the TGT is being re-created, that connection may fail because the cache was empty at the exact time it was accessed ("race condition"). The next attempt will be successful, but expect a few rogue warnings in your logs.

~~~~~~~~~~

Final advice: you can use a private ticket cache for your app (i.e. you can run multiple apps on the same node with the same Linux account but different Kerberos principals) by setting KRB5CCNAME environment variable, as long as it's a "FILE:" cache.

edited Nov 02 '15 at 12:59

answered Oct 20 '15 at 17:45

Samson Scharfrichter

8,884
1
17
36

1

This is good answer. Would you mind having a quick look on my java question which is related but not fully answered here? http://stackoverflow.com/questions/34616676/should-i-call-ugi-checktgtandreloginfromkeytab-before-every-action-on-hadoop – Jan Zyka Jan 06 '16 at 12:42
@AW, for a long-lived cluster slider needs a keytab; it'll push it out to the containers. What may be surfacing here is that Hadoop 2.6-recent-Java7 incompatibility, where ticket renewal fails —even when you call checkTGTandRelogin. If you are using Java 7 of 1.7u45 or later, you'll need Hadoop 2.6.2 or later – stevel Jan 09 '16 at 18:37
Thanks for the explanation! How do you renew in Java if you started with UserGroupInformation.getUGIFromTicketCache()? I don't see any renewal option, and I find that launching a "kinit -R" doesn't seem to work either. – Wheezil Jun 21 '21 at 17:59
@Wheezil > if your original ticket is _not_ renewable then `kinit -R` won't help... just check its properties with `klist` (it has some options to display extra flags, but expiration-of-renewal-period is shown by default) – Samson Scharfrichter Jun 21 '21 at 18:20
Samson, thanks for the response. But what I really meant was, what should I call instead of checkTGTAndReloginFromKeytab() to renew the ticket, if I logged in using getUGIFromTicketCache() instead of loginUserFromKeytab()? – Wheezil Jun 21 '21 at 19:04
Hi Samson, could you comment on using KRB5CCNAME env var vs System.setProperty( "java.security.krb5.conf", path)? Is one better than the other? – Wheezil Jun 21 '21 at 19:15
`kinit` / `klist` ignore the Java property but honor the env var. That's a big difference if your Java app runs `kinit` in a subprocess... – Samson Scharfrichter Jun 21 '21 at 19:33