1

My Flink streaming job writes to secure hdfs. Job runs smoothly after kinit till kerberos ticket lifetime expiry. I can not increase the lifetime which is now 1d. Please tell me how should I handle this error.

Error log:

Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:554)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:369)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:723)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:719)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:718)
        ... 26 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
        ... 35 more
Maxim
  • 9,701
  • 5
  • 60
  • 108
Anand Shaw
  • 229
  • 5
  • 13
  • Use a headless keytab to allow Flink to retain access for a longer period of time. – Ben Watson Jun 11 '16 at 17:46
  • Thanks Ben. Flink is accessing external secure hdfs cluster and keytab and kdc server maintained by other team. I have to use the keytab which is provided. Is there any other way. – Anand Shaw Jun 12 '16 at 03:46
  • You can renew the ticket cache with "kinit -R" anytime before it's expired. But there's a max times limit. You can ask your kdc manager to set the expire time of your principal to max value so your ticket will never expire. – waltersu Jun 12 '16 at 06:17
  • You've got the same problem with HBase, Spark, etc. http://stackoverflow.com/questions/33211134/hbase-kerberos-connection-renewal-strategy/33243360#33243360 *(except that Spark has a kind of built-in solution...)* – Samson Scharfrichter Jun 14 '16 at 08:56

0 Answers0