1

I've found some similar questions, but they're not quite the same situation as this.

I have a Kafka Streams application which authenticates with brokers using Kerberos ticket details found within a Credential Cache.

The application works great until the original ticket's expiry is reached, then I get the following error.

04:21:45.630 [kafka-producer-network-thread | sample-app-StreamThread-1-producer] ERROR org.apache.kafka.clients.NetworkClient - [Producer clientId=sample-app-StreamThread-1-producer] Connection to node 2 (<Hostname>/<ipAddress>:<Port>) failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.

Now, that would all seem expected, but my ticket is renewed every 2 hours by another system, and yet, the Kafka Streams application isn't detecting that the ticket has been renewed. Querying the ticket using 'klist' tells me that there is a valid ticket at the time when the error occurs.

Ticket cache: FILE:/var/ABC/SYSTEM_ACCOUNT/cc/krb5cc_12345
Default principal: 12345@EXCHAD.ABC123.com

Valid starting     Expires            Service principal
04/02/20 02:28:02  04/02/20 12:28:02  krbtgt/EXCHAD.ABC123.com@EXCHAD.ABC123.com
        renew until 04/08/20 08:28:04

Oddly, I can bounce my application again, and it'll work, but only until the new current ticket's expiry is reached in approx 10 hours.

Why isn't Kafka Streams looking for the latest ticket? Is this potentially a bug within Kafka Streams itself? I can't find any other settings related to this beyond the initial JAAS configuration.

com.sun.security.auth.module.Krb5LoginModule required
refreshKrb5Config=true
useKeyTab=false
useTicketCache=true
renewTGT=true
doNotPrompt=true
ticketCache="/var/ABC/SYSTEM_ACCOUNT/cc/krb5cc_12345"
principal="12345@EXCHAD.ABC123.com"

I'm using Java 8, and Kafka Streams 2.4.0

As always, any help or guidance would be greatly appreciated.

Thanks!

Scott Smith
  • 23
  • 2
  • I am not familiar with security details, but Kafka Streams inherits the behavior from the underlying `KafkaConsumer` and `KafkaProducer` -- hence, if there is an issue, it's not related to Kafka Streams itself, but to the clients. Maybe reading up on the client docs helps. – Matthias J. Sax Apr 03 '20 at 20:35
  • Side note: I guess that `renewTGT` would apply if you used a keytab to generate a private in-memory ticket, but not here (JAAS never writes anything into the kerberos cache, it only reads the existing TGT). – Samson Scharfrichter Apr 05 '20 at 11:26
  • _"my ticket is renewed every 2 hours by another system" ... `FILE :`_ > from a security standpoint it's less than ideal to use a file to store the credentials for one (or even many) service(s)... That might explain why it's not fully covered by the Kafka libs. You should inspect the source code to sort it out. – Samson Scharfrichter Apr 05 '20 at 11:32

0 Answers0