0

I have an application that uses Kafka-Streams-API. There is no problem while I working at the local. I want to get connected to the remote Kafka broker for the stage test. And the remote Kafka broker is set to work with the GSSAPI sasl mechanism and used Kerberos. I got an error when I run my Streams Application which wrote with java. After I looked for the error message I found an answer but still has a problem.

Error message; Error while fetching metadata with correlation id 3 : {[APPID]-KTABLE-AGGREGATE-STATE-STORE-0000000008-repartition=UNKNOWN_TOPIC_OR_PARTITION

The answer I found said;

When using an Authorizer and a user doesn't have Describe authorization on a topic, the broker will no longer return TOPIC_AUTHORIZATION_FAILED errors to requests since this leaks topic names. Instead, the UNKNOWN_TOPIC_OR_PARTITION error code will be returned. This may cause unexpected timeouts or delays when using the producer and consumer since Kafka clients will typically retry automatically on unknown topic errors. You should consult the client logs if you suspect this could be happening.

So my actual question is how to Authorize the Remote Topic which using to GSAPI sasl mechanism?

Kadir Alan
  • 209
  • 1
  • 13

1 Answers1

0

A little late but this is the answer. There are two ways to connect a Kerberized Kafka cluster with a Stream App.

Details

  1. Cached credentials: Create a jaas.conf file with cached credentials. To use cached Kerberos credentials, where you use kinit first, use this configuration. It is suitable for just manual runs.

    KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true; };

  2. By using keytabs: Create a jaas.conf file with keyTab and principal info in it. It is suitable for automatic runs.

    KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/mykafkaclient.keytab" principal="mykafkaclient/clients.hostname.com@EXAMPLE.COM"; };

Define an environment variable with

export KAFKA_OPTS="-Djava.security.auth.login.config=/home/user/jaas.conf"

Do not forget to control reaching a file named KRB5.conf (in unix) or KRB5.ini (in Windows)

krb5

Of course, there are some parameters that must be defined properly. If you want to access kafka topics with Kafka CLI, then create a client.properties file with the content

<code>security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=kafka
sasl.mechanism=GSSAPI
</code>

With the Kafka stream app you will give these parameters within a properties.

  • Kerberos is a complex topic. If someone is interested in details of it, I highyl recommend the "Hadoop and Kerberos: The Madness beyond the Gate" git-book
Vezir
  • 101
  • 7