0

I need to develop a Python program which would act as a Kafka Consumer and do some processing based on that. I used kafka-python and it did the job fine with local testing.

However, my production environment (RHEL7) requires Kerberos authentication. Keytab file is to be used for this purpose.

I couldn't find a way to refer to this Keytab file with kafka-python (e.g.: by configuring Jaas config file in a Java Kafka Consumer).

I found confluent-kafka-python, which seems to support Kerberos authentication via a Keytab file.

Also, I read that kafka-python could be used if a Java process handles Kerberos authentication.

Which method would be the better solution?

dab92
  • 155
  • 8
  • There are ways to convert JKS / JAAS properties into PEM files that kafka-python can use, AFAIK – OneCricketeer Jul 02 '21 at 14:42
  • Ahem. With due respect, the comment above seems off topic. JAAS is a generic Java security library, which handles many authentication protocols. But in that question the Kafka broker is not accepting SSL client certs _(whether stored in PEM format or JKS format)_ but Kerberos tickets; the issue is to have the Python client generate 2 tickets on-the-fly, first a TGT from a keytab file _(which contains the encrypted pwd for the headless client "principal")_ then a service ticket to give to the Kafka broker. – Samson Scharfrichter Jul 02 '21 at 22:26
  • Anyway, "Confluent's Python Client for Kafka" is a thin wrapper around the C client, which supports Kerberos out-of-the-box -- **except** that the version available on PyPi does not support Kerberos... _"NOTE: The pre-built Linux wheels do NOT contain SASL Kerberos/GSSAPI support. If you need SASL Kerberos/GSSAPI support you must install librdkafka and its dependencies using the repositories below and then build confluent-kafka using the command in the "Install from source from PyPi" section below."_ – Samson Scharfrichter Jul 02 '21 at 22:35
  • If you are brave enough to build the package yourself, then you have to pass the Kerberos configuration properties with custom code because the Python wrapper does not handle them -- cf. the example in https://github.com/confluentinc/confluent-kafka-python/blob/master/examples/sasl_producer.py – Samson Scharfrichter Jul 02 '21 at 22:41
  • Anyway, there is a Cloudera KB entry about running `kafka-python` against a kerberized Kafka broker: https://community.cloudera.com/t5/Community-Articles/Running-a-producer-in-a-kerberized-HDP-3-1-Kafka-2-0-0/ta-p/244205 – Samson Scharfrichter Jul 02 '21 at 22:53
  • @SamsonScharfrichter Sorry for the late reply. I installed librdkafka and Confluent's Python Client for Apache Kafka was installed accordingly. I found [link](https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka), which seems to guide how to handle Kerberos authentication. I'm not clear about what you meant by '.....then you have to pass the Kerberos configuration properties with custom code because the Python wrapper does not handle them...'. Can't I use above mentioned method with Confluent's Python Client for Apache Kafka? – dab92 Jul 22 '21 at 07:44
  • Your link deals with the C library. More specifically, with features of the C library that are _not supported_ by the Python wrapper offered by Confluent. If you want to patch the Python wrapper to support these features, then go for it, and happy testing! – Samson Scharfrichter Jul 22 '21 at 14:22

0 Answers0