KafkaProducer - GSSAPI lib not available

Question

I am writing a python program to connect to Kafka and read/write message.

Producer error on executing python3 producer.py

File "/opt/local/bgdatapp/anaconda3/lib/python3.7/site-packages/kafka/conn.py", line 255, in init
assert gssapi is not None, 'GSSAPI lib not available'
AssertionError: GSSAPI lib not available
Exception ignored in: <function BrokerConnection.del at 0x7f715f4b3378>
Traceback (most recent call last):
File "/opt/local/bgdatapp/anaconda3/lib/python3.7/site-packages/kafka/conn.py", line 696, in del
self._close_socket()
File "/opt/local/bgdatapp/anaconda3/lib/python3.7/site-packages/kafka/conn.py", line 691, in _close_socket
if self._sock:
AttributeError: 'BrokerConnection' object has no attribute '_sock'
INFO:kafka.producer.kafka:Kafka producer closed

OS - Red Hat Enterprise Linux Server release 6.10 (Santiago) Python3 - 3.7.1 - Anaconda3 Python Python3 path - /opt/local/bgdatapp/anaconda3/bin/python Kerberos - 5 Kafka - Cloudera 13.1

I am able to access my kafka from shell and i am able to push and read messages.

kafka-console-producer --broker-list host.domain.com:9092 --topic Topic1 --producer.config client.properties

cat client.properties security.protocol=SASL_PLAINTEXT sasl.kerberos.service.name=kafka

Simulating the same from anaconda python is throwing error.

import os
import socket
import gssapi
import logging
from kafka import KafkaProducer
KAFKA_TOPIC = 'Topic1'
KAFKA_BROKERS = 'host.domain.com:9092'
os.environ['KAFKA_OPTS'] = '-Djava.security.auth.login.config=/opt/local/account1/jaas.conf'
logging.basicConfig(level=logging.INFO)
messages = [b'hello kafka', b'I am sending', b'3 test messages']
producer = KafkaProducer(bootstrap_servers=KAFKA_BROKERS, api_version=(0 , 10), security_protocol='SASL_PLAINTEXT', sasl_mechanism='GSSAPI', sasl_kerberos_service_name='kafka', max_request_size=3173440261)
for m in messages:
  print (producer.send(KAFKA_TOPIC, m).get(timeout=30))

ERROR:

File "/opt/local/bgdatapp/anaconda3/lib/python3.7/site-packages/kafka/conn.py", line 255, in init
    assert gssapi is not None, 'GSSAPI lib not available'
    AssertionError: GSSAPI lib not available
    Exception ignored in: <function BrokerConnection.del at 0x7f715f4b3378>
    Traceback (most recent call last):
    File "/opt/local/bgdatapp/anaconda3/lib/python3.7/site-packages/kafka/conn.py", line 696, in del
    self._close_socket()
    File "/opt/local/bgdatapp/anaconda3/lib/python3.7/site-packages/kafka/conn.py", line 691, in _close_socket
    if self._sock:
    AttributeError: 'BrokerConnection' object has no attribute '_sock'
    INFO:kafka.producer.kafka:Kafka producer closed

Can you suggest any fix to resolve this issue?

Thanks

score 1 · Answer 1 · answered Mar 31 '19 at 03:58

1

There are actually two errors here. The first is the gssapi import error; the second is an error during gc/del (self._sock AttributeError). The second error is just a bug. We'll fix that in the next release. But assuming you fix the gssapi import error, then this second AttributeError should not happen.

Ok, so what gssapi library do you have installed? Are you able to do this in your environment:

from gssapi.raw.misc import GSSError

It looks like you may have installed the older / deprecated python-gssapi module. You want this one: https://pypi.org/project/gssapi/ (not this one: https://pypi.org/project/python-gssapi/)

answered Mar 31 '19 at 03:58

dpkp

1,369
7
14

Thanks dpkp for your inputs. The gssapi.raw is not setup & it threw the below error: -bash-4.1$ python3 Python 3.7.1 (default, Dec 14 2018, 19:28:38) [GCC 7.3.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information. >>> from gssapi.raw.misc import GSSError Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'gssapi.raw' We have installed the older module https://pypi.org/project/python-gssapi/ . Let me get the new one installed. – arvj Apr 01 '19 at 19:18

KafkaProducer - GSSAPI lib not available

1 Answers1