0

I am working on kafka producer docker app which is working fine on MAC however it is failing on windows and Linux machines with the following error. The docker build is working on windows/linux/mac however when we run docker run, the apps seems to ONLY work on MAC.

Error:

[2022-09-06 17:53:42,174] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
 File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2051, in wsgi_app
   response = self.full_dispatch_request()
 File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1501, in full_dispatch_request
   rv = self.handle_user_exception(e)
 File "/usr/local/lib/python3.8/site-packages/flask_cors/extension.py", line 161, in wrapped_function
   return cors_after_request(app.make_response(f(*args, **kwargs)))
 File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1499, in full_dispatch_request
   rv = self.dispatch_request()
 File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1485, in dispatch_request  
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
 File "/function/model/api.py", line 21, in home
   return modelcode()
 File "/function/model/api.py", line 49, in modelcode
   producer = SerializingProducer(producer_conf)
 File "/usr/local/lib/python3.8/site-packages/confluent_kafka/serializing_producer.py", line 109, in __init__
   super(SerializingProducer, self).__init__(conf_copy)
cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Failed to create producer: No
provider for SASL mechanism GSSAPI: recompile librdkafka with libsasl2 or openssl support. Current build options: PLAIN SASL_SCRAM OAUTHBEARER"}

Dockerfile:

ARG FUNCTION_DIR="/function"
FROM python:3.8
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update &&       apt-get -y install  &&    apt-get -y install
RUN  apt-get -y install librdkafka-dev
RUN pip install --no-binary :all: confluent-kafka==0.11.0
ARG FUNCTION_DIR
RUN mkdir -p ${FUNCTION_DIR}
RUN apt-get -y install libsasl2-modules-gssapi-mit
COPY cacert.pem ${FUNCTION_DIR}
COPY CP_KAFKA_EAC_CC_DEV.keytab ${FUNCTION_DIR}
COPY ./requirements.txt ${FUNCTION_DIR}/requirements.txt
RUN  pip install -r ${FUNCTION_DIR}/requirements.txt
COPY . ${FUNCTION_DIR}/
WORKDIR ${FUNCTION_DIR}
EXPOSE 8080
ENTRYPOINT exec gunicorn --timeout 80 --bind 0.0.0.0:8080 wsgi

requirements.txt:

Flask_Cors==3.0.0
python_jose==3.0.1
flask_restplus==0.11.0
Flask_JWT_Extended==3.18.0
Flask==2.0
requests==2.20.0
PyJWT==1.7.1
cryptography==35.0
py-healthcheck==1.9.0
gunicorn==20.0.4
Werkzeug~=2.0.0
jinja2==3.0.3
confluent_kafka==1.7.0
pandas==1.4.2
numpy==1.23.2
pyOpenSSL==22.0.0
kafka-python

$ winpty docker run -it -p 8088:8080 sample-flask-app:0.0.1

[2022-09-09 02:16:25 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2022-09-09 02:16:25 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2022-09-09 02:16:25 +0000] [1] [INFO] Using worker: sync
[2022-09-09 02:16:25 +0000] [8] [INFO] Booting worker with pid: 8
sample output
[2022-09-09 02:16:44,672] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2051, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1501, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1499, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1485, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/function/model/api.py", line 21, in home
    return modelcode()
  File "/function/model/api.py", line 49, in modelcode
    producer = SerializingProducer(producer_conf)
  File "/usr/local/lib/python3.8/site-packages/confluent_kafka/serializing_producer.py", line 109, in __init__
    super(SerializingProducer, self).__init__(conf_copy)
cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Failed to create producer: No provider for SASL mechanism GSSAPI: recompile librdkafka with libsasl2 or openssl support. Curr
ent build options: PLAIN SASL_SCRAM OAUTHBEARER"}
^C[2022-09-09 02:16:50 +0000] [1] [INFO] Handling signal: int
[2022-09-09 02:16:50 +0000] [8] [INFO] Worker exiting (pid: 8)
[2022-09-09 02:16:50 +0000] [1] [INFO] Shutting down: Master
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
bomsabado
  • 35
  • 6
  • On Mac, `librdkafka` installation (done with `pip install confluent_kafka`) includes the necessary libraries. I do not think this is true elsewhere. You do not need **both** `confluent_kafka` and `kafka-python`. Use the latter to remove the C bindings – OneCricketeer Sep 10 '22 at 01:57
  • @OneCricketeer Are you saying I need to recreate remove kafka-python from my requirement.txt? I was not clear when you say 'Use the latter to remove the C bindings' Can you please elaborate? – bomsabado Sep 10 '22 at 02:02
  • You have an error with `librdkafka` not being compiled with GSSAPI (it is a C library). `kafka-python` is a native Python Kafka client that does not need `librdkafka`. But it does not include capabilities with Confluent Schema Registry, so pick accordingly... – OneCricketeer Sep 10 '22 at 02:05
  • what do I need to update to make sure the docker run works fine on Windows. – bomsabado Sep 10 '22 at 02:05
  • You're building an Ubuntu container, so "docker run on Windows" doesn't seem relevant. e.g. `docker run` on Mac should have the same issue – OneCricketeer Sep 10 '22 at 02:05
  • Also, you should probably remove `confluent-kafka==0.11.0` from your Dockerfile since a higher version is in the requirements.txt – OneCricketeer Sep 10 '22 at 02:09
  • No, I am using docker desktop with wsl2 installed on windows OS. The app runs fine on the MAC OS. Just having issues on Windows/Linux operating systems – bomsabado Sep 10 '22 at 02:10
  • If it runs fine on Mac, can you `docker push` that image somewhere so you can pull and run in Windows/Linux? – OneCricketeer Sep 10 '22 at 02:11
  • I am able to run the docker image (built on MAC OS) on my windows OS. BUT, I am trying to understand why it does not work if I build the image on windows machine – bomsabado Sep 10 '22 at 02:14
  • What happens when you remove this `RUN pip install --no-binary :all: confluent-kafka==0.11.0`? – OneCricketeer Sep 10 '22 at 02:17
  • And what happens when you use [the latest version](https://github.com/confluentinc/confluent-kafka-python/releases)? – OneCricketeer Sep 10 '22 at 02:18
  • Possibly related, remove the `apt-get install -t librdkafka-dev`... [That will **not** get the latest version](http://archive.ubuntu.com/ubuntu/pool/universe/libr/librdkafka/) (1.9.2) – OneCricketeer Sep 10 '22 at 02:24
  • I still get the same error. Here are my files - – bomsabado Sep 10 '22 at 02:25
  • You seem to be having the same issue as this person - https://github.com/edenhill/librdkafka/issues/3963#issue-1361550606 – OneCricketeer Sep 10 '22 at 02:30
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/247973/discussion-between-bomsabado-and-onecricketeer). – bomsabado Sep 12 '22 at 00:22

0 Answers0