kafka-python producer - SSL connection failure - Trustore Only

Question

I'm trying to publish messages via python to a kafka topic and am receiving an error. I can connect and publish via the CLI. Hoping for some guidance. I've googled and read the docs. Thanks!!

Successful CLI command:

 kafka-console-producer --broker-list 
 123.45.67.891:1234,123.45.67.892:1234,123.45.67.893:1234 -- 
 producer.config C:\Users\fake_user\Kafka\client-ssl.properties --topic FakeTopic

 Contents of client-ssl.properties:
   security.protocol = SSL
   ssl.truststore.location = C:/Users/fake_user/Kafka/kafka-truststore
   ssl.truststore.password = fakepass

Code:

from kafka import KafkaProducer

  producer = KafkaProducer(bootstrap_servers=['123.45.67.891:1234', '123.45.67.892:1234', '123.45.67.893:1234'],
                           security_protocol='SSL',
                           ssl_certfile=r'C:\Users\fake_user\Kafka\kafka-truststore',
                           ssl_password='fakepass')

  producer.send('FakeTopic', value='python_test', key='test')

Resultant Error:

Traceback (most recent call last):
  File "kafka_post_test.py", line 6, in <module>
    ssl_password='fakepass')
  File "C:\Users\fake_user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\kafka\producer\kafka.py", line 381, in __init__
    **self.config)
  File "C:\Users\fake_user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\kafka\client_async.py", line 239, in __init__
    self.config['api_version'] = self.check_version(timeout=check_timeout)
  File "C:\Users\fake_user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\kafka\client_async.py", line 874, in check_version
    version = conn.check_version(timeout=remaining, strict=strict, topics=list(self.config['bootstrap_topics_filter']))
  File "C:\Users\fake_user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\kafka\conn.py", line 1078, in check_version
    if not self.connect_blocking(timeout_at - time.time()):
  File "C:\Users\fake_user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\kafka\conn.py", line 331, in connect_blocking
    self.connect()
  File "C:\Users\fake_user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\kafka\conn.py", line 420, in connect
    if self._try_handshake():
  File "C:\Users\fake_user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\kafka\conn.py", line 496, in _try_handshake
    self._sock.do_handshake()
  File "C:\Users\fake_user\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1051)

Are you able to produce messages with the console producer? – Giorgos Myrianthous Jun 07 '19 at 16:46 — Giorgos Myrianthous, Jun 07 '19 at 16:46
Yes I am. Also via KafkaTool. – Justin Bailey Jun 07 '19 at 17:24 — Justin Bailey, Jun 07 '19 at 17:24

score 1 · Answer 1 · answered Jun 07 '19 at 17:24

1

Check out this link.

You have to add the SSL Cert to the JVM keystore for any pretty much any program that is run by Java.

answered Jun 07 '19 at 17:24

ThatCampbellKid

561
1
5
19

Thanks for the link. I dug into it. Please know that using the same documentation I'm able to connect with only the truststore file via the console-producer provided with the kafka distribution. These servers do not require client side certification, so there is no keystore to include. I'm stuck trying to duplicate the trustsore-only connection with the python library. – Justin Bailey Jun 07 '19 at 20:07

score 1 · Answer 2 · answered Oct 06 '20 at 19:44

1

I've found that by default, the python-kafka library, sets the ssl_cafile property to None. Setting it to the default os (/etc/pki/tls/cert.pem on linux) allowed me to connect to the kafka brokers.

https://kafka-python.readthedocs.io/en/master/_modules/kafka/producer/kafka.html#KafkaProducer.send

answered Oct 06 '20 at 19:44

Ameya Panse

11
1

Thank you for contributing to our community! Please, focus on the question and improve your answer. You can add more formatting and details, which could help in problem solution. – Fedor Soldatkin Oct 06 '20 at 20:45

kafka-python producer - SSL connection failure - Trustore Only

2 Answers2