0

I need to connect to kafka instance which has multiple brokers with SSL. I am using kafka-python to consume and process data.

I have seen link where they used kafka-python to connect to single broker with SSL.

Link : http://maximilianchrist.com/python/databases/2016/08/13/connect-to-apache-kafka-from-python-using-ssl.html

Specific code 

consumer = KafkaConsumer(bootstrap_servers='my.server.com',
                          security_protocol='SSL',
                          ssl_check_hostname=True,
                          ssl_cafile='CARoot.pem',
                          ssl_certfile='certificate.pem',
                          ssl_keyfile='key.pem')

I need to know since there are multiple brokers how to specify multiple ssl_ca, ssl_cert, ssl_key in constructor ??

Lijo Jose
  • 317
  • 1
  • 3
  • 13

2 Answers2

0

SSL should be set up such that the client can authenticate with any of your brokers. I recommend you take a look at this tutorial to get a better idea on SSL basics https://docs.confluent.io/current/kafka/encryption.html#kafka-ssl-encryption

dawsaw
  • 2,283
  • 13
  • 10
  • Archaic question but I am commenting since those questions are recurring. The doc you are presenting refers to Java Kafka Clients. On Python side things are a bit different therefore this answer is not helpful – rpd Jan 10 '23 at 12:10
0

Old question but because I faced the same issue (available answers most of the time are Java oriented, which has a different architecture using keystores and truststores), I am posting a working solution. Please note I am using confluent Kafka, but the solution should work the same with minor changes for other python kafka libraries.

Kafka brokers is just a single string of comma separated values. If each broker has different certificate, then just copy all server certificates in a single .pem file.

A consumer which consumes from 3 brokers with separate certificates should look as follows:

kafka_brokers = 'broker_1,broker_2,broker_3'
ssl_ca_location = 'all_server_certs.pem'
consumer = Consumer({
    'bootstrap.servers': kafka_brokers,
    'security.protocol': 'SSL',
    'enable.ssl.certificate.verification': 'true',
    'ssl.certificate.location': 'client-cert.pem',
    'ssl.ca.location': ssl_ca_location,
    'ssl.key.location': 'client-key.pem', 
    'group.id': 'my_group',
    'auto.offset.reset': 'earliest'
    })
rpd
  • 462
  • 1
  • 9
  • 24