2

I am using kafka-python package to publish messages to kafka confluent cloud cluster.

My code looks like below:

    from kafka import KafkaProducer
    producer=KafkaProducer(
                    bootstrap_servers='pkc-epgnk.us-central1.gcp.confluent.cloud:9092',
                    security_protocol='SASL_SSL',
                    sasl_mechanism='PLAIN',
                    ssl_certfile='/usr/local/etc/openssl/cert.pem',
                    sasl_plain_username='[api_key]',
                    sasl_plain_password='[api_secret]')

    producer.prodcue(topic='file-ingestion',key=b'',value=b'test')

Running above code I get below error:

kafka.errors.NoBrokersAvailable: NoBrokersAvailable

The confluent kafka manual suggested me to set below values for a C/C++ producer; however, I can not set all values by the kafka-python producer

bootstrap.servers=pkc-epgnk.us-central1.gcp.confluent.cloud:9092
api.version.request=true
broker.version.fallback=0.10.0.0
api.version.fallback.ms=0
sasl.mechanisms=PLAIN
security.protocol=SASL_SSL
ssl.ca.location=/usr/local/etc/openssl/cert.pem
sasl.username=<CLUSTER_API_KEY>
sasl.password=<CLUSTER_API_SECRET>
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Mahdi
  • 787
  • 1
  • 8
  • 33
  • @luckylukee did you manage to connect to confluent cloud kafka with `kafka-python` package? – Erik K Sep 20 '19 at 13:00

1 Answers1

-2

You could instead use the confluent-kafka-python client to set those properties and to have better integration and tested support between products

Example code from Github

from confluent_kafka import Producer, Consumer

p = Producer({
    'bootstrap.servers': '<ccloud bootstrap servers>',
    'broker.version.fallback': '0.10.0.0',
    'api.version.fallback.ms': 0,
    'sasl.mechanisms': 'PLAIN',
    'security.protocol': 'SASL_SSL',
    'sasl.username': '<ccloud key>',
    'sasl.password': '<ccloud secret>'
})
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • 1
    this really doesnt answer the question of how to do it with `kafka-python` library – Erik K Sep 20 '19 at 13:00
  • So we cannot connect to Confluent Kafka using python kafka?? Has anybody connected using kafka-python package? – Arpan Sharma Jan 30 '20 at 11:05
  • @Arpan There's no such thing as "Confluent Kafka". Everything is Apache Kafka. This is the only Python library I've used, therefore it's my answer. If you have a specific error, please create your own post – OneCricketeer Jan 30 '20 at 14:17
  • @cricket_007 I have used both libraries kafka-python and confluent-kafka. With python-kafta i get NoBrokerAvailable as mentioned in this post and with confluent-kafka package I am able to connect. Isn't it strange!!! – Arpan Sharma Jan 30 '20 at 19:34
  • @ArpanSharma Sounds like you have networking or certificate issues. That isn't a Kafka or Confluent Cloud problem. – OneCricketeer Jan 30 '20 at 22:48