Is there a way to use kafka consumers and producers in the same Python file using confluent_kafka

Question

I am currently trying to allow for two scripts to communicate in Python using confluent_kafka.

The idea is that script 1 produces a message to script 2 for a temperature reading. Script 2 consumes the message from s1 and produces the current temperature back to script 1 and then prints the temperature to the console.

The code below has been very finicky for me and works sometimes but most of the time script2.py will run but consume no messages. I was wondering if this is possible or if there are better alternatives for communicating between 2 different python scripts over the internet.

Script1.py

from confluent_kafka import Consumer, Producer
#from confluent_kafka import Producer
from dotenv import load_dotenv
import os

load_dotenv(".env")

#Gathers sensitive data from the .env file
bootstrap_server = os.getenv("BOOTSTRAP_SERVER")
sasl_user_name = os.getenv("CLIENT_ID")
sasl_password = os.getenv("CLIENT_SECRET")

#Set up the Kafka producer
p = Producer({
      'bootstrap.servers': bootstrap_server,
      'security.protocol': 'SASL_SSL',
      'sasl.mechanisms': 'PLAIN',
      'sasl.username': sasl_user_name,
      'sasl.password': sasl_password,
})

c = Consumer({
    'bootstrap.servers': bootstrap_server,
    'security.protocol': 'SASL_SSL',
    'sasl.mechanisms': 'PLAIN',
    'sasl.username': sasl_user_name,
    'sasl.password': sasl_password,
    'group.id': 'script1-group',
    'enable.auto.commit': False,
    'auto.offset.reset': 'latest',
    
})

def delivery_report(err, msg):
    if err is not None:
        print('Message delivery failed: {}'.format(err))
    else:
        print('Message delivered to {} [{}]'.format(msg.topic(), msg.partition()))



p.poll(0)
data = 'temperature'
p.produce('script2', data.encode('utf-8'), callback=delivery_report)
p.flush()

c.subscribe(['script1'])

x = True
while x == True:
    msg = c.poll(1.0)
    if msg is None:
        continue
    if msg.error():
        print("Consumer error: {}".format(msg.error()))
        continue
    temp = msg.value().decode('utf-8')
    print("The temperature is " + temp)
    x = False

Script2.py

from confluent_kafka import Consumer, Producer
#from confluent_kafka import Producer
from dotenv import load_dotenv
import os

load_dotenv(".env")

#Gathers sensitive data from the .env file
bootstrap_server = os.getenv("BOOTSTRAP_SERVER")
sasl_user_name = os.getenv("CLIENT_ID")
sasl_password = os.getenv("CLIENT_SECRET")

#Set up the Kafka producer
p = Producer({
      'bootstrap.servers': bootstrap_server,
      'security.protocol': 'SASL_SSL',
      'sasl.mechanisms': 'PLAIN',
      'sasl.username': sasl_user_name,
      'sasl.password': sasl_password,
})

c = Consumer({
    'bootstrap.servers': bootstrap_server,
    'security.protocol': 'SASL_SSL',
    'sasl.mechanisms': 'PLAIN',
    'sasl.username': sasl_user_name,
    'sasl.password': sasl_password,
    'group.id': 'script2Group',
    'enable.auto.commit': False,
    'auto.offset.reset': 'latest',
    
})

def delivery_report(err, msg):
    if err is not None:
        print('Message delivery failed: {}'.format(err))
    else:
        print('Message delivered to {} [{}]'.format(msg.topic(), msg.partition()))

c.subscribe(['script2'])

x = True
while x == True:
    msg = c.poll(1.0)
    if msg is None:
        continue
    if msg.error():
        print("Consumer error: {}".format(msg.error()))
        continue
    if msg.value().decode('utf-8') == 'temperature':
        p.poll(0)
        data = "20 C"
        p.produce('script1', data.encode('utf-8'), callback=delivery_report)
        print("Temperature sent to Script 1")
        p.flush()
        x = False

OneCricketeer · Answer 1 · 2023-03-03T17:01:49.923

better alternatives for communicating between 2 different python scripts over the internet.

You could use gPRC or HTTP, and not need Kafka or any external message queue...

What you're doing is very similar to Kafka Steams design, which another library Fluvii, built on that python library is capable of doing.

Regarding your code, your scripts stop on the first consumed message. This is not recommended way to use Kafka. Consumers, at least, should always be long-running.

You also have disabled offset commits, and are not committing on your own. This, combined with auto.offset.reset=latest means you will never read any existing data from either topic; and so, it'll only work when you start script2 before script1

Is there a way to use kafka consumers and producers in the same Python file using confluent_kafka

1 Answers1