0

I would like to know if it's possible to configure 2 different Kafka cluster in a Kafka producer.

Currently I'm trying to have my producers & consumer failback automatically to a passive cluster without reconfiguring (bootstrap.servers) and restarting their application.

I'm using Apache Kafka 2.8 and the confluent_kafka==1.8.2 package with Python 3.7.

Below the producer code:

from time import sleep

from confluent_kafka import Producer

p = Producer({'bootstrap.servers': 'clusterA:32531, clusterB:30804'})


def delivery_report(err, msg):
    """ Called once for each message produced to indicate delivery result.
        Triggered by poll() or flush(). """
    if err is not None:
        print('Message delivery failed: {}'.format(err))
    else:

        print(f'Message delivered to {msg.offset()}')


with open('test_data.csv', 'r') as read_obj:
    csv_reader = reader(read_obj)
    header = next(csv_reader)
    # Check file as empty
    if header is not None:
        # Iterate over each row after the header in the csv
        for row in csv_reader:
            sleep(0.02)
            p.produce(topic='demo', key=row[5], value=str(row), callback=delivery_report)
p.flush()

When I killed clusterB I got the following error message.

%4|1643837239.074|CLUSTERID|rdkafka#producer-1| [thrd:main]: Broker clusterA:32531/bootstrap reports different ClusterId "MLWCRsVXSxOf2YGPRIivjA" than previously known "6ZtcQCRPQ5msgeD3r7I11w": a client must not be simultaneously connected to multiple clusters
%3|1643837240.995|FAIL|rdkafka#producer-1| [thrd:clusterB:30804/bootstrap]: 172.27.176.222:30804/bootstrap: Connect to ipv4#clusterB:30804 failed: Unknown error (after 2044ms in state CONNECT)
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Astream
  • 11
  • 2

1 Answers1

0

At the moment, You will have to update the bootstrap information to secondary Cluster manually and this will require the restart of the client to failover.

Programmatically, Inorder to connect to a separate cluster you will have to stop the current producer instance and start a new instance with the new bootstrap server config. However this can get quite complicated.

Other options are,

  • You configure kafka with a LB or a VIP (Not recommended because by nature a direct connection from the client to broker is required)
  • Configure a shared store (memcached or redis) where you store the bootstrap server config. Your client will fetch the bootstrap server during the bootstrap process. During failure you change the value in the store and restart your clients. (This makes the operation quite easy)
glitch99
  • 264
  • 2
  • 7
  • 1
    A separate Zookeeper znode could also be used to store/change the bootstrap address if one doesn't want to run additional infrastructure – OneCricketeer Feb 03 '22 at 13:59