Unable to push subsequent transactions to Kafka Producer

Question

I'm trying to send events to Kafka in batches (let's say 10 events in a batch) to maintain transactional guarantees. But I'm able to successfully send only the first batch of events to Kafka. All subsequent batches fail with the reason KafkaError{code=_STATE,val=-172,str="Operation not valid in state Ready"}

Here is the piece of code I'm using to use transactions in Kafka.

DEFAULT_PRODUCER_CONFIG = {
"bootstrap.servers": BOOTSTRAP_SERVERS,
"schema.registry.url": SCHEMA_REGISTRY_URL,
"enable.idempotence": True,
"retries": 3,
"acks": "all",
"security.protocol": "SSL",
"transactional.id": uuid.uuid4().hex
}


producer = AvroProducer(
        DEFAULT_PRODUCER_CONFIG, default_value_schema=avro.loads(value_schema)
)

send_in_transaction method is used to send events in a batches.

def send_in_transaction(producer, topic, events):
    producer.init_transactions()
    producer.begin_transaction()
    try:
        for event in events:
            producer.produce(topic=topic, key=None, value=event)
        producer.commit_transaction(60)
    except AvroTypeException as ex:
        producer.abort_transaction()
        logger.error("Avro serialization error: " + str(ex))
        raise KafkaException("Error sending events to kafka")
    except Exception as ex:
        producer.abort_transaction()
        logger.error("Error sending events to kafka: " + str(ex))
        raise KafkaException("Error sending events to kafka")

Even though, I don't use transactions, I'm still not able to push subsequent batches of events.

Soultions Tried -

I tried creating kafka producer for every transaction with uuid as producer's transactional.id and it works fine. But this doesn't seem to be a good approach.

*Note - Here, I'm using confluent_kafka==1.7.0 and avro-python3==1.10.0

score 0 · Answer 1 · answered Jul 04 '22 at 02:54

You may get some glimpse about this error and solution from the post https://github.com/confluentinc/confluent-kafka-dotnet/issues/1496 and https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+scalability+for+exactly+once+semantics

Key points from the post

This error could occur due to multiple producer with the same transactional.id started after this application. side note: you need to make sure you disable auto commit on the consumer (which is on by default) - not sure if you're doing that.
Increasing default TransactionTimout setting could help resolving this issue.

A producer has unique transactional.id only. Issue resolves when I initialize the avro producer every time before creating next the transaction. But this is not the correct approach. — Shivam, Jul 05 '22 at 11:18

Unable to push subsequent transactions to Kafka Producer

1 Answers1