Confluent-kafka client OutOfOrderSequenceException while Kafka/MSK cluster had security patch applied

Question

When sending data to kafka using an async, idempotent producer, we received errors showing below. We restarted producer applications and resolved an issue. We opened a ticket with Confluent - Ticket Link. They mentioned that it's an existing bug in the Kafka client we use and suggested by restarting producers or resetting the transactional producer will help to address the issue in mean time they release new client updates (not in their release pipeline yet).

We tried to reproduce the issue in our dev and QA environments with the higher load but could not re-produce it. Does anyone have an idea how can we possible re-produce the same issue? Also, if are there any other suggestions to handle this issue in production?

Error, we received in our producer application, most likely caused by the confluent-kafka-dotnet client we use. -

"%3|1668238507.285|ERROR|<PUBLISHER_APP_NAME>#producer-1|: Fatal error: Broker: Broker received an out of order sequence number: ProduceRequest for <TOPIC_NAME> [38] with 1 message(s) failed due to sequence desynchronization with broker 1 (PID{Id:22181,Epoch:0}, base seq 0, idemp state change 8184098ms ago, last partition error NO_ERROR (actions , base seq 0..0, base msgid 0, -1ms ago)"

Our Producer configuration looks like:

config = new ProducerConfig {
    BootstrapServers = appConfiguration.KafkaBootStrapServers,
    SecurityProtocol = SecurityProtocol.Ssl,
    Acks = Acks.All,
    EnableIdempotence = true,
    ClientId = appConfiguration.KafkaClientID
};

Our Asp.net C# Producer method call looks like:

try {
        Headers headers = new Headers();
        headers.Add(<CONSTANT>.ATT_TRANSACTION_ID, Encoding.ASCII.GetBytes(transactionId));
        DeliveryResult<string, string> response =
                                   await kafkaProducer.ProduceAsync(topic, new Message<string, string> { Key = key, Value = eventData, Headers = headers });
        return response; // logging
    }
    catch (ProduceException<string, string> ex)
    {
        // catch exception, processing and logging
        return null;
    }
    catch (Exception ex)
    {
        // catch exception and logging
        return null;
    }

Environment -

1. Kafka cluster (v2.8.0) with 3 brokers and in-sync replicas

2. Confluent.Kafka nuget library 1.7.0 with librdkafka.redist v1.7.0