Kafka Producer can't connect to servers and will not throw any exception

Question

I have an extremely simple setup for sending message to Kafka:

        var producerConfig = new ProducerConfig
        {
            BootstrapServers = "www.example.com",
            SecurityProtocol = SecurityProtocol.SaslSsl,
            SaslMechanism = SaslMechanism.ScramSha512,
            SaslUsername = _options.SaslUsername,
            SaslPassword = _options.SaslPassword,
            MessageTimeoutMs = 1
        };

        var producerBuilder = new ProducerBuilder<Null, string>(producerConfig);

        using var producer = producerBuilder.Build();

        producer.Produce("Some Topic", new Message<Null, string>()
        {
            Timestamp = Timestamp.Default,
            Value = "hello" 
        });

Before, this code was working fine. Today it has decided to stop working and I'm trying to figure out why. I'm trying to get the Producer to throw an exception when failing to deliver a message, but it never seems to crash. Even when I fill in a wrong username and password, the producer still doesn't crash. Not even a logline in my local output window. How can I debug my Kafka connection when the producer never shows any problems?

score 2 · Answer 1 · answered Apr 14 '22 at 09:55

Produce is asynchronous and not blocking, function signature is

void Produce(string topic, Message<TKey, TValue> message, Action<DeliveryReport<TKey, TValue>> deliveryHandler = null)

In order to verify that a message was delivered without error

you can add a delivery report handler function e.g.


 private void DeliveryReportHandler(DeliveryReport<int, T> deliveryReport)
        {
            if (deliveryReport.Status == PersistenceStatus.NotPersisted)
            {
                _logger.LogError($"Failed message delivery: error reason:{deliveryReport.Error?.Reason}");
                _messageWasNotDelivered = true;
            }
        }

_messageWasNotDelivered = false;
_producer.Produce(topic,
                   new Message<int, T>
                   {
                       Key = key,
                       Value = entity
                   },
                   DeliveryReportHandler)
_producer.Flush(); // Wait until all outstanding produce requests and delivery report callbacks are completed

if(_messageWasNotDelivered ){
     // handle non delivery
}

This code can be trivially adjusted for batch producing like this

_messageWasNotDelivered = false;
foreach(var entity in entities){
    _producer.Produce(topic,
                   new Message<int, T>
                   {
                       Key = entity.Id,
                       Value = entity
                   },
                   DeliveryReportHandler)
}

_producer.Flush(); // Wait until all outstanding produce requests and delivery report callbacks are completed

if(_messageWasNotDelivered ){
     // handle non delivery
}

score 1 · Answer 2 · answered Mar 17 '22 at 08:48

You can add SetErrorHandler() to the ProducerBuilder. It would look like this:

    var producerBuilder = new ProducerBuilder<Null, string>(producerConfig)
                              .SetErrorHandler(errorMessageString => .....);

Set a breakpoint in that lambda and you can break on errors.

Kafka Producer can't connect to servers and will not throw any exception

2 Answers2