0

I am running into some issues when trying to publish a number of messages onto a Kafka topic using the Confluent.Kafka nuget package.

Everything is set up correctly to the topic and I am able to write to it (before it terminates). It seems that the number of messages I am trying to publish might have something to do with it.

I have set up a Kafka Client with DI that is used in a class constructor to instantiate a producer that is shared for that class. Doing it this way will work for a little (< 20 secs) before I get the following error:

%4|1660846799.980|TERMINATE|rdkafka#producer-1| [thrd:app]: Producer terminating with 2 messages (3790 bytes) still in queue or transit: use flush() to wait for outstanding message delivery

Assertion failed: (r == 0), function rwlock_wrlock, file tinycthread_extra.c, line 157.

However, if I change from using a class instance producer and instead set up a producer for each message that I want to publish, I get no error.

My problem is that I really don't want to be doing it that way and would love to figure out if I can use a single producer instance for all the messages.

Any help or insight into why this error is happening would be greatly appreciated.

Ben Ritter
  • 1

1 Answers1

1

You should explicitly call flush() function, as the error indicates when you are done sending events, or close the producer instance, which should trigger the same. Specifically, sending a record will batch the events into larger network requests, which are not sent to the broker immediately. If you don't flush the producer, then you drop records when the app stops.

Creating one producer per record is not recommended.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • I had tried that before and just tried putting in a ```flush()``` call, however I am still seeing this error: ```Assertion failed: (r == 0), function rwlock_wrlock, file tinycthread_extra.c, line 157.``` – Ben Ritter Aug 23 '22 at 13:17
  • 1
    Also just to clarify, this is written as a long running service, so it should not end and just continuously receive and produce messages onto Kafka topic – Ben Ritter Aug 23 '22 at 13:52
  • The line before the assertion error is the real error... Perhaps you can post a reproducible example at https://github.com/confluentinc/confluent-kafka-dotnet/issues – OneCricketeer Aug 23 '22 at 18:40
  • Yeah I will open up an issue there. Thanks for looking at this and the suggestions – Ben Ritter Aug 24 '22 at 13:03