6

Lately, we are having some performance issues with our Kafka consumers and producers. We use Kafka Java API in scala. What is considered to be good practice wrt opening and closing of consumer and producer objects? I believe this is a quite open-ended question and the right answer is always depends but I am trying to reason about this.

Can Consumers can be long-running connections and left open?

Should producers be closed whenever we are done producing messages?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Achilleus
  • 1,824
  • 3
  • 20
  • 40
  • If you're not actually using a Closable object anymore, yes, it's a *good JVM practice* to close those resources... That should translate directly to Kafka API as well – OneCricketeer Nov 16 '18 at 01:12
  • Performance issues could exist at the broker or network level, and there's plenty of other buffering options in the clients that just saying you have some "performance issue" is rather vague – OneCricketeer Nov 16 '18 at 01:14
  • I do agree performance issues could exist at the broker or network level but we are having issues where in we are opening and closing consumer and producer objects very frequently. So I am assuming this could be a reason. Hence posted this question to get insights from professionals who deal with these kind of issues. – Achilleus Nov 16 '18 at 06:24
  • You might want to clarify "frequently", and even better with a code example – OneCricketeer Nov 16 '18 at 06:36

1 Answers1

2

Can Consumers can be long-running connections and left open?

In general, yes.

In detail: depending on your consumer configuration.

If your consumers are members of consumer group they certainly should be closed - to trigger the rebalance at earliest possible time.

If your consumers are using auto-commiting of offsets, they would still keep committing every N ms (AFAIK 60k), possibly wasting resources.

Otherwise, they can stay - but why waste resources?

Should producers be closed whenever we are done producing messages?

In general, yes.

Depends on your design, but if you can say at certain time you won't be sending any more messages, then you can close. That does not mean you should be closing and re-creating a producer after every sent message.

Adam Kotwasinski
  • 4,377
  • 3
  • 17
  • 40
  • 2
    my producer needs to send message everytime there's an order from customer, if i dont close and re-create producer everytime, would there be a bad things like memory leaks and issues? – user1955934 Dec 04 '19 at 00:27
  • 1
    i have this dilemma: it aint efficient if i open and close each time a message needs to be sent, but i hate to see memory leaks and warnings against not closing objects that implements autocloseable... pls someone clarify – user1955934 Jan 26 '20 at 16:32
  • warnings for not closing autocloseable implies code which performs the close cannot be found. this is bad, regardless of closing/opening frequency, and is what causes leaks. if you are still using it (and not ready to close it, yet) you aren't leaking. if you assume it will be closed when your app dies, you may not be correct. this is why the warning is shown: you should always close, even if its during app shutdown. use try-with-resources and (figuratively) never worry about closing ever again! – Derek White Apr 22 '21 at 01:27