2

I also want to know that since heartbeat requests are sent in Apache Kakfa Consumer, does it also affect the connections.max.idle.ms?

How do you handle errors in kafka-apache client (Producer and Consumer and best practices around them)?

Thanks :]

king.reflex
  • 313
  • 4
  • 10

1 Answers1

1

The idle socket timeout is reset on poll and heartbeat connections and could be considered an upper bound for any protocol request

If the idle timeout is less than either poll interval, session timeout, or heartbeat interval, then you might expect to see some dropped network connections

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Thanks for the response. Just for clarification, You mean that the heart beat request will reset the connections.max.idle.ms timeout? If possible could you also shed light on error handling kafka in production, I see most of the exceptions in kafka-python are supressed and reconnections and retries are baked in – king.reflex Feb 22 '22 at 03:24
  • 2
    I've not had to write a Python client in production with that library, so I'm not sure about catching errors. Obviously, catch any errors when processing your records... And there's multiple connections that are opened for consumers like metadata requests, fetching, heartbeats, and commits. The connection idle timeout should be greater than any one of those individual timeouts/intervals, and, individually, those requests' connection idle times are reset when they occur – OneCricketeer Feb 22 '22 at 14:03
  • If connections.max.idle.ms is less than say max.poll.interval.ms but not less than heartbeat.interval.ms, and assuming heartbeat.interval.ms, which operates on a different thread continues to connect normally, I don't understand how you will encounter dropped network connections. Unless you mean "all" instead of "either"? – awgtek Jul 02 '23 at 23:47
  • If not when you say reset, it's scheme-specific, i.e. despite heartbeats continuing, if say, poll interval takes longer than max.idle it'll drop the consumer causing a rebalance? – awgtek Jul 02 '23 at 23:55