I'm a beginner in KAFKA. I was assigned a simple task to enable the producer to retry upon send failure. We have already added a kafka producer callback for this send() method, that just logs the response(before introducing retries).
My objective: To attempt maximum 5 retries, in case of any failure with producer send(), and if the last retry attempt also fails, it should write a log, stating that the attempts to send the message failed after 5 retries or so.
I did the following.
- retries = 5
- retry.backoff.ms =35000 (Set this with an assumption that this retry interval should be greater than request.timeout.ms(30000 ms)- Not sure about this though)
- tried giving a broker that is down - to fail the producer send(), So that I can check if the retries config works [max.in.flight.requests.per.connection is not set to 1, as ordering wasn't important for us.]
I would like to know the following,
Does kafka in any way log the retries attempt? - Ex: logging during each attempt like "retrying...", or at the end of all retries like "failed after n retries"
I've been getting errors like these continuously say for n number of times. I'm confused if this is the retry attempt that I have configured. If it is so, why does it exceed more than 5 times. Or is it totally unrelated to the producer retries?
[kafka-producer-network-thread | producer-1] WARN o.apache.kafka.clients.NetworkClient: brokers may not be available
Project details:
- Gradle project
- kafka-clients-0.11.0.0
And please do bare with this type of vague question. I'm not able to give the complete details due to company policies and that I won't be able to copy paste the errors as this site is not accessible in my company's virtual machine.
