6

kafka consumer is failing to commit offset only for a particular partition.

aklsfoipafasldmaknfa    asiofuasofiusaofasd
[2019-01-04 12:22:22,691] ERROR [Consumer clientId=consumer-1, groupId=console-consumer-11955] Offset commit failed on partition my-topic-2-9 at offset 0: The request timed out. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2019-01-04 12:22:28,617] ERROR [Consumer clientId=consumer-1, groupId=console-consumer-11955] Offset commit failed on partition my-topic-2-9 at offset 1: The request timed out. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
as;lkasl;dkas;faskfasfasfasodaspd   qdoiwudqouoaisdiaduasodiuasd
[2019-01-04 12:23:18,875] ERROR [Consumer clientId=consumer-1, groupId=console-consumer-11955] Offset commit failed on partition my-topic-2-9 at offset 1: The request timed out. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

Can anyone explain me this error and what could possibly cause this?

Our cluster has 5 brokers running in AWS. We use Apache Kafka 2.1.

I am running a very simple Kafka-console-producer and consuming the same message using a Kafka console consumer.

I am seeing this error after the console-consumer consumes the message.

// PRODUCER
./bin/kafka-console-producer.sh   --broker-list kafka1:9092   --topic my-topic-2 --property "parse.key=true"   --property "key.separator=,"

 //CONSUMER
./bin/kafka-console-consumer.sh --bootstrap-server kafka1:9092 --from-beginning --topic my-topic-2 --property="print.key=true"

Mind that our cluster has over 200 topics with many producers and consumers.

It's just that I am not able to understand this behavior.

He's a screenshot of grafana. enter image description here

EDIT:

Please feel free to ask for any more details. This error is really frustrating.

EDIT 2:

./bin/kafka-topics.sh --describe --zookeeper zookeeper1:2181/kafka --topic my-topic-2
Topic:my-topic-2    PartitionCount:10   ReplicationFactor:3 Configs:
Topic: my-topic-2   Partition: 0    Leader: 4   Replicas: 4,2,3 Isr: 4,2,3
Topic: my-topic-2   Partition: 1    Leader: 5   Replicas: 5,3,4 Isr: 5,4,3
Topic: my-topic-2   Partition: 2    Leader: 1   Replicas: 1,4,5 Isr: 1,4,5
Topic: my-topic-2   Partition: 3    Leader: 2   Replicas: 2,5,1 Isr: 2,1,5
Topic: my-topic-2   Partition: 4    Leader: 3   Replicas: 3,1,2 Isr: 3,2,1
Topic: my-topic-2   Partition: 5    Leader: 4   Replicas: 4,3,5 Isr: 4,3,5
Topic: my-topic-2   Partition: 6    Leader: 5   Replicas: 5,4,1 Isr: 5,4,1
Topic: my-topic-2   Partition: 7    Leader: 1   Replicas: 1,5,2 Isr: 1,2,5
Topic: my-topic-2   Partition: 8    Leader: 2   Replicas: 2,1,3 Isr: 2,3,1
Topic: my-topic-2   Partition: 9    Leader: 3   Replicas: 3,2,4 Isr: 3,2,4

EDIT 3:

I am more interested in knowing the possible causes of this issue, this might help us figure out other problems with our cluster.

EDIT 4:

All the brokers, consumers and producers are in the same VPC in same region.
I understand that offset commit timeout can be increased, but why? Whats causing such latency? 5000 ms itself is too much for a system which is supposed to be realtime.
Its possible that kafka brokers are overloaded or network is congested, but why? As you can see that data input rate is at max 2-3 mbps, Is it too much for a kafka cluster of 5 machines (r5.xlarge)? Tell me if its the case, I am quite new to kafka.
What can become bottleneck in a setup like this?

Ankur rana
  • 580
  • 10
  • 27
  • I think an output from `$ kafka-topics --describe --zookeeper --topic ` can help more than a grafana screen. – mazaneicha Jan 04 '19 at 18:35
  • @mazaneicha : added the requested information – Ankur rana Jan 05 '19 at 09:55
  • Can you run the tool with logging set to `TRACE` and paste (a small extract of) the output? By default this tool picks `config/tools-log4j.properties`. Just set the rootLogger level to `TRACE` instead of `WARN` – Mickael Maison Jan 05 '19 at 13:55
  • @Ankur rana I guess you have already seen these (and other) SO https://stackoverflow.com/questions/51088681/kafka-streams-error-offset-commit-failed-on-partition-request-timed-out, https://stackoverflow.com/questions/50135591/kafka-console-consumer-error-offset-commit-failed-on-partition. Have you tried any of the solutions proposed? – mazaneicha Jan 05 '19 at 14:08
  • @MickaelMaison: I was unable to reproduce the error as of now, Maybe the issue only occurs during high load time. I will try this again tomorrow when the load is high. – Ankur rana Jan 05 '19 at 17:06
  • @mazaneicha: I did see that that question but the error messages are different. I am more interested in knowing the possible causes for this error. I am worried that this issue maybe an indication of a bigger issue with our cluster. – Ankur rana Jan 05 '19 at 17:10
  • Run the describe command on __consumer_offsets topic and see that topic has enough ISR (In Sync Replica) count. – Ajinkya Jan 05 '19 at 17:51
  • @Ajinkya : Yes, I verified it when the issue occurred. All replicas of __consumer_offsets were in sync all the time. – Ankur rana Jan 05 '19 at 18:35
  • At the risk of sounding silly, one of the causes could be that “The request timed out.” What is the timeout set to? Is it the default value? Timeouts can happen if the brokers are overloaded, or the network is congested, or the clients are located far away from the brokers (like over a WAN with a large round trip time). – Hans Jespersen Jan 05 '19 at 19:10
  • @Ankur rana One of the suggestions in the SO I cited before is to increase **`offsets.commit.timeout.ms`** on the brokers from default 5000 to something higher. If it does indeed help, you can start investigating what causes high latency. – mazaneicha Jan 05 '19 at 19:23
  • @HansJespersen I have added some more details to the question. Do you really think that brokers are overloaded with just 2-3 Mbps input data? Do you think this problem can be improved by adding more brokers to the system? P.S: Load average in our brokers remains around 7-8. Does that mean our brokers are overloaded? Ahh, I just saw that you work with Confluent :). Please help us. – Ankur rana Jan 05 '19 at 19:59
  • @mazaneicha : I am sure increasing timeout will help us as this problem only occurs when the load is high (not too high though, i think). But I am unable to figure out the bottleneck. – Ankur rana Jan 05 '19 at 20:00
  • @Ankur rana Just noticed that one of your brokers (kafka3) indicates 0 GC time despite having memory utilization on par with the rest. Looks strange. But I believe we've drifted pretty far off from the original question -- if the commits don't fail anymore, it becomes a new -- server config/tuning exercise vs programming -- question. – mazaneicha Jan 05 '19 at 20:34
  • @Ankurrana Did you manage to solve the problem, and if so, what worked for you? Thanks – xmar Jul 10 '19 at 07:37
  • Yes. The problem was with Kafka version itself. The problem was solved in Kafka version 2.1.1 – Ankur rana Jul 10 '19 at 08:50
  • Great, do you know what is the related KIP/JIRA for this improvement? – xmar Jul 10 '19 at 11:40
  • https://issues.apache.org/jira/browse/KAFKA-7697 – Ankur rana Jul 11 '19 at 09:41

1 Answers1

4

What is your ratio between your consumer threads to topic partitions ?

I found in my cluster that this error more likely to occur when small number of consumer threads consuming from big number of partitions (for instance 1 thread assigned to 30 topic partitions).

The best configuration that made this error disappeared for me was 1:1 (1 consumer thread for each topic partition) but now I got scales issue when want to add more consumer threads to group.

I deal with it by developing a mechanism of consumers deployment that enforce the 1:1 ration for example when deploying 3 consumers to consume 30 partitions each one will open 10 threads and for scaling let's say deploying 10 consumers, each one will open 3 threads...

I don't know if I'm following the best-practices here but it does the job for now

Tomer Lev
  • 61
  • 2
  • 7
  • I upvoted for the idea behind it: too many tasks for too few threads, it means too many partitions to be handled by a thread, so it takes longer for the same thread to "come back" to a previous partitions, potentially causing timeouts to be triggered. – xmar Jul 09 '19 at 14:41